Video image denoising method, device, electronic device and readable storage medium

Abstract

The present application provides a method, device, electronic device and readable storage medium for denoising a video image. The electronic device performs denoising on each frame to be processed in the forward and reverse order videos of the video to be processed; then, the denoising results of the forward and reverse order videos are fused to obtain a denoised video corresponding to the video to be processed; wherein, during the denoising process, the denoising result of the frame to be processed is related to the denoising result of the reference frame; the reference frame is one or more video frames located before the frame to be processed. The above method can improve the denoising effect of the video to be processed.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, device, electronic device and readable storage medium for denoising a video image. Background Art

[0002] In the existing video image denoising technology, the video image can be denoised to improve the display effect of the video image. The denoising process can be time domain denoising, frequency domain denoising, or spatial domain denoising.

[0003] However, the above denoising process may easily cause ghosting and other phenomena to appear in the denoised video image, resulting in poor denoising effect of the video image. Summary of the invention

[0004] Embodiments of the present application provide a method, device, electronic device, and readable storage medium for denoising a video image.

[0005] A method for denoising a video image, comprising:

[0006] For each frame to be processed in the forward sequence video and the reverse sequence video of the processed video, denoising the frame to be processed; in the denoising process, the denoising result of the frame to be processed is related to the denoising result of the reference frame; the reference frame is one or more video frames located before the frame to be processed;

[0007] The denoising results of the forward-order video and the reverse-order video are fused to obtain a denoised video corresponding to the video to be processed.

[0008] A video image denoising device, comprising:

[0009] A denoising module is used to perform denoising on each frame to be processed in the forward and reverse order videos of the processed video; during the denoising process, the denoising result of the frame to be processed is related to the denoising result of the reference frame; the reference frame is one or more video frames located before the frame to be processed;

[0010] The fusion module is used to fuse the denoising results of the normal-order video and the denoising results of the reverse-order video to obtain a denoised video corresponding to the video to be processed.

[0011] An electronic device comprises a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of the above-mentioned video image denoising method.

[0012] A computer-readable storage medium stores a computer program, which implements the steps of the above-mentioned video image denoising method when executed by a processor.

[0013] The above-mentioned video image denoising method, device, electronic device and readable storage medium, the electronic device performs denoising on each frame to be processed in the forward sequence video and reverse sequence video of the video to be processed; then, the denoising results of the forward sequence video and the denoising results of the reverse sequence video are fused to obtain the denoised video corresponding to the video to be processed; wherein, during the denoising process, the denoising result of the frame to be processed is related to the denoising result of the reference frame; the reference frame is one or more video frames located before the frame to be processed. Since the denoising result of the frame to be processed is related to the denoising result of the previous one or more reference frames during the denoising process, after the forward sequence video and the reverse sequence video of the video to be processed are denoised respectively, the denoising results corresponding to each video frame are different; by fusion processing of the denoising results of the forward sequence video and the denoising results of the reverse sequence video, the positive denoising effect of the forward sequence video and the reverse sequence video can be fused in the denoising video, and the negative effects brought by the forward sequence video or the reverse sequence video during the denoising process, such as image ghosting, can also be reduced, thereby improving the denoising effect of the video to be processed as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 is a flow chart of a method for denoising a video image in one embodiment;

[0016] Figure 2 is a flow chart of a method for denoising a video image in one embodiment;

[0017] Figure 3 is a schematic diagram of a method for denoising a video image in one embodiment;

[0018] Figure 4 is a flow chart of a method for denoising a video image in one embodiment;

[0019] Figure 5 is a schematic diagram of a method for denoising a video image in one embodiment;

[0020] Figure 6 is a flow chart of a method for denoising a video image in one embodiment;

[0021] Figure 7 is a flow chart of a method for denoising a video image in one embodiment;

[0022] Figure 8is a flow chart of a method for denoising a video image in one embodiment;

[0023] Fig. 9 is a flow chart of a method for denoising a video image in one embodiment;

[0024] Fig.10 is a schematic diagram of a method for denoising a video image in one embodiment;

[0025] Fig.11 is a flow chart of a method for denoising a video image in one embodiment;

[0026] Fig.12 is a flow chart of a method for denoising a video image in one embodiment;

[0027] Fig.13 is a structural block diagram of a video image denoising device in one embodiment;

[0028] Fig.14 is a structural block diagram of a video image denoising device in one embodiment;

[0029] Fig.15 is a structural block diagram of a video image denoising device in one embodiment;

[0030] Fig.16 is a structural block diagram of a video image denoising device in one embodiment;

[0031] Fig.17 is a structural block diagram of a video image denoising device in one embodiment;

[0032] Fig.18 is a structural block diagram of a video image denoising device in one embodiment;

[0033] Fig.19 FIG. 4 is a structural block diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0035] The video image denoising method in the present application can be applied to electronic devices. The electronic devices can be personal computers, mobile terminals, personal digital assistants, wearable electronic devices, etc. The electronic devices can perform denoising on the video to be processed, and the video to be processed can be a video captured by the electronic device or a video downloaded from a server by the electronic device, which is not limited here.

[0036] Figure 1FIG. 1 is a flow chart of a method for denoising a video image in one embodiment. Figure 1 As shown, the denoising method of the video image includes:

[0037] S101, performing denoising on each frame to be processed in the forward-sequence video and the reverse-sequence video of the video to be processed; during the denoising process, the denoising result of the frame to be processed is related to the denoising result of the reference frame; the reference frame is one or more video frames located before the frame to be processed.

[0038] The video to be processed may be a video collected by the electronic device, a video downloaded by the electronic device from a server, or a video sent by other electronic devices. The source of the video to be processed is not limited. The electronic device may perform denoising on the video to be processed during the process of collecting the video to be processed, may perform denoising on the video to be processed during the process of sending the video to be processed, or may perform denoising on the video to be processed after extracting it from the storage space. The timing of the denoising is not limited.

[0039] The forward-order video of the video to be processed may be a video arranged in the playback order or acquisition order of the video frames. The reverse-order video of the video to be processed may be a video in which each video frame in the video to be processed is arranged in the reverse order of the playback order or acquisition order; the electronic device may arrange each video frame in the video to be processed in reverse order to obtain a reverse-order video.

[0040] In addition, the above-mentioned forward-sequence video and reverse-sequence video can be obtained by arranging the video frames of the complete video to be processed in sequence, or can be obtained by arranging some video frames in the video to be processed in sequence, that is, the forward-sequence video and reverse-sequence video of the video to be processed can include multiple video clips. For example, the above-mentioned video to be processed can include 10 frames, and the above-mentioned forward-sequence video can be the video to be processed, that is, the 1st frame to the 10th frame, arranged in the playback order; the above-mentioned reverse-sequence video can be a video sequence arranged from the 10th frame to the 1st frame. In addition, the above-mentioned forward-sequence video can include a video clip formed by the 1st frame to the 5th frame, and a video clip formed by the 6th frame to the 10th frame; the above-mentioned reverse-sequence video can include a video clip formed by the 5th frame to the 1st frame, and a video clip formed by the 10th frame to the 6th frame; when the electronic device performs denoising on the reverse-sequence video, the 4th frame can use the 5th frame as a reference frame, and the 6th frame can use the 7th frame as a reference frame.

[0041] The electronic device can treat each video frame in the forward-order video and the reverse-order video as a frame to be processed for denoising. That is, the same frame image in the video to be processed can be treated as two frames to be processed for denoising twice after being arranged in forward order and reverse order, and different denoising results can be obtained.

[0042] The electronic device can perform denoising on each frame to be processed in the forward and reverse sequence videos. The denoising can be time domain denoising, space-time joint denoising, or joint denoising of time domain and color domain or frequency domain, which is not limited here.

[0043] During the denoising process, the denoising result of the frame to be processed is related to the denoising result of the reference frame; wherein the reference frame is one or more video frames located before the frame to be processed. During the denoising process, the denoising result of the frame to be processed may be related to the time domain denoising result of the frame to be processed and the time domain denoising result of the reference frame, or may be related to the denoising result of the reference frame in multiple domains such as the spatial domain and the time domain, which is not limited here.

[0044] In a video sequence of a forward-sequence video or a reverse-sequence video, the reference frame is one or more video frames located before the frame to be processed, and the number of the multiple video frames may be 2 or 3; the multiple reference frames may be adjacent reference frames or reference frames with a preset interval; the reference frame may be adjacent to the frame to be processed or may have a preset interval with the video to be processed; the selection method of the reference frame is not limited here. Since the arrangement order of each video frame in the forward-sequence video and the reverse-sequence video is different, for the same frame image, the corresponding reference frame in the forward-sequence video is different from the corresponding reference frame in the reverse-sequence video; based on the denoising results of different reference frames, the electronic device can obtain different denoising results for the same frame image.

[0045] S102: Fusing the denoising result of the normal-sequence video and the denoising result of the reverse-sequence video to obtain a denoised video corresponding to the video to be processed.

[0046] After the electronic device performs denoising on each frame to be processed in the forward-sequence video and the reverse-sequence video, a denoising result of the forward-sequence video and a denoising result of the reverse-sequence video are obtained. The denoising result of the forward-sequence video may include initial denoised frames of each frame to be processed obtained based on the arrangement order of the video frames in the forward-sequence video; the denoising result of the reverse-sequence video may include initial denoised frames of each frame to be processed obtained based on the arrangement order of the video frames in the reverse-sequence video.

[0047] Furthermore, the electronic device may fuse the denoising results of the forward-sequence video and the denoising results of the reverse-sequence video. During the above fusion process, the electronic device fuses the initial denoising frames that are time-matched in the various denoising results, that is, fuses the denoising frames corresponding to the same frame in the video to be processed. For example, the denoising results of the forward-sequence video include the initial denoising frame of the first frame of the video to be processed, and the denoising results of the reverse-sequence video may also include the initial denoising frame of the first frame of the video to be processed (for example, corresponding to the tenth frame in the reverse-sequence video). The electronic device may fuse the initial denoising frames corresponding to the first frames to obtain the denoising frame corresponding to the first frame of the video to be processed.

[0048] When the electronic device performs fusion processing on each initial denoised frame, it may perform fusion in units of frames, pixels, or image blocks in the denoised frame, which is not limited here.

[0049] Taking pixel-level fusion as an example, the electronic device performs fusion processing on the pixels at the same position corresponding to each initial denoised frame. The electronic device can calculate the pixel average value of the pixels at the same position in each initial denoised frame and use the average value as the target pixel value of the pixel; or, the electronic device can fuse each initial denoised frame based on the image features in each initial denoised frame, such as texture features, brightness features, ghost features, etc.; the fusion processing method is not limited here.

[0050] In the above-mentioned video image denoising method, the electronic device performs denoising on each frame to be processed in the forward sequence video and reverse sequence video of the video to be processed; then, the denoising results of the forward sequence video and the denoising results of the reverse sequence video are fused to obtain the denoised video corresponding to the video to be processed; wherein, during the denoising process, the denoising result of the frame to be processed is related to the denoising result of the reference frame; the reference frame is one or more video frames located before the frame to be processed. Since the denoising result of the frame to be processed is related to the denoising result of the previous one or more reference frames during the denoising process, after the forward sequence video and the reverse sequence video of the video to be processed are denoised separately, the denoising results corresponding to each video frame are different; by fusion processing of the denoising results of the forward sequence video and the denoising results of the reverse sequence video, the positive denoising effect of the forward sequence video and the reverse sequence video can be fused in the denoising video, and the negative effects brought by the forward sequence video or the reverse sequence video during the denoising process, such as image ghosting, can also be reduced, thereby improving the denoising effect of the video to be processed as a whole.

[0051] Figure 2 FIG. 1 is a flow chart of a method for denoising a video image in another embodiment. This embodiment relates to a method for performing denoising on a frame to be processed by an electronic device. Based on the above embodiment, Figure 2As shown, the above S101 includes:

[0052] S201 , performing spatial domain denoising processing on the chrominance components of the frame to be processed to obtain the chrominance components of the spatial domain denoised frame of the frame to be processed.

[0053] The electronic device can perform format conversion on each frame to be processed, convert the frame to be processed into a YUV format, and obtain the chrominance component of the frame to be processed and the brightness component of the frame to be processed. The chrominance component of the frame to be processed includes the chrominance value of each pixel of the frame to be processed; the brightness component of the frame to be processed includes the brightness value of each pixel of the frame to be processed.

[0054] With respect to the above chrominance components, the electronic device may perform spatial domain denoising processing on the chrominance components of the frame to be processed to obtain the chrominance components of the spatial domain denoised frame corresponding to the frame to be processed.

[0055] The electronic device may use a spatial domain filter to filter the chrominance component of the frame to be processed, so that the chrominance component of the frame to be processed is smoother. The spatial domain filter may be a Gaussian filter, a median filter, or a mean filter, which is not limited here.

[0056] Optionally, the electronic device may perform spatial denoising on the chrominance component of the frame to be processed using a non-local mean spatial filtering method. The non-local mean spatial filtering method is specifically as follows:

[0057] For the chrominance components of the frame to be processed, the electronic device can traverse the chrominance components of the frame to be processed with a preset size to determine whether each image block needs to be subjected to spatial denoising. The preset size of the above-mentioned image block can be set according to the noise form of the frame to be processed. The noise ranges from fine low-frequency noise to patchy high-frequency noise, and the size of the above-mentioned image block can be larger and larger. When the electronic device traverses the chrominance components of the frame to be processed, the traversal compensation can be less than n, and the convenience compensation can be adjusted with the noise form. For example, the compensation can be smaller and smaller when the noise ranges from fine low-frequency noise to patchy high-frequency noise; in addition, the above-mentioned traversal step size can also be adjusted with the noise size. The larger the noise, the smaller the step size.

[0058] Taking an n×n image block as an example, the electronic device can divide the frame to be processed into multiple image blocks, and then calculate the pixel gradient corresponding to each image block respectively, where the pixel gradient is used to measure the degree of change of the pixel value in the image block. If the pixel gradient corresponding to the image block is greater than the preset gradient threshold, the electronic device can determine not to perform spatial denoising on the image block; if the pixel gradient corresponding to the image block is less than or equal to the preset gradient threshold, the electronic device can determine that spatial denoising is required for the image block. The preset gradient threshold can be set according to the noise level of the frame to be processed; the greater the noise, the greater the corresponding gradient threshold.

[0059] Furthermore, for the current image block that needs to be processed by spatial domain denoising, the electronic device may determine multiple image blocks around the current image block Cur as reference blocks Ref, such as Figure 3 As shown; then a preset matching method (such as SAD matching method) is used to match the reference block with the current image block. If the SAD is less than a preset matching threshold, the reference block is considered to match the current image block. The above matching threshold can be set according to the noise level of the frame to be processed; the greater the noise, the greater the corresponding matching threshold.

[0060] If the reference block matches the current image block successfully, all matched reference blocks are weightedly fused with the current image block to obtain the spatial denoising block corresponding to the current image block; after each image block that needs to be denoised is processed in the above manner, the chrominance component of the spatial denoising frame of the frame to be processed can be obtained.

[0061] S202 , performing spatial domain denoising processing on the brightness component of the frame to be processed to obtain the brightness component of the spatial domain denoised frame corresponding to the frame to be processed.

[0062] The manner of performing spatial denoising on the brightness component of the frame to be processed is similar to the manner of performing spatial denoising on the chrominance component described above, and will not be elaborated here. It should be noted that when the electronic device performs spatial denoising on the brightness component and the chrominance component, the denoising parameters used may be the same or different. For example, when the electronic device uses the above-mentioned non-local mean spatial filtering method to perform spatial denoising on the brightness component, the parameters such as the gradient threshold, matching threshold, and image block size used may be different from the parameters used to process the chrominance component.

[0063] S203, using the target weight of each pixel, weightedly fuse the brightness component of the reference frame of the frame to be processed and the brightness component of the spatial domain denoised frame to obtain the brightness component of the temporal domain denoised frame corresponding to the frame to be processed; the target weight is related to the saturation, brightness and difference value of the pixel.

[0064] The electronic device may calculate a target weight for each pixel in the frame to be processed, and the target weight may be related to the saturation, brightness, and difference value of the pixel. The difference value may be the difference in pixel values ​​between the frame to be processed and the reference frame at the same position, and may be used to characterize the degree of difference between the frame to be processed and the reference frame.

[0065] When calculating the above target weight value, the electronic device may respectively calculate the fusion weights related to the saturation, brightness and difference value of each pixel, and then obtain the target weight of each pixel based on the various fusion weights.

[0066] When calculating each fusion weight, the electronic device may first calculate image parameters such as saturation, brightness, and difference value of each pixel, and then determine each fusion weight based on the above image parameters. Furthermore, for each pixel, the electronic device may perform weighted summation, multiplication, and other processing on each fusion weight to obtain a target weight for the pixel.

[0067] Optionally, the electronic device determines the target weight of each pixel according to the first brightness component, the first chrominance component and the brightness component of the reference frame; wherein the first brightness component includes the brightness component of the frame to be processed or the brightness component of the spatial denoising frame; the first chrominance component includes the chrominance component of the frame to be processed or the chrominance component of the spatial denoising frame. In other words, the electronic device can calculate the target weight using the brightness component of the frame to be processed or the brightness component of the spatial denoising frame. By introducing the first brightness component, the first chrominance component and the brightness component of the reference frame to calculate the target weight, the electronic device can make the target weight combine more image features of the frame to be processed, so that the target weight is more accurate.

[0068] When the electronic device performs time domain denoising on the brightness component after spatial denoising, the target weights mentioned above can be used to weightedly fuse the brightness component of the reference frame of the frame to be processed and the brightness component of the spatial denoised frame. When the electronic device performs weighted fusion on each brightness component, a preset weighted calculation formula can be used to obtain the fused brightness component; for example, the weighted calculation formula mentioned above can be a weighted summation formula, and the electronic device can multiply the brightness component of the spatial denoised frame by the target weight and then add it to the brightness component of the reference frame, or can multiply the brightness component of the reference frame by the target weight and then add it to the brightness component of the spatial denoised frame, and there is no limitation here.

[0069] Optionally, the electronic device can be configured according to the formula Get the brightness component F of the time domain denoised frame of the frame to be processed i,j ; Among them, S i,j Represents the brightness component of pixel (i, j) in the spatial denoising frame; R i,j Represents the brightness component of pixel (i, j) in the reference frame; jnt i,j Represents the target weight of pixel (i,j).

[0070] S204 , obtaining a denoised output frame of the frame to be processed according to the luminance component of the temporal denoised frame and the chrominance component of the spatial denoised frame.

[0071] On the basis of the above steps, the electronic device performs spatial denoising and temporal denoising on the luminance component of the frame to be processed in sequence, and obtains the luminance component of the temporal denoising frame; and performs spatial denoising on the chrominance component of the frame to be processed, and obtains the chrominance component of the spatial denoising frame. Further, the electronic device can obtain the denoised output frame of the frame to be processed based on the luminance component of the temporal denoising frame and the chrominance component of the spatial denoising frame. The electronic device can directly combine the luminance component of the temporal denoising frame with the chrominance component of the spatial denoising frame to obtain the denoised output frame of the frame to be processed, and can also filter or fuse the luminance component of the temporal denoising frame or the chrominance component of the spatial denoising frame, and then combine the processed luminance component and chrominance component to obtain the denoised output frame of the frame to be processed; the method for obtaining the above denoising output frame is not limited here.

[0072] In the above-mentioned video image denoising method, the electronic device processes the brightness component and the chrominance component of the frame to be processed separately. The electronic device performs spatial denoising on the chrominance component to obtain a black date image of the chrominance component. For the brightness component, spatial denoising is first performed, and then the brightness component of the spatial denoising frame is fused with the brightness component of the reference frame through the target weight of each pixel to achieve temporal denoising, thereby obtaining a noise-free image of the brightness component; in addition, when the electronic device performs temporal denoising on the brightness component, the target weight of the pixel is determined using multiple dimensions such as the saturation, brightness and difference value of the pixel, so that the frame to be processed obtains a better denoising effect.

[0073] Figure 4 FIG. 1 is a flow chart of a method for denoising a video image in another embodiment. This embodiment relates to a method for an electronic device to obtain a denoised output frame. Based on the above embodiment, Figure 4 As shown, the above S204 includes:

[0074] S301, calculating the texture size of each pixel in the brightness component of the frame to be processed.

[0075] The electronic device can calculate the texture size of each pixel based on the brightness component of the frame to be processed. The above texture size is used to characterize whether there is a texture feature at the pixel position of the frame to be processed, or the size of the image texture corresponding to the pixel. The electronic device can perform texture extraction on the brightness component of the frame to be processed, and then determine the texture size of each pixel based on the texture extraction result. Optionally, the electronic device can use an edge operator to process the brightness component of the frame to be processed to obtain the texture size of each pixel.

[0076] S302, determining the texture fusion weight value of each pixel in the frame to be processed according to the texture size and a preset texture weight mapping relationship; in the texture weight mapping relationship, the larger the texture size of the pixel, the larger the corresponding texture fusion weight value.

[0077] The electronic device obtains the texture size of each pixel, realizes the quantitative representation of the texture feature of each pixel, and further determines the texture fusion weight of the pixel according to the texture size. The texture fusion weight is used to fuse the brightness component of the frame to be processed with the brightness component of the spatial domain denoising frame.

[0078] A texture weight mapping relationship may be preset in an electronic device. The texture weight mapping relationship may include texture fusion weight values ​​corresponding to different texture sizes. In the texture weight mapping relationship, the larger the texture size of the pixel, the larger the corresponding texture fusion weight value. In the texture weight mapping relationship, the texture size may have a linear or nonlinear change relationship with the texture fusion weight value; in addition, the texture weight mapping relationship may also include texture fusion weight values ​​corresponding to each interval of the texture size, for example, when the texture size is in the first interval, the corresponding texture fusion weight value is L1; when the texture size is in the second interval, the corresponding texture fusion weight value is L2; ​​wherein the lower limit of the second interval is greater than the upper limit of the first interval, and the L2 is greater than L1.

[0079] It can be understood that the larger the texture size of each pixel in the frame to be processed, the richer the corresponding image texture; in order to avoid the loss of image texture caused by time domain denoising processing, the electronic device can set a higher weight to fuse the original image texture of the frame to be processed into the image after time domain denoising to avoid the loss of texture details.

[0080] The electronic device may calculate the texture fusion weight of each pixel based on the texture weight mapping and the texture size of each pixel.

[0081] S303 , based on the texture fusion weight value, fuse the brightness component of the time-domain denoised frame with the brightness component of the frame to be processed to obtain the brightness component of the denoised output frame of the frame to be processed.

[0082] On the basis of obtaining the texture fusion weight value of each texture pixel, the electronic device can perform weighted summation on the brightness component of the time domain denoised frame and the brightness component of the frame to be processed to obtain the brightness component of the denoised output frame of the frame to be processed.

[0083] S304: Combine the luminance component of the denoised output frame with the chrominance component of the spatial domain denoised frame to obtain a denoised output frame of the frame to be processed.

[0084] Furthermore, the electronic device may combine the luminance component of the denoised output frame with the chrominance component of the spatial domain denoised frame to obtain a denoised output frame of the frame to be processed.

[0085] by Figure 5Take as an example, the denoising process in this embodiment is explained. Wherein, CurFrm represents the frame to be processed, CurFrmY represents the brightness component of the frame to be processed, and CurFrmC represents the chrominance component of the frame to be processed. SNR is a spatial denoising processing module, and TNR is a temporal denoising processing module. The electronic device inputs CurFrmY and CurFrmC into SNR respectively to obtain the brightness component SnrFrmY of the spatial denoised frame and the chrominance component SnrFrmC of the spatial denoised frame. For SnrFrmY, the electronic device obtains the brightness component RefFrmY of the temporal denoised frame corresponding to the reference frame of the frame to be processed, such as the brightness component of the temporal denoised frame of the previous frame to be processed output by TNR; then RefFrmY, SnrFrmY, CurFrmY and CurFrmC are input into TNR together, and SnrFrmY is subjected to temporal denoising by TNR to obtain the brightness component TnrFrmY of the temporal denoised frame. Among them, RefFrmY, SnrFrmY, CurFrmY and CurFrmC can be used in TNR to calculate the target weight of each pixel, and then SnrFrmY and RefFrmY are denoised and fused based on the target weight to obtain TnrFrmY. Furthermore, the electronic device records TnrFrmY in the reference frame list for time domain denoising of the next image, and inputs TnrFrmY and CurFrmY into the MERGE module. The above-mentioned MERGE module is used to fuse TnrFrmY and CurFrmY based on the texture fusion weight value of the pixel to obtain the brightness component OutFrmY of the denoised output frame. The electronic device directly uses SnrFrmC as the chrominance component OutFrmC of the denoised output frame, and obtains the denoised output frame OutFrm by combining OutFrmY with OutFrmC.

[0086] In the above-mentioned video image denoising method, when the time domain denoising process may cause the loss of image texture, the electronic device calculates the texture fusion weight value of each pixel, and fuses the brightness component of the time domain denoised frame with the brightness component of the frame to be processed based on the texture fusion weight value, so that the texture characteristics of the frame to be processed can be retained in the brightness component of the denoised output frame, thereby improving the denoising effect of the frame to be processed.

[0087] Figure 6 FIG. 1 is a flow chart of a method for denoising a video image in another embodiment. This embodiment relates to a method for an electronic device to determine a target weight of a pixel. Based on the above embodiment, Figure 6 As shown, the above method also includes:

[0088] S401. Calculate the saturation weight of each pixel in the frame to be processed according to the first brightness component and the first chrominance component.

[0089] Among them, the first brightness component includes the brightness component CurFrmY of the image to be processed or the brightness component SnrFrmY of the spatial denoising frame. The first chrominance component includes the chrominance component CurFrmC of the image to be processed or the chrominance component SnrFrmC of the spatial denoising frame. That is, when calculating the saturation weight of each pixel in the frame to be processed, select one from CurfrmY and SnrfrmY, and select one from CurfrmC and SnrfrmC to calculate the saturation weight of each pixel in the frame to be processed. In this application, the combination of selecting one from CurfrmY and snrfrmY and selecting one from CurfrmC and snrfrmC is not limited.

[0090] Specifically, when calculating the saturation weight of each pixel in the frame to be processed according to the first brightness component and the first chromaticity component, the saturation map of the image to be processed can be calculated based on the first brightness component and the first chromaticity component of each pixel in the frame to be processed. Then, each saturation in the saturation map is input into the corresponding function to calculate the saturation weight Sat_wgt.

[0091] Optionally, the electronic device performs color gamut mapping on the first brightness component and the first chrominance component to obtain the saturation of each pixel in the frame to be processed; then, based on the saturation of each pixel and a preset saturation weight mapping relationship, the saturation weight of each pixel in the frame to be processed is calculated; in the saturation weight mapping relationship, the greater the saturation of the pixel, the smaller the corresponding saturation weight.

[0092] A saturation weight mapping relationship may be preset in an electronic device. The saturation weight mapping relationship may include saturation weights corresponding to different saturations. In the saturation weight mapping relationship, the greater the saturation of the pixel, the smaller the corresponding saturation weight. In the saturation weight mapping relationship, the saturation may have a linear or nonlinear change relationship with the saturation weight; in addition, the saturation weight mapping relationship may also include saturation weights corresponding to each interval of saturation. The electronic device may also preset a saturation threshold, and determine the saturation weight of each pixel based on the saturation threshold and the saturation weight mapping relationship. For example, when the saturation of the pixel is greater than the saturation threshold, the corresponding saturation weight is S1, and when the saturation of the pixel is less than or equal to the saturation threshold, the corresponding saturation weight is S2.

[0093] S402: Calculate the brightness weight of each pixel in the frame to be processed according to the first brightness component.

[0094] The first brightness component includes the brightness component CurFrmY of the image to be processed or the brightness component SnrFrmY of the spatial denoising frame. That is, when calculating the brightness weight of each pixel in the frame to be processed, one of CurfrmY and SnrFrmY can be selected.

[0095] Specifically, when calculating the brightness weight of the pixel of the frame to be processed according to the first brightness component, a brightness map of the frame to be processed can be calculated based on the first brightness component of each pixel in the frame to be processed; wherein the brightness map includes the brightness value of each pixel. Then, each brightness value in the brightness map is input into the corresponding function to calculate the brightness weight Lum_wgt.

[0096] Optionally, the electronic device can generate a brightness mapping map of the frame to be processed based on the first brightness component; then, calculate the brightness weight of each pixel in the frame to be processed based on the brightness mapping map and a preset brightness weight mapping relationship; in the brightness weight mapping relationship, the larger the brightness value of the pixel, the smaller the corresponding brightness weight.

[0097] A brightness weight mapping relationship may be preset in an electronic device. The brightness weight mapping relationship may include brightness weights corresponding to different brightnesses. In the brightness weight mapping relationship, the greater the brightness of the pixel, the smaller the corresponding brightness weight. In the brightness weight mapping relationship, the brightness value may have a linear or nonlinear change relationship with the brightness weight; in addition, the brightness weight mapping relationship may also include brightness weights corresponding to each interval of brightness. The electronic device may also preset a brightness threshold and determine the brightness weight of each pixel based on the brightness threshold and the brightness weight mapping relationship. For example, when the brightness value of the pixel is greater than the brightness threshold, the corresponding brightness weight is L1, and when the brightness value of the pixel is less than or equal to the brightness threshold, the corresponding brightness weight is L2.

[0098] S403 , calculating the difference value weight of each pixel in the frame to be processed according to the brightness component of the spatial denoising frame and the brightness component of the reference frame.

[0099] Specifically, when calculating the difference value weight of each pixel in the frame to be processed according to the brightness component SnrFrmY of the spatial denoising frame and the brightness component RefFrmY of the reference frame, the boundary map of the frame to be processed can be calculated first; wherein the boundary map includes the difference value of each pixel. Then, each difference value in the boundary map is input into the corresponding function to calculate the difference value weight Bdr_wgt.

[0100] Optionally, the electronic device can perform a differential calculation on the brightness component of the spatial denoising frame and the brightness component of the reference frame to obtain a boundary mapping map; then, based on the boundary mapping map and a preset difference value weight mapping relationship, determine the difference value weight of each pixel in the frame to be processed; in the difference value weight mapping relationship, the larger the difference value of the pixel, the smaller the corresponding difference value weight.

[0101] A difference value weight mapping relationship may be preset in the electronic device. The above-mentioned difference value weight mapping relationship may include difference value weights corresponding to different difference values. In the above-mentioned difference value weight mapping relationship, the larger the difference value of the pixel, the smaller the corresponding difference value weight. In the above-mentioned difference value weight mapping relationship, the difference value may be linearly changed with the difference value weight, or may be nonlinearly changed; in addition, the above-mentioned difference value weight mapping relationship may also include difference value weights corresponding to each interval of the difference value. The electronic device may also preset a difference value threshold, and determine the difference value weight of each pixel according to the difference value threshold and the difference value weight mapping relationship. For example, when the difference value of the pixel is greater than the difference value threshold, the corresponding difference value weight is S1, and when the difference value of the pixel is less than or equal to the difference value threshold, the corresponding difference value weight is S2.

[0102] S404: Calculate a target weight for each pixel according to the saturation weight, brightness weight, and difference value weight of each pixel.

[0103] Finally, the calculated saturation weight Sat_wgt, brightness weight Lum_wgt and difference value weight Bdr_wgt are subjected to preset calculation to generate a target weight. Here, the preset budget may be a weighted average, or other types of calculation methods, which are not limited in this application.

[0104] The above-mentioned video image denoising method realizes the calculation of target weights from three dimensions: saturation weight, brightness weight and difference value weight. Therefore, the target weights calculated by the electronic device can take into account the characteristics of multiple dimensions, so that the target weights are subsequently used to perform time domain denoising on the brightness component of the reference frame and the brightness component of the spatial domain denoising frame, and the processing results obtained are more accurate.

[0105] Figure 7 FIG. 1 is a flow chart of a method for denoising a video image in another embodiment. This embodiment relates to a method for an electronic device to determine a weight of a difference value of a pixel. The frame to be processed corresponds to multiple reference frames. Based on the above embodiment, Figure 7 As shown, the electronic device calculates the boundary map in the following manner:

[0106] S501 , performing difference calculation on the brightness component of the spatial denoising frame and the brightness component of each reference frame respectively, to obtain a difference image corresponding to each reference frame.

[0107] The electronic device can select the brightness components RefFrmY of multiple reference frames and the brightness components SnrFrmY of the spatial denoising frame of the image to be processed to perform differential operations to obtain differential images corresponding to each reference frame. The multiple differential images obtained by the electronic device are differential images difFrm with equal resolution.

[0108] S502 . For each pixel, extract the maximum pixel value in each differential image to obtain a maximum differential image.

[0109] Furthermore, the electronic device may superimpose a plurality of differential images difFrm, and then extract the maximum pixel value of each pixel to obtain a maximum differential image difMaxFrm.

[0110] S503: Obtain a local maximum image block in the maximum difference image according to a preset size.

[0111] The preset size may be 4×4 or other values, which is not limited in this application. The electronic device may calculate the block maximum value according to the preset size and obtain the local maximum image block difMaxFrm4×4 in the maximum differential image.

[0112] S504: Filter and dilate the local maximum image block to obtain a boundary map corresponding to each reference frame.

[0113] The electronic device may use a smoothing filter to filter the local maximum image block difMaxFrm4×4, for example, the dimension of the smoothing filter may be 5×5, to obtain a filtered differential frame, and then perform morphological dilation processing on the differential frame to obtain a boundary map bdr_map.

[0114] Further, the electronic device determines the difference value weight based on the boundary map, including:

[0115] S505 . For each pixel, extract an initial difference value corresponding to the pixel from a boundary map corresponding to each reference frame.

[0116] S506: Determine the maximum value of each initial difference value, or the average value of each initial difference value, as the target difference value of the pixel.

[0117] Among them, the formula for finding the maximum difference of each pixel is as follows:

[0118]

[0119] The formula for calculating the weighted average of the differences of each pixel is as follows:

[0120]

[0121] Among them, i and j are the position coordinates of the pixel on the current bdr_map.

[0122] S507 , determining the difference value weights of the pixels in the frame to be processed according to the target difference value and the difference value weight mapping relationship.

[0123] Furthermore, the electronic device may determine the difference value weights of the pixels in the frame to be processed according to the target difference value and the difference value weight mapping relationship.

[0124] In the above-mentioned video image denoising method, when each frame to be processed corresponds to multiple reference frames, the electronic device generates a boundary mapping map for each reference frame, and determines the target difference value of the pixel based on the multiple boundary mapping maps, so that the above-mentioned target difference value can more accurately reflect the difference between the Y component of the spatial denoising frame and the Y component of the reference frame, so that the obtained difference value weight is more accurate.

[0125] Figure 8 FIG. 1 is a flow chart of a method for denoising a video image in another embodiment. This embodiment relates to a method in which an electronic device determines a frame to be processed based on a video to be processed. Based on the above embodiment, Figure 8 As shown, the above method also includes:

[0126] S601: Determine a plurality of basic segments in a video to be processed; a set of time periods of the plurality of basic segments includes a time period of the video to be processed.

[0127] The basic segment is a sequence segment in the video to be processed. The video to be processed may correspond to multiple basic segments. The set of time periods of the multiple basic segments may include the time period of the video to be processed, that is, the video frames in the multiple basic segments include all the video frames in the video to be processed. The multiple basic segments may be adjacent in time to each other, or there may be overlapping video frames in two adjacent basic segments.

[0128] The electronic device may extract multiple basic segments from the video to be processed at preset intervals; or may randomly extract multiple basic segments from the video to be processed, as long as the set of time periods of the basic segments includes the time period of the video to be processed.

[0129] S602: For each basic segment, determine each video frame in the forward segment and the reverse segment of the basic segment as a frame to be processed, and perform denoising on the frame to be processed.

[0130] After determining a plurality of basic segments, the electronic device may arrange the video frames of each basic segment in reverse order to obtain reverse segment of each basic segment. The electronic device may determine each video frame in the forward segment and the reverse segment corresponding to each basic segment as a frame to be processed. Further, the electronic device may perform denoising on each frame to be processed.

[0131] In the above-mentioned video image denoising method, the electronic device performs segmented processing on the video to be processed, and can process the basic segments as a group, which can simplify the process of generating reverse-order segments, and at the same time can simplify the process of determining the time-matched video frames during and after the denoising process, thereby improving the denoising efficiency of the video to be processed, thereby achieving a real-time denoising effect.

[0132] Fig. 9 FIG. 1 is a flow chart of a method for denoising a video image in another embodiment. This embodiment relates to a method for an electronic device to determine a basic segment. Based on the above embodiment, Fig. 9 As shown, the above method also includes:

[0133] S701: Copy a video to be processed to obtain at least one copied video.

[0134] When determining the basic segment, the electronic device may first copy the video to be processed to obtain at least one copied video. The electronic device may mark the video to be processed and the copied video to distinguish different videos.

[0135] S702: Segment the video to be processed and the copied video according to different segmentation methods to obtain multiple video segments corresponding to each segmentation method.

[0136] Furthermore, the electronic device may adopt different segmentation methods to segment each video. The above segmentation methods may include at least one of the following:

[0137] The first segmentation method is to segment the video to be processed or the copied video at equal time intervals. The equal time intervals can be the video duration or the number of video frames; the electronic device can segment the video to be processed or the copied video into multiple video segments according to the preset time intervals.

[0138] The second segmentation method is to segment the video to be processed or the copied video according to content similarity. The electronic device can calculate the content similarity of adjacent video frames based on the image features of each video frame; for example, the more pixels with the same pixel values, the higher the content similarity of the adjacent video frames. The electronic device can segment the video to be processed into multiple video segments based on content similarity, so that the content of the video frames in each video segment is similar.

[0139] The third segmentation method is to segment the video to be processed or the copied video according to the similarity of the noise magnitude. The electronic device can calculate the noise magnitude of each video frame, cluster the video frames based on the noise magnitude of the video frames, and segment the video to be processed into multiple video segments.

[0140] It should be noted that the segmentation method in this embodiment may also be any combination of the above-mentioned multiple segmentation methods, which is not limited here.

[0141] S703: Select a basic segment from each video segment according to the time period of each video segment.

[0142] After obtaining multiple video segments corresponding to each segmentation method, the electronic device can select basic segments from each video segment according to the time period of the video segment. For example, the electronic device can randomly select multiple video segments from each video segment, and if the set of time periods of the multiple video segments can include the time period of the video to be processed, the multiple video segments can be determined as multiple basic segments.

[0143] In one implementation, the electronic device selects the basic segment according to whether there is a corresponding reference segment, wherein the time period of the reference segment needs to include the time period of the basic segment.

[0144] If the video segments of the video to be processed have corresponding reference segments in each video segment, the video segment is determined as the basic segment.

[0145] If there is no corresponding reference segment in each video segment of the video segment to be processed, a basic segment is selected from the video segment corresponding to the copied video; the time period of the basic segment includes the time period of the video segment.

[0146] by Fig.10 For example, the electronic device copies the processed video (video A) to obtain a copied video (video B). The electronic device uses an equal time interval segmentation method to segment video A to obtain in-order segments A1 and in-order segments A2; and uses a content similarity segmentation method to split and combine video B to obtain in-order segments B1, in-order segments B2, and in-order segments B3. Further, the electronic device obtains reverse segments of each in-order segment, including: reverse segment A1, reverse segment A2, reverse segment B1, reverse segment B2, and reverse segment B3. In video A, in-order segment A1 has a corresponding reference segment in-order segment B1; while in-order segment A2 does not have a corresponding reference segment. Further, the electronic device selects in-order segments B2 and in-order segments B3 as basic segments in video B, and the corresponding reference segments are all in-order segments A2.

[0147] Furthermore, after determining multiple basic segments, the electronic device can select a reference segment corresponding to each basic segment from each video segment for each basic segment; then, each video frame in the normal segment of the basic segment, the reverse segment of the basic segment, the normal segment of the reference segment, and the reverse segment of the reference segment is determined as a frame to be processed. It should be noted that the same basic segment can correspond to one or more reference segments, and the number of reference segments corresponding to different basic segments can be the same or different.

[0148] In the above-mentioned video image denoising method, the electronic device copies the video to be processed and segments it in different segmentation methods, so that the reference segment of each basic segment can be determined in the obtained video segment; since the basic segment and the reference segment contain different numbers of video frames, when the reverse sequence segments of the basic segment and the reference segment are denoised, the denoising results of the reference frame corresponding to the same video frame are different, so that the reverse sequence videos of the basic segment and the reference segment can obtain different denoising effects, further improving the denoising effect of the video to be processed.

[0149] Fig.11 FIG. 1 is a flow chart of a method for denoising a video image in another embodiment. This embodiment relates to a method for electronic equipment to perform fusion processing on denoising results. Based on the above embodiment, Fig.11 As shown, the above S102 includes:

[0150] S801. For each video frame in the base segment, extract an initial denoised frame that matches the time of the video frame in the base segment from the denoising results corresponding to the forward segment of the base segment, the denoising results corresponding to the reverse segment of the base segment, the denoising results corresponding to the forward segment of the reference segment, and the denoising results corresponding to the reverse segment of the reference segment.

[0151] For each basic segment, the electronic device can obtain a denoising result combination corresponding to the basic segment, including a denoising result corresponding to a forward segment of the basic segment, a denoising result corresponding to a reverse segment of the basic segment, a denoising result corresponding to a forward segment of the reference segment, and a denoising result corresponding to a reverse segment of the reference segment. Further, the electronic device can determine, for each video frame in the basic segment, a plurality of initial denoising frames that match the time of the video frame in the above denoising result combination.

[0152] S802: Perform a first fusion process on each initial denoised frame to obtain a target denoised frame corresponding to the video frame in the basic segment.

[0153] Furthermore, the electronic device may perform a first fusion process on the multiple initial denoised frames to obtain a target denoised frame corresponding to the video frame.

[0154] When the electronic device performs the first fusion processing on each initial denoised frame, the fusion may be performed in units of frames, pixels, or image blocks in the denoised frame, which is not limited here.

[0155] Optionally, the electronic device may average the pixel values ​​of the pixels at the same position in each initial denoised frame to obtain a target denoised frame of the basic video frame. Alternatively, the electronic device may perform weighted summation of the pixel values ​​of the pixels at the same position in each initial denoised frame according to the fusion weight of each pixel to obtain a target denoised frame of the basic video frame; wherein the pixel fusion weight is related to the weight of the pixel gradient or the weight of the pixel ghost.

[0156] Taking the correlation between the fusion weight and the pixel gradient weight as an example, the electronic device can calculate the pixel gradient of each pixel in the video frame of the basic segment; further, according to the mapping relationship between the pixel gradient and the gradient fusion weight, the gradient fusion weight corresponding to each pixel is determined. In the above gradient fusion weight mapping relationship, the larger the pixel gradient, the larger the corresponding gradient fusion weight; conversely, the smaller the pixel gradient, the smaller the corresponding gradient fusion weight.

[0157] Taking the correlation between the fusion weight and the pixel ghost weight as an example, the electronic device can detect the pixel ghost of each pixel in the video frame of the basic clip; further, according to the mapping relationship between the pixel ghost and the ghost fusion weight, the ghost fusion weight corresponding to each pixel is determined. In the above ghost fusion weight mapping relationship, the larger the pixel ghost, the smaller the corresponding ghost fusion weight; conversely, the smaller the pixel ghost, the larger the corresponding ghost fusion weight.

[0158] It should be noted that the first fusion processing mentioned above may also be a combination of the above-mentioned multiple fusion methods.

[0159] S803 , arranging the target denoised frames of the basic segment in positive order to obtain a denoised segment of the basic segment.

[0160] After obtaining the target denoised frames corresponding to the respective video frames of the basic segment, the electronic device may arrange the respective target denoised frames in positive order to obtain the denoised segment of the basic segment.

[0161] S804: performing splicing processing on the denoised segments corresponding to the basic segments to obtain a denoised video corresponding to the video to be processed.

[0162] Furthermore, after obtaining the denoised segments of each basic segment, the electronic device may perform splicing processing on each denoised segment to obtain a denoised video of the video to be processed.

[0163] The electronic device can splice the denoised segments according to the time periods of the denoised segments. If the time periods of two adjacent denoised segments do not overlap, the two denoised segments are directly spliced; the electronic device can add the latter denoised segment to the former denoised segment, or copy the two denoised segments to generate a new denoised segment; the above splicing method is not limited here.

[0164] If two adjacent denoised segments overlap, the overlapping segments are subjected to a second fusion process. For the overlapping segments, the electronic device can group two time-matched target denoised frames together, and then perform a second fusion process on each group of target denoised frames. The specific limitations of the above-mentioned second fusion process are similar to those of the above-mentioned first fusion process, and are not limited here. It should be noted that the above-mentioned second fusion processing method may be the same as or different from the first fusion processing method.

[0165] In the above-mentioned video image denoising method, the electronic device performs denoising processing on the forward segment of the basic segment, the reverse segment of the basic segment, the forward segment of the reference segment and the reverse segment of the reference segment respectively, so that the same video frame of the basic segment can correspond to different initial denoising frames; further, by fusing multiple initial denoising frames, the target denoising frame corresponding to the video frame can include the positive denoising effects of each initial denoising frame, and reduce the negative denoising effects in each initial denoising frame, thereby further improving the denoising effect of the target denoising frame, thereby improving the denoising effect of the denoised video.

[0166] In one embodiment, a method for denoising a video image is provided. Based on the above embodiment, Fig.12 As shown, the above method includes:

[0167] S901, copying the video to be processed to obtain at least one copied video;

[0168] S902, segmenting the video to be processed and the copied video respectively according to different segmentation methods, and obtaining a plurality of video segments corresponding to each segmentation method;

[0169] S903, selecting a basic segment from each video segment according to the time period of each video segment;

[0170] S904: For each basic segment, select a reference segment corresponding to the basic segment from each video segment;

[0171] S905, determining each video frame in the normal sequence segment of the base segment, the reverse sequence segment of the base segment, the normal sequence segment of the reference segment, and the reverse sequence segment of the reference segment as a frame to be processed;

[0172] S906, performing spatial domain denoising processing on the chrominance component of the frame to be processed to obtain the chrominance component of the spatial domain denoised frame of the frame to be processed;

[0173] S907, performing spatial domain denoising processing on the brightness component of the frame to be processed to obtain a brightness component of a spatial domain denoised frame corresponding to the frame to be processed;

[0174] S908, using the target weight of each pixel, weightedly fuse the brightness component of the reference frame of the frame to be processed and the brightness component of the spatial domain denoised frame to obtain the brightness component of the temporal domain denoised frame corresponding to the frame to be processed;

[0175] S909, calculating the texture size of each pixel in the brightness component of the frame to be processed;

[0176] S910, determining a texture fusion weight value of each pixel in the frame to be processed according to the texture size and a preset texture weight mapping relationship;

[0177] S911, based on the texture fusion weight value, fuse the brightness component of the time-domain denoised frame with the brightness component of the frame to be processed to obtain the brightness component of the denoised output frame of the frame to be processed;

[0178] S912, combining the luminance component of the denoised output frame with the chrominance component of the spatial domain denoised frame to obtain a denoised output frame of the frame to be processed;

[0179] S913, for each video frame in the base segment, respectively extract an initial denoised frame that matches the time of the video frame in the base segment from the denoising results corresponding to the forward segment of the base segment, the denoising results corresponding to the reverse segment of the base segment, the denoising results corresponding to the forward segment of the reference segment, and the denoising results corresponding to the reverse segment of the reference segment;

[0180] S914, performing a first fusion process on each initial denoised frame to obtain a target denoised frame corresponding to the video frame in the basic segment;

[0181] S915, arranging the target denoised frames of the basic segment in positive order to obtain a denoised segment of the basic segment;

[0182] S916: Splice the denoised segments corresponding to the basic segments to obtain a denoised video corresponding to the video to be processed.

[0183] The implementation principle and technical effect of the above-mentioned video image denoising method are similar to those of the above-mentioned embodiment and will not be described in detail here.

[0184] It should be understood that although Figure 1-Figure 12The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-12 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0185] Fig.13 FIG. 1 is a structural block diagram of a video image denoising device according to an embodiment of the present invention. Fig.13 As shown, the above device comprises:

[0186] The denoising module 10 is used to perform denoising on each frame to be processed in the forward sequence video and the reverse sequence video of the processed video; during the denoising process, the denoising result of the frame to be processed is related to the denoising result of the reference frame; the reference frame is one or more video frames located before the frame to be processed;

[0187] The fusion module 20 is used to fuse the denoising result of the normal-sequence video and the denoising result of the reverse-sequence video to obtain a denoised video corresponding to the video to be processed.

[0188] In one embodiment, based on the above embodiment, Fig.14 As shown, the denoising module 10 comprises:

[0189] The determining unit 101 is used to determine a plurality of basic segments in the video to be processed; the set of time periods of the plurality of basic segments includes the time period of the video to be processed;

[0190] The denoising unit 102 is configured to determine, for each basic segment, each video frame in the forward segment and the reverse segment of the basic segment as a frame to be processed, and perform denoising on the frame to be processed.

[0191] In one embodiment, based on the above embodiment, Fig.15 As shown, the above-mentioned determination unit 101 includes:

[0192] A copy subunit 1011 is used to copy the video to be processed to obtain at least one copy video;

[0193] The segmentation subunit 1012 is used to segment the video to be processed and the copy video according to different segmentation methods, and obtain multiple video segments corresponding to each segmentation method;

[0194] The selection subunit 1013 is used to select a basic segment from each video segment according to the time period of each video segment.

[0195] In one embodiment, based on the above embodiment, the above selection subunit 1013 is specifically used for: when the video segment of the video to be processed has a corresponding reference segment in each video segment, the video segment of the video to be processed is determined as the basic segment; the time period of the reference segment includes the time period of the basic segment.

[0196] In one embodiment, based on the above embodiment, the above selection subunit 1013 is also used for: when there is no corresponding reference segment in each video segment of the video segment to be processed, a basic segment is selected from the video segment corresponding to the copied video; the time period of the basic segment includes the time period of the video segment.

[0197] In one embodiment, based on the above embodiment, the denoising unit 102 is specifically used to: for each basic segment, select a reference segment corresponding to the basic segment in each video segment; determine each video frame in the positive segment of the basic segment, the reverse segment of the basic segment, the positive segment of the reference segment, and the reverse segment of the reference segment as a frame to be processed.

[0198] In one embodiment, based on the above embodiment, Fig.16 As shown, the above fusion module 20 includes:

[0199] The extraction unit 201 is used to extract, for each video frame in the base segment, an initial denoised frame that matches the time of the video frame in the base segment from the denoising results corresponding to the forward segment of the base segment, the denoising results corresponding to the reverse segment of the base segment, the denoising results corresponding to the forward segment of the reference segment, and the denoising results corresponding to the reverse segment of the reference segment;

[0200] A fusion unit 202 is used to perform a first fusion process on each initial denoised frame to obtain a target denoised frame corresponding to the video frame in the basic segment;

[0201] An arrangement unit 203 is used to arrange the target denoised frames of the basic segment in positive order to obtain a denoised segment of the basic segment;

[0202] The splicing unit 204 is used to splice the denoised segments corresponding to the basic segments to obtain a denoised video corresponding to the video to be processed.

[0203] In one embodiment, based on the above embodiment, the fusion unit 202 is specifically used to: average the pixel values ​​of pixels at the same position in each initial denoised frame to obtain a target denoised frame of the basic video frame; or, according to the fusion weight of each pixel, perform weighted summation on the pixel values ​​of pixels at the same position in each initial denoised frame to obtain a target denoised frame of the basic video frame; wherein the pixel fusion weight is related to the weight of the pixel gradient or the weight of the pixel ghost.

[0204] In one embodiment, based on the above embodiment, the splicing unit 204 is specifically configured to: if two adjacent denoised segments overlap, perform a second fusion process on the overlapping segment parts.

[0205] In one embodiment, based on the above embodiment, the segmentation method includes at least one of the following: segmenting the video to be processed or the copied video at equal time intervals; segmenting the video to be processed or the copied video according to content similarity; segmenting the video to be processed or the copied video according to the similarity of noise size; wherein the segmentation method of the video to be processed and the copied video is different.

[0206] In one embodiment, based on the above embodiment, Fig.17 As shown, the denoising unit 102 includes:

[0207] The spatial denoising subunit 1021 is used to perform spatial denoising on the chrominance component of the frame to be processed to obtain the chrominance component of the spatial denoising frame of the frame to be processed; perform spatial denoising on the luminance component of the frame to be processed to obtain the luminance component of the spatial denoising frame corresponding to the frame to be processed;

[0208] The temporal denoising subunit 1022 is used to perform weighted fusion of the brightness component of the reference frame of the frame to be processed and the brightness component of the spatial denoising frame using the target weight of each pixel to obtain the brightness component of the temporal denoising frame corresponding to the frame to be processed; the target weight is related to the saturation, brightness and difference value of the pixel; the difference value is used to characterize the degree of difference between the frame to be processed and the reference frame;

[0209] The denoising output subunit 1023 is used to obtain a denoising output frame of the frame to be processed according to the luminance component of the temporal denoising frame and the chrominance component of the spatial denoising frame.

[0210] In one embodiment, based on the above embodiment, Fig.18 As shown, the denoising unit 102 also includes a weight calculation subunit 1024, which is used to determine the target weight of each pixel according to the first brightness component, the first chrominance component and the brightness component of the reference frame; wherein the first brightness component includes the brightness component of the frame to be processed or the brightness component of the spatial denoising frame; the first chrominance component includes the chrominance component of the frame to be processed or the chrominance component of the spatial denoising frame.

[0211] In one embodiment, based on the above embodiment, the weight calculation subunit 1024 is specifically used to: calculate the saturation weight of each pixel in the frame to be processed according to the first brightness component and the first chrominance component; calculate the brightness weight of each pixel in the frame to be processed according to the first brightness component; calculate the difference value weight of each pixel in the frame to be processed according to the brightness component of the spatial denoising frame and the brightness component of the reference frame; calculate the target weight of each pixel according to the saturation weight, brightness weight and difference value weight of each pixel.

[0212] In one embodiment, based on the above embodiment, the weight calculation subunit 1024 is specifically used to: perform color gamut mapping on the first brightness component and the first chrominance component to obtain the saturation of each pixel in the frame to be processed; calculate the saturation weight of each pixel in the frame to be processed according to the saturation of each pixel and a preset saturation weight mapping relationship; in the saturation weight mapping relationship, the greater the saturation of the pixel, the smaller the corresponding saturation weight.

[0213] In one embodiment, based on the above embodiment, the weight calculation unit 1024 is specifically used to: generate a brightness mapping map of the frame to be processed according to the first brightness component; the brightness mapping map includes the brightness value of each pixel; according to the brightness mapping map and a preset brightness weight mapping relationship, calculate the brightness weight of each pixel in the frame to be processed; in the brightness weight mapping relationship, the larger the brightness value of the pixel, the smaller the corresponding brightness weight.

[0214] In one embodiment, based on the above embodiment, the weight calculation subunit 1024 is specifically used to: perform differential calculation on the brightness component of the spatial denoising frame and the brightness component of the reference frame to obtain a boundary mapping map; the boundary mapping map includes the difference values ​​of each pixel; according to the boundary mapping map and a preset difference value weight mapping relationship, determine the difference value weight of each pixel in the frame to be processed; in the difference value weight mapping relationship, the larger the difference value of the pixel, the smaller the corresponding difference value weight.

[0215] In one embodiment, based on the above embodiment, the weight calculation subunit 1024 is specifically used to: perform differential calculation on the brightness component of the spatial denoising frame and the brightness component of each reference frame to obtain a differential image corresponding to each reference frame; for each pixel, extract the maximum pixel value in each differential image to obtain a maximum differential image; obtain the local maximum image block in the maximum differential image according to a preset size; filter and dilate the local maximum image block to obtain a boundary mapping image corresponding to each reference frame.

[0216] In one embodiment, based on the above embodiment, the weight calculation subunit 1024 is specifically used to: for each pixel, extract the initial difference value corresponding to the pixel in the boundary mapping map corresponding to each reference frame; determine the maximum value of each initial difference value, or the average value of each initial difference value as the target difference value of the pixel; determine the difference value weight of the pixel in the frame to be processed according to the target difference value and the difference value weight mapping relationship.

[0217] In one embodiment, based on the above embodiment, the time domain denoising subunit 1022 is specifically used to: Get the brightness component F of the time domain denoised frame of the frame to be processed i,j ; Among them, S i,j Represents the brightness component of pixel (i, j) in the spatial denoising frame; R i,j Represents the brightness component of pixel (i, j) in the reference frame; jnt i,j Represents the target weight of pixel (i,j).

[0218] In one embodiment, based on the above embodiment, the denoising output subunit 1023 is specifically used to: calculate the texture size of each pixel in the brightness component of the frame to be processed; determine the texture fusion weight value of each pixel in the frame to be processed according to the texture size and a preset texture weight mapping relationship; in the texture weight mapping relationship, the larger the texture size of the pixel, the larger the corresponding texture fusion weight value; based on the texture fusion weight value, fuse the brightness component of the time domain denoised frame with the brightness component of the frame to be processed to obtain the brightness component of the denoised output frame of the frame to be processed; merge the brightness component of the denoised output frame with the chrominance component of the spatial domain denoised frame to obtain the denoised output frame of the frame to be processed.

[0219] The division of the various modules in the above-mentioned video image denoising device is only for illustration. In other embodiments, the video image denoising device may be divided into different modules as needed to complete all or part of the functions of the above-mentioned video image denoising device.

[0220] For the specific definition of the denoising device for video images, please refer to the definition of the denoising method for video images above, which will not be repeated here. Each module in the above-mentioned denoising device for video images can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0221] Fig.19 FIG. 1 is a schematic diagram of the internal structure of an electronic device in one embodiment. Fig.19As shown, the electronic device includes a processor and a memory connected via a system bus. The processor is used to provide computing and control capabilities to support the operation of the entire electronic device. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by the processor to implement a video image denoising method provided in the following embodiments. The internal memory provides a cache operating environment for the operating system computer program in the non-volatile storage medium. The electronic device can be any terminal device such as a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), a vehicle-mounted computer, a wearable device, etc.

[0222] The implementation of each module in the video image denoising device provided in the embodiment of the present application can be in the form of a computer program. The computer program can be run on a terminal or a server. The program module constituted by the computer program can be stored in the memory of an electronic device. When the computer program is executed by a processor, the steps of the method described in the embodiment of the present application are implemented.

[0223] The embodiment of the present application also provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, the processors execute the steps of the method for denoising a video image.

[0224] A computer program product comprising instructions, when executed on a computer, causes the computer to perform a method for denoising a video image.

[0225] Any reference to memory, storage, database or other medium used in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0226] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for denoising a video image, characterized in that: include: Copy the video to be processed to obtain at least one copied video; According to different segmentation methods, the video to be processed and the copied video are segmented and processed respectively to obtain a plurality of video segments corresponding to each segmentation method; According to the time period of each of the video segments, a basic segment is selected from each of the video segments, and a set of time periods of multiple basic segments includes the time period of the video to be processed; For each basic segment, each video frame in the forward segment and the reverse segment of the basic segment is determined as a frame to be processed in the forward video and the reverse video of the video to be processed, and denoising is performed on the frame to be processed, wherein the denoising result of the frame to be processed is correlated with the denoising result of the reference frame during the denoising process; The reference frame is one or more video frames located before the frame to be processed; The denoising result of the forward-sequence video and the denoising result of the reverse-sequence video are fused to obtain a denoised video corresponding to the video to be processed.

2. The method according to claim 1, characterized in that The selecting the basic segment from each of the video segments according to the time period of each of the video segments includes: If the video segment of the video to be processed has a corresponding reference segment in each video segment, the video segment of the video to be processed is determined as the basic segment; and the time period of the reference segment includes the time period of the basic segment.

3. The method according to claim 2, characterized in that The method further comprises: If there is no corresponding reference segment in each video segment of the video segment to be processed, a basic segment is selected from the video segment corresponding to the copied video; the time period of the basic segment includes the time period of the video segment.

4. The method according to claim 2, characterized in that: For each basic segment, determining each video frame in the forward segment and the reverse segment of the basic segment as a to-be-processed frame in the forward video and the reverse video of the to-be-processed video includes: For each basic segment, selecting a reference segment corresponding to the basic segment from among the video segments; Each video frame in the forward segment of the base segment, the reverse segment of the base segment, the forward segment of the reference segment, and the reverse segment of the reference segment is determined as the frame to be processed.

5. The method according to claim 4, characterized in that The denoising result of the forward-order video and the denoising result of the reverse-order video are fused to obtain a denoised video corresponding to the video to be processed, including: For each video frame in the base segment, respectively extract an initial denoised frame that matches the time of the video frame in the base segment from the denoising results corresponding to the forward segment of the base segment, the denoising results corresponding to the reverse segment of the base segment, the denoising results corresponding to the forward segment of the reference segment, and the denoising results corresponding to the reverse segment of the reference segment; Performing a first fusion process on each of the initial denoised frames to obtain a target denoised frame corresponding to the video frame in the basic segment; Arranging the target denoised frames of the basic segment in positive order to obtain a denoised segment of the basic segment; The denoised segments corresponding to the basic segments are spliced ​​together to obtain a denoised video corresponding to the video to be processed.

6. The method according to claim 5, characterized in that The performing a first fusion process on each of the initial denoised frames to obtain a target denoised frame corresponding to the video frame in the basic segment includes: Averaging pixel values ​​of pixels at the same position in each of the initial denoised frames to obtain the target denoised frame; or, According to the fusion weight of each pixel, the pixel values ​​of the pixels at the same position in each of the initial denoised frames are weighted and summed to obtain the target denoised frame; wherein the pixel fusion weight is related to the weight of the pixel gradient or the weight of the pixel ghost.

7. The method according to claim 5, characterized in that The step of splicing the denoised segments corresponding to the basic segments to obtain the denoised video corresponding to the video to be processed includes: If two adjacent denoised segments overlap, the second fusion process is performed on the overlapping segment parts.

8. The method according to claim 1, characterized in that The segmentation method includes at least one of the following: Segmenting the to-be-processed video or the copied video at equal time intervals; Segmenting the video to be processed or the copied video according to content similarity; Segmenting the video to be processed or the copied video according to the similarity of the noise magnitude; The to-be-processed video and the copied video are segmented in different ways.

9. The method according to any one of claims 1 to 8, characterized in that: The denoising process is performed on the frame to be processed, comprising: Performing spatial domain denoising processing on the chrominance component of the frame to be processed to obtain the chrominance component of the spatial domain denoised frame of the frame to be processed; Performing spatial domain denoising processing on the brightness component of the frame to be processed to obtain a brightness component of a spatial domain denoised frame corresponding to the frame to be processed; Using the target weight of each pixel, the brightness component of the reference frame of the frame to be processed and the brightness component of the spatial domain denoised frame are weightedly fused to obtain the brightness component of the temporal domain denoised frame corresponding to the frame to be processed; the target weight is related to the saturation, brightness and difference value of the pixel; the difference value is used to characterize the degree of difference between the frame to be processed and the reference frame; A denoised output frame of the frame to be processed is obtained according to the luminance component of the temporal denoised frame and the chrominance component of the spatial denoised frame.

10. The method according to claim 9, characterized in that The method further comprises: The target weight of each pixel is determined according to the first luminance component, the first chrominance component and the luminance component of the reference frame; wherein the first luminance component includes the luminance component of the frame to be processed or the luminance component of the spatial denoising frame; and the first chrominance component includes the chrominance component of the frame to be processed or the chrominance component of the spatial denoising frame.

11. The method according to claim 10, characterized in that The determining the target weight of each pixel according to the first brightness component, the chrominance component of the frame to be processed, and the brightness component of the reference frame includes: Calculating a saturation weight of each pixel in the frame to be processed according to the first brightness component and the first chrominance component; Calculating the brightness weight of each pixel in the frame to be processed according to the first brightness component; Calculating a difference value weight of each pixel in the frame to be processed according to a brightness component of the spatial domain denoising frame and a brightness component of the reference frame; The target weight of each pixel is calculated according to the saturation weight, the brightness weight and the difference value weight of each pixel.

12. The method according to claim 11, characterized in that The step of calculating the saturation weight of each pixel in the frame to be processed according to the first brightness component and the first chrominance component includes: Performing color gamut mapping on the first brightness component and the first chrominance component to obtain the saturation of each pixel in the frame to be processed; The saturation weight of each pixel in the frame to be processed is calculated according to the saturation of each pixel and a preset saturation weight mapping relationship; in the saturation weight mapping relationship, the greater the saturation of the pixel, the smaller the corresponding saturation weight.

13. The method according to claim 11, characterized in that Calculating the brightness weight of each pixel in the frame to be processed according to the first brightness component includes: Generate a brightness map of the frame to be processed according to the first brightness component; the brightness map includes a brightness value of each pixel; According to the brightness mapping diagram and a preset brightness weight mapping relationship, the brightness weight of each pixel in the frame to be processed is calculated; in the brightness weight mapping relationship, the larger the brightness value of the pixel, the smaller the corresponding brightness weight.

14. The method according to claim 11, characterized in that The calculating the difference value weight of each pixel in the frame to be processed according to the brightness component of the spatial domain denoising frame and the brightness component of the reference frame comprises: Performing differential calculation on the brightness component of the spatial denoising frame and the brightness component of the reference frame to obtain a boundary map; the boundary map includes a difference value of each pixel; The difference value weight of each pixel in the frame to be processed is determined according to the boundary mapping diagram and a preset difference value weight mapping relationship; in the difference value weight mapping relationship, the larger the difference value of the pixel, the smaller the corresponding difference value weight.

15. The method according to claim 14, characterized in that The frame to be processed corresponds to a plurality of reference frames, and performing differential calculation on the brightness component of the spatial domain denoised frame and the brightness component of the reference frame to obtain a boundary mapping diagram includes: Performing differential calculations on the brightness component of the spatial domain denoised frame and the brightness component of each reference frame to obtain a differential image corresponding to each reference frame; For each pixel, extracting the maximum pixel value in each of the differential images to obtain a maximum differential image; Obtaining a local maximum image block in the maximum difference image according to a preset size; The local maximum image block is filtered and expanded to obtain a boundary map corresponding to each reference frame.

16. The method according to claim 15, characterized in that Determining the difference value weight of each pixel in the frame to be processed according to the boundary mapping map and a preset difference value weight mapping relationship includes: For each pixel, extracting an initial difference value corresponding to the pixel from a boundary map corresponding to each reference frame; Determine the maximum value of each of the initial difference values, or the average value of each of the initial difference values, as the target difference value of the pixel; The difference value weights of the pixels in the frame to be processed are determined according to the target difference value and the difference value weight mapping relationship.

17. The method according to claim 9, characterized in that The method adopts the target weight of each pixel to perform weighted fusion on the brightness component of the reference frame of the frame to be processed and the brightness component of the spatial domain denoised frame to obtain the brightness component of the temporal domain denoised frame corresponding to the frame to be processed, including: According to the formula Obtain the brightness component F of the time domain denoised frame of the frame to be processed i,j ; Among them, S i,j Represents the brightness component of pixel (i, j) in the spatial denoising frame; R i,j Represents the brightness component of pixel (i, j) in the reference frame; jnt i,j Represents the target weight of pixel (i,j).

18. The method according to claim 9, characterized in that The step of obtaining the denoised output frame of the frame to be processed according to the luminance component of the temporal denoised frame and the chrominance component of the spatial denoised frame comprises: Calculating the texture size of each pixel in the brightness component of the frame to be processed; Determining the texture fusion weight value of each pixel in the frame to be processed according to the texture size and a preset texture weight mapping relationship; in the texture weight mapping relationship, the larger the texture size of the pixel, the larger the corresponding texture fusion weight value; Based on the texture fusion weight value, the brightness component of the time-domain denoised frame is fused with the brightness component of the frame to be processed to obtain the brightness component of the denoised output frame of the frame to be processed; The luminance component of the denoised output frame is combined with the chrominance component of the spatial domain denoised frame to obtain the denoised output frame of the frame to be processed.

19. A video image denoising device, characterized in that: include: A denoising module is used to copy the video to be processed to obtain at least one copied video; segment the video to be processed and the copied video according to different segmentation methods to obtain multiple video segments corresponding to each segmentation method; select a basic segment from each video segment according to the time period of each video segment, and the set of time periods of multiple basic segments includes the time period of the video to be processed; for each basic segment, determine each video frame in the forward sequence segment and the reverse sequence segment of the basic segment as the frame to be processed in the forward sequence video and the reverse sequence video of the video to be processed, and perform denoising on the frame to be processed, and in the denoising process, the denoising result of the frame to be processed is related to the denoising result of the reference frame; The reference frame is one or more video frames located before the frame to be processed; The fusion module is used to fuse the denoising result of the normal-sequence video and the denoising result of the reverse-sequence video to obtain a denoised video corresponding to the video to be processed.

20. An electronic device, comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the video image denoising method according to any one of claims 1 to 18.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 18 are implemented.

Images