Face image noise reduction method, device and equipment and readable storage medium

Abstract

The invention provides a face image noise reduction method and device, equipment and a readable storage medium. The method comprises the following steps: acquiring a current frame image with a human face; according to the face key points of the current frame image and the current reference frame image, performing deformation registration processing on the face area of the current reference frame image to obtain a deformed reference frame image; wherein the current reference frame image is used for providing face key points for deformation registration processing of the current frame image; the deformed reference frame image is matched with the face posture of the current frame image; and according to the deformed reference frame image, performing noise reduction processing on the current frame image to obtain a noise-reduced image of the current frame image. According to the embodiment of the invention, the noise reduction accuracy of the face image can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and to, but is not limited to, a method, apparatus, device, and readable storage medium for denoising facial images. Background Technology

[0002] Temporal denoising is a common processing method in video processing. Its principle is to achieve noise reduction by fusing two adjacent frames of images in the video. The two adjacent frames can refer to each other, and by the similarity between the two adjacent frames, it can be determined which information in the picture is real and which is noise. Single-frame denoising cannot distinguish between noise and real information in the image. Therefore, temporal denoising has a noise reduction effect that is difficult to achieve with single-frame image denoising.

[0003] In related technologies, temporal noise reduction is usually achieved through optical flow or block matching algorithms. However, when using the above methods, the accuracy of finding corresponding pixels in previous and next frames is not high. In actual application scenarios, ghosting is very likely to occur when the screen is moving, resulting in poor temporal noise reduction effect. Summary of the Invention

[0004] This application provides a method, apparatus, device, and readable storage medium for denoising facial images, which can improve the accuracy of facial image denoising.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a method for denoising facial images, the method comprising:

[0007] Get the current frame image containing a face;

[0008] Based on the facial key points of the current frame image and the current reference frame image, deformation registration processing is performed on the facial region of the current reference frame image to obtain the deformed reference frame image; wherein, the current reference frame image is used to provide facial key points for the current frame image to perform the deformation registration processing; the deformed reference frame image is matched with the facial pose of the current frame image;

[0009] Based on the deformed reference frame image, the current frame image is denoised to obtain the denoised image of the current frame image.

[0010] Secondly, embodiments of this application provide a noise reduction device for facial images, the device comprising an acquisition unit, a processing unit, and a noise reduction unit; wherein,

[0011] The acquisition unit is used to acquire the current frame image containing a human face;

[0012] The processing unit is configured to perform deformation registration processing on the face region of the current reference frame image based on the face key points of the current frame image and the current reference frame image to obtain a deformed reference frame image; wherein, the current reference frame image is used to provide face key points for the current frame image to perform the deformation registration processing; the deformed reference frame image is matched with the face pose of the current frame image;

[0013] The noise reduction unit is used to perform noise reduction processing on the current frame image based on the deformed reference frame image to obtain a noise-reduced image of the current frame image.

[0014] Thirdly, embodiments of this application provide an electronic device, including:

[0015] Memory, used to store executable instructions;

[0016] The processor, when executing executable instructions stored in the memory, implements the above-described method for denoising facial images.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing executable instructions that, when executed by one or more processors, implement the face image noise reduction method described above.

[0018] This application provides a method, apparatus, device, and readable storage medium for denoising facial images. The method includes: acquiring a current frame image containing a face; performing deformation registration processing on the face region of the current reference frame image based on facial key points of the current frame image and a current reference frame image to obtain a deformed reference frame image; wherein the current reference frame image is used to provide facial key points for deformation registration processing of the current frame image; matching the facial pose of the deformed reference frame image with that of the current frame image; and performing denoising processing on the current frame image based on the deformed reference frame image to obtain a denoised image of the current frame image. On the one hand, deformation registration processing can align the face region of the current frame image with the face region of the previous frame reference frame image, maintaining a consistent facial pose. This helps eliminate pose changes between different frames, making subsequent processing more accurate. On the other hand, both the deformed reference frame image and the subsequent current frame image undergo denoising processing, which can reduce noise in the image, improve image quality, and help make facial features more clearly visible. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0020] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0021] Figure 1 A flowchart illustrating an optional face image noise reduction method provided in this application embodiment. Figure 1 ;

[0022] Figure 2 A flowchart illustrating an optional face image noise reduction method provided in this application embodiment. Figure 2 ;

[0023] Figure 3 A flowchart illustrating an optional face image noise reduction method provided in this application embodiment. Figure 3 ;

[0024] Figure 4 A schematic diagram illustrating the framework of an optional face image noise reduction method provided in an embodiment of this application;

[0025] Figure 5 A schematic diagram illustrating an optional deformation registration process provided in an embodiment of this application;

[0026] Figure 6 A schematic diagram illustrating an optional grid division of a face region provided in an embodiment of this application;

[0027] Figure 7 A schematic diagram illustrating an optional noise reduction process provided in an embodiment of this application;

[0028] Figure 8 A schematic diagram of an optional face image noise reduction device provided in an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of an optional electronic device provided in an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0032] In the following description, references to "some embodiments," "this embodiment," "this application embodiment," and examples, etc., describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subset of all possible embodiments and may be combined with each other without conflict.

[0033] In related technologies, temporal denoising is a technique to reduce or eliminate noise in a signal, primarily performed in the time domain. Existing temporal denoising front-ends use optical flow or block matching algorithms, which have low accuracy in finding corresponding pixels between frames. In practical applications, this easily leads to ghosting when the image is moving, resulting in poor temporal denoising performance in motion scenes.

[0034] The face image denoising method provided in this application embodiment is executed by a face image denoising device and an electronic device. The face image denoising device can be stored in the electronic device as a software functional model, or it can be integrated into the electronic device as a hardware functional module. The face image denoising device can also be combined with the electronic device in both software and hardware to realize the face image denoising method. This application does not impose any limitations on this.

[0035] In this application embodiment, the electronic device can be a server, which can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. This application embodiment does not limit this.

[0036] In addition, the electronic device can also be a terminal device, which can be a mobile phone, tablet personal computer (TPC), media player, smart TV, laptop computer (LC), personal digital assistant (PDA), personal computer (PC), camera, camcorder, smartwatch, wearable device (WD), or autonomous vehicle, etc. The embodiments of this application do not limit this.

[0037] This application provides a method for denoising face images. The basic idea of ​​this method is as follows: First, acquire a current frame image containing a face. Second, perform deformation registration processing on the face region of the current reference frame image based on facial key points from the current frame image and a current reference frame image to obtain a deformed reference frame image. The current reference frame image provides facial key points for deformation registration processing. Third, match the face pose of the deformed reference frame image with that of the current frame image. Finally, perform denoising processing on the current frame image based on the deformed reference frame image to obtain a denoised image of the current frame. On one hand, deformation registration processing aligns the face region of the current frame image with the face region of the previous reference frame image, maintaining a consistent face pose. This helps eliminate pose variations between different frames, making subsequent processing more accurate. On the other hand, both the deformed reference frame image and the subsequent current frame image undergo denoising processing, which reduces noise in the image, improves image quality, and helps make facial features more clearly visible.

[0038] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0039] Figure 1 A flowchart illustrating an optional face image noise reduction method provided in this application embodiment. Figure 1 ,like Figure 1 As shown, including S101 to S103:

[0040] S101. Obtain the current frame image containing a human face.

[0041] In this embodiment of the application, the electronic device acquires a current frame image containing a human face.

[0042] In the embodiments of this application, a current frame image with a face refers to an image containing one or more faces in real-time or still images.

[0043] In this embodiment of the application, the current frame image may be a real-time video frame captured by a camera, or it may be a frame in a still photograph.

[0044] In this embodiment of the application, the current frame image may also be obtained from a camera, scanner or other image capture device, or from other sources such as image libraries, the Internet, etc. This embodiment of the application does not limit the source of the current frame image.

[0045] S102. Based on the facial key points of the current frame image and the current reference frame image, perform deformation registration processing on the facial region of the current reference frame image to obtain the deformed reference frame image; wherein, the current reference frame image is used to provide facial key points for the current frame image to perform deformation registration processing; the deformed reference frame image is matched with the facial pose of the current frame image.

[0046] In the embodiments of this application, facial key points are marker points or coordinates at specific locations in a facial image, used to describe various features and expressions of the face.

[0047] In the embodiments of this application, facial key points include, but are not limited to: facial feature points, corner points; facial outer contour points, facial feature contour points, and eyebrow contour points.

[0048] For example, features of the eyes may include the positions of the inner and outer corners of the eyes and the pupil. Features of the nose may include the positions of the tip of the nose and the wings of the nose. Features of the mouth may include the positions of the upper and lower lip lines, the corners of the mouth, and the center of the mouth. Features of nose hairs may include the positions of the upper and lower edges of the eyebrows. Features of the facial contour may include the positions of the cheeks, chin, and the overall facial contour. The features of key points described above are merely examples; in specific application scenarios, other features may be included, and this application does not impose any limitations on them.

[0049] In this embodiment, the electronic device performs facial landmark detection on the current frame image and the current reference frame image using a preset facial landmark detection algorithm to obtain the facial landmarks of the current frame image and the current reference frame image.

[0050] In the embodiments of this application, the preset facial landmark detection algorithm includes, but is not limited to, the following methods: deep learning models and machine learning algorithms.

[0051] In the embodiments of this application, the deep learning model may include: Convolutional Neural Network (CNN), multi-task learning model, Hourglass network, etc.

[0052] For example, convolutional neural networks (CNNs) can learn features in images and perform keypoint detection by stacking multiple convolutional and fully connected layers. Examples include LeNet, AlexNet, VGG, and ResNet. Multi-task learning models are deep learning models that handle multiple tasks simultaneously. For face keypoint detection, tasks such as face detection and keypoint detection can be combined into a single model, sharing network layers and improving overall model performance. Multi-task Cascaded Convolutional Networks (MTCNN) is a typical example of a multi-task learning model. Hourglass networks are a special type of CNN architecture specifically designed for pose estimation and keypoint detection. Hourglass networks achieve high-precision keypoint detection by stacking multiple Hourglass modules, gradually refining and detailing features.

[0053] In this application embodiment, the machine learning algorithm may include: Support Vector Machine (SVM) algorithm, Random Forest Regression algorithm, etc. For example, the SVM algorithm can be used for feature classification and regression in facial keypoint detection. Typically, the coordinates of the keypoints are used as the target for regression; SVM can fit the keypoints by finding the optimal hyperplane. The Random Forest Regression algorithm is an ensemble learning method that can be used for keypoint detection; it predicts the location of keypoints by combining multiple decision trees.

[0054] In this embodiment, the electronic device performs deformation registration processing on the face region of the current reference frame image to obtain a deformed reference frame image, so that the deformed reference frame image is consistent with the face pose of the current frame image.

[0055] Specifically, first, the electronic device performs face detection and keypoint extraction on the current frame image and the current reference frame image to identify the location of face regions and keypoints, for example, using deep learning models (such as MTCNN, facial landmark detection networks, etc.). Second, after obtaining the facial keypoints of the current frame image and the current reference frame image, the electronic device can use deformation registration methods to process the reference frame image to make it consistent with the face pose of the current frame. Finally, after completing the deformation registration processing, the electronic device can perform noise reduction processing on the deformed reference frame image to improve image quality. This can include removing noise from the image, smoothing the image, or applying other noise reduction techniques.

[0056] In this embodiment, the Morphing Transform process is used to align an image or image region to match a reference image or reference template, i.e., the current reference frame image.

[0057] In this embodiment of the application, when the current reference frame image is obtained by denoising at least two previous frames of images with faces, the denoised images obtained by denoising at least two previous frames of images with faces can be fused together.

[0058] Understandably, the pose consistency between the deformed reference frame image and the current frame image can help the system more easily match facial features, improve accuracy, and improve the quality of the denoised image in subsequent noise reduction processing.

[0059] S103. Based on the deformed reference frame image, perform noise reduction processing on the current frame image to obtain the noise-reduced image of the current frame image.

[0060] In this embodiment of the application, after the electronic device obtains the deformed reference frame image, the electronic device performs noise reduction processing on the current frame image based on the deformed reference frame image to obtain the noise-reduced image of the current frame image.

[0061] In the embodiments of this application, the noise reduction processing methods include, but are not limited to: temporal noise reduction (TNR), frequency noise reduction (FNR), and deep learning noise reduction.

[0062] In this embodiment, temporal denoising refers to processing noise along the time axis of a signal or image. Common temporal denoising methods include sliding window averaging, median filtering, and Gaussian filtering. These methods smooth noise in a signal or image by performing operations within a local window on the time axis. Frequency domain denoising refers to converting a signal or image from the time domain to the frequency domain, and then processing the frequency domain representation to remove noise. Common frequency domain denoising methods include Fourier transform and wavelet transform. These methods typically convert the signal or image into a spectral representation and then filter out noise frequency components. Deep learning denoising can use network models such as convolutional neural networks (CNNs) and autoencoders to learn how to restore a clean image from a noisy image.

[0063] In some embodiments of this application, the implementation of denoising the current frame image based on the deformed reference frame image in S103 to obtain a denoised image of the current frame image may include: performing temporal fusion processing on the deformed reference frame image and the current frame image to obtain a denoised image of the current frame image, thereby realizing the denoising processing of the current frame image.

[0064] In this embodiment, the prerequisite for performing temporal fusion processing on the deformed reference frame image and the current frame image is that the size and resolution of the deformed reference frame image match those of the current frame image. After satisfying this prerequisite, the electronic device can select an appropriate temporal fusion method to combine the two images and reduce noise. Temporal fusion processing can be implemented using various methods, including weighted averaging, median filtering, and Gaussian filtering. Furthermore, the electronic device can combine the deformed reference frame image and the current frame image using the selected temporal fusion method by combining the values ​​of corresponding pixels, typically by adding or averaging the pixels.

[0065] In the embodiments of this application, after obtaining the denoised image, other post-processing techniques can be applied, such as at least one of sharpening, contrast enhancement, edge enhancement, color correction, false color mapping, and deblurring, to improve the visual effect of the image.

[0066] In this application, sharpening is a technique to enhance image details and edges. Common sharpening methods include using convolutional kernels (such as Laplacian or Sobel) to highlight high-frequency information in the image. Contrast enhancement is used to enhance the difference in brightness between different regions in an image, and can be achieved through methods such as histogram equalization, contrast stretching, or CLAHE (contrast-limited adaptive histogram equalization). Contrast enhancement can make the image look more vivid and clear. Edge enhancement techniques can be used to highlight edge features in an image, making them more prominent. This can be achieved through convolutional kernels or high-pass filters, such as Sobel, Prewitt, or Canny edge detectors. If the image has color deviation or balance issues, color correction techniques can be applied to adjust colors and tones to obtain a more accurate color representation. In some specific application scenarios, the grayscale values ​​of an image can be mapped to a pseudo-color image to present information more intuitively. This is common in fields such as medical image processing and thermal imaging. If the image is affected by blur or jitter, deblurring techniques can be applied to restore sharpness.

[0067] Understandably, the choice of appropriate post-processing techniques depends on the specific needs of the image and the desired effect in the specific application scenario. The post-processing described above is merely an example and can be used in combination to meet the needs of specific applications while ensuring that the quality and visual effect of the final image reach the expected level, thereby further improving the quality and clarity of the denoised image.

[0068] In this embodiment, for a current frame image containing a face, the face region of the current frame image can be denoised based on a deformed reference frame image to obtain a denoised image of the face region. For non-face regions of the current frame image, a preset denoising method can be used to denoise, resulting in a denoised image of the non-face regions. Based on the denoised images of the face and non-face regions of the current frame image, a denoised image of the current frame image is obtained. By applying different denoising methods to different regions of the current frame image, the accuracy of face image denoising can be improved.

[0069] This application provides a method for denoising face images, comprising: acquiring a current frame image containing a face; performing deformation registration processing on the face region of the current reference frame image based on facial key points of the current frame image and a current reference frame image to obtain a deformed reference frame image; wherein the current reference frame image is obtained by denoising at least one previous frame image containing a face; the deformed reference frame image maintains the same face pose as the current frame image; and performing denoising processing on the current frame image based on the deformed reference frame image to obtain a denoised image of the current frame image. On one hand, deformation registration processing can align the face region of the current frame image with the face region in the previous frame reference frame image, maintaining a consistent face pose. This helps eliminate pose changes between different frames, making subsequent processing more accurate. On the other hand, both the deformed reference frame image and the subsequent current frame image undergo denoising processing, which can reduce noise in the image, improve image quality, and help make facial features more clearly visible.

[0070] In some embodiments of this application, such as Figure 2 As shown, in S102, deformation registration processing is performed on the face region of the current reference frame image based on the face key points of the current frame image and the current reference frame image to obtain the deformed reference frame image. This can include S1021 to S1023:

[0071] S1021. Perform facial key point pairing on the facial key points of the current frame image and the current reference frame image to obtain facial key point pairs.

[0072] In this embodiment of the application, the electronic device performs face landmark pairing on the face landmarks of the current frame image and the current reference frame image to obtain face landmark pairs.

[0073] In this embodiment, the electronic device first uses facial landmark detection algorithms (such as MTCNN, Dlib, OpenCV, etc.) to detect and extract facial landmarks in the current frame image and the current reference frame image. Each facial landmark represents a specific location on the face, such as the eyes, nose, or mouth. Then, for each detected facial landmark, the electronic device typically calculates its feature descriptors, which represent the appearance and location information of each landmark. Further, the electronic device can find the correspondence between the feature descriptors in the two images by comparing them, which can be done using algorithms such as feature matching (e.g., nearest neighbor matching). Finally, once matching landmarks are found, facial landmark pairs can be established, mapping the landmarks in the current frame image and the current reference frame image one-to-one.

[0074] S1022. Based on the facial key point pairs, divide the facial regions of the current frame image and the current reference frame image into grids to obtain the first grid image corresponding to the current frame image and the second grid image corresponding to the current reference frame image; wherein, the number of grids in the first grid image and the second grid image is the same.

[0075] In this embodiment of the application, the electronic device divides the image into grids based on facial key point pairs as a method for obtaining local image blocks for subsequent deformation registration processing.

[0076] In this embodiment, the electronic device first determines the position and size of the face region based on facial key point pairs, which can be accomplished by calculating the distance and angle between the key points. Then, the electronic device determines the grid division parameters, including the number of rows and columns of the grid, and the size of each grid cell. These parameters can be selected according to application requirements. Further, the electronic device divides the face region into grids, ensuring that the same grid division method and parameters are used in the current frame image and the current reference frame image. The grid can be divided evenly according to the number of rows and columns, or different division strategies can be used according to other requirements. Finally, for each grid cell, the electronic device extracts corresponding local image patches from the current frame image and the current reference frame image. These image patches are components of the first grid image and the second grid image.

[0077] It should be noted that the mesh images corresponding to the face regions in the current frame image and the current reference frame image are further used for subsequent deformation registration to ensure that the mesh division method and parameters are consistent in the two images, so as to enable effective comparison and analysis.

[0078] In some embodiments of this application, when the facial key points of the current frame image and the current reference frame image are divided into triangular meshes, the facial key points of the current frame image and the current reference frame image are located at the vertices of the triangular mesh; or, when the facial key points of the current frame image and the current reference frame image are divided into quadrilateral meshes, the facial key points of the current frame image and the current reference frame image are anchor points close to the vertices of the mesh.

[0079] In this embodiment, the electronic device first performs keypoint detection on the faces in the current frame image and the current reference frame image. These keypoints typically include facial feature points such as eyes, nose, and mouth. Then, the electronic device generates a triangular mesh such that each keypoint is located at a vertex of a triangle, which can be accomplished using a triangulation algorithm to ensure that the keypoints correspond to the vertices of the triangles. Finally, these triangles are used for deformation registration to maintain consistency in the face regions of the current frame image and the current reference frame image, maximizing the similarity between the corresponding triangles.

[0080] In this embodiment, a quadrilateral mesh is used for deformation. The mesh vertices do not need to completely overlap with the key points of the face. The quadrilateral mesh vertex closest to the key points of the face is used as the important anchor point for movement. At the same time, the surrounding vertices of the anchor point are dragged to move in the same direction, which can also complete the deformation.

[0081] S1023. Based on the first grid image, perform pixel interpolation on the second grid image to obtain the deformed reference frame image.

[0082] In some embodiments of this application, the implementation of step S1023, which involves pixel interpolation of the second grid image based on the first grid image to obtain the deformed reference frame image, may include steps S10231 to S10233:

[0083] S10231. Map the grid vertices in the second grid image to the corresponding grid vertices in the first grid image to obtain the intermediate grid region.

[0084] In this embodiment, the electronic device moves the grid vertices (usually facial key points) of the face region of the reference frame to the grid vertices corresponding to the face region of the current frame to ensure that the corresponding key point positions match.

[0085] S10232. Interpolate the internal pixels of each grid in the middle grid region to obtain the internal pixels of each grid in the middle grid region.

[0086] In this embodiment of the application, the electronic device performs interpolation calculations based on the grid vertex coordinates for each pixel point inside each grid in the middle grid region. This interpolation process is to find the coordinates of each pixel point in the new position before deformation.

[0087] S10233: Map the pixel values ​​of each grid in the second grid image to the internal pixels in the corresponding middle grid region to obtain the deformed reference frame image.

[0088] In the embodiments of this application, after the electronic device determines the new coordinates of the pixel point before deformation, it can map the pixel value before deformation to the position after deformation. This means that the pixel value in the reference frame is moved to the corresponding position in the current frame, thereby obtaining the deformed reference frame image.

[0089] In this embodiment, firstly, the electronic device needs to determine the correspondence between the vertices of the triangular mesh (facial key points) of the reference frame and the vertices of the triangular mesh of the current frame. This can be achieved through facial key point matching and mesh division, ensuring that the triangles of each reference frame have the same vertices as the corresponding triangles of the current frame. Then, for each triangle in the reference frame, position interpolation of the internal pixels needs to be performed. This can be accomplished by performing interpolation calculations within the triangle. Typically, bilinear interpolation or triangular interpolation is used to find the (x, y) coordinates of each pixel before deformation. Finally, once the position of each pixel before deformation is determined, the electronic device can map the pixel values ​​before deformation to the pixels inside the deformed triangle. For each pixel, based on its position and color in the reference frame, and the result of position interpolation, its position and color in the deformed image can be calculated. Through the above steps, a deformed reference frame image is obtained, where the shape of the face region in the reference frame corresponds to that in the current frame.

[0090] For example, the steps of pixel interpolation of a first grid image and a second grid image to generate a deformed reference frame image may include: First, ensuring that the number of grids and the division method of the first and second grid images are consistent. This means that each grid cell in both images has the same shape and size. Then, the electronic device matches each grid vertex in the first grid image with the nearest corresponding grid vertex in the second grid image, thus establishing a correspondence between grid vertices. Subsequently, for each pixel, the corresponding grid cell and local coordinates are found based on its position in the first grid. Then, interpolation is performed in the second grid image using these local coordinates to obtain the deformed pixel value. Typically, methods such as bilinear interpolation or bicubic interpolation are used to estimate the deformed pixel value, which considers the weight of the pixel in its surrounding neighboring pixels to achieve a smooth deformation effect. Further, the interpolation process is repeated for each pixel in the first grid image until all pixels of the entire image have been processed. Finally, after interpolation is completed, the electronic device obtains a deformed reference frame image in which the pixel values ​​have been deformed according to the current frame image and keypoint information, so that the deformed current frame image is consistent with the face pose of the current frame.

[0091] In this embodiment, pixel interpolation can be performed based on a first grid image and a second grid image to generate a deformed reference frame image. This process involves mapping pixels from the second grid image onto the first grid image and interpolating at appropriate locations to obtain the deformed pixel values. This interpolation process can use various methods, such as bilinear interpolation, bicubic interpolation, etc., depending on the application requirements and algorithm. The interpolation result forms the deformed reference frame image, maintaining the same face pose as the current frame image.

[0092] In this embodiment, pixel interpolation is used to generate a deformed reference frame image to ensure that it is consistent with the face pose of the current frame image, so as to achieve deformation registration of the face region.

[0093] Understandably, on the one hand, the deformed reference frame image provides more consistent reference information, helping to remove noise from the current frame image. Because the deformed reference frame image has a similar facial pose to the current frame image, the denoising algorithm can more accurately preserve facial features while reducing the impact of noise. On the other hand, due to pixel interpolation, the deformed reference frame image can retain more detail, especially in the facial region, helping to maintain image clarity and quality. Furthermore, for the denoising process, the deformed reference frame image provides a more consistent structure, reducing artifacts or distortions generated during denoising. Finally, by removing noise and refining details, the deformed reference frame image can improve the overall visual quality of the image. In summary, the above processes help improve the image denoising effect, making the denoised image more suitable for subsequent image processing and analysis tasks, thus improving image quality and usability.

[0094] In some embodiments of this application, such as Figure 3 As shown, the implementation of temporal fusion processing between the deformed reference frame image and the current frame image to obtain the denoised image of the current frame image may include steps S201 to S203:

[0095] S201. For each first pixel in the current frame image, determine the matching pixel corresponding to the first pixel in each second pixel in the deformed reference frame image.

[0096] In some embodiments of this application, the implementation of determining the matching pixel corresponding to the first pixel in each of the second pixels in the deformed reference frame image for each first pixel in the current frame image in step S201 may include steps S2011 to S2012:

[0097] S2011. For each first pixel in the current frame image, determine the target pixel corresponding to the position of the first pixel in the deformed reference frame image.

[0098] In this embodiment of the application, for each first pixel in the current frame image, the electronic device finds the target pixel corresponding to the position of each first pixel in the deformed reference frame image, and can map the coordinates of the first pixel to the coordinates in the deformed reference frame image.

[0099] S2012. Based on the pixel difference value of the matching block centered on the first pixel point and the pixel difference value of the matching block corresponding to each second pixel point within a preset search range centered on the target pixel point in the deformed reference frame image, determine the matching pixel point corresponding to the first pixel point.

[0100] In this embodiment, after obtaining the correspondence between the first pixel and the target pixel, the electronic device needs to calculate the pixel difference value of the matching block centered on the first pixel. Within a preset search range centered on the target pixel in the deformed reference frame image, the pixel difference value of the matching block corresponding to each second pixel is calculated. By comparing the pixel difference values ​​of the matching block of the first pixel with the matching blocks of each second pixel within the target search range, it can be determined which second pixel best matches the first pixel, minimizing its matching block pixel difference value.

[0101] In this embodiment of the application, the matching block is pre-defined, for example, the size of the matching block is NxN.

[0102] In this embodiment of the application, the preset search range can be the range of [x0-R, x0+R] and [y0-R, y0+R].

[0103] In this embodiment, firstly, the electronic device selects a reference point P0 and chooses a position P0(x0, y0) within the face region of the current frame. This position may be a key point of the face or another point of interest. Then, the matching block size is set. The electronic device defines a matching block size, typically NxN pixels, used to calculate the grayscale difference between image blocks. Further, the search range is determined. To find the matching block corresponding to position P0 in the current frame within the reference frame, a radius R of the search region needs to be defined. The coordinate range of this search region on the reference frame is [x0-R, x0+R] and [y0-R, y0+R]. Further, the search region is traversed. The electronic device traverses each pixel within the search region on the reference frame. Further, the matching block is extracted. For each pixel P1'(x1, y1) within the search region, the electronic device extracts a matching block (NxN in size) centered on P1'. Further, the grayscale difference is calculated. The electronic device compares the extracted matching block with the matching block centered on position P0 in the current frame and calculates the grayscale difference between them. The difference is typically measured using methods such as Mean Absolute Difference (MAD) or Sum of Absolute Differences (SAD). Finally, to find the best match, the electronic device selects the pixel P1'(x1, y1) with the smallest SAD difference within the search area as the corresponding point on the reference frame that matches P0. Through this process, the best match is determined by calculating the grayscale difference between the matching block in each search area and the matching block in the current frame, thus finding the position on the reference frame corresponding to the position of P0 in the current frame.

[0104] Understandably, on the one hand, by mapping each pixel in the current frame image to the target pixel in the deformed reference frame image, deformation can be compensated, thus achieving pixel-level correspondence between the two images. On the other hand, by comparing the pixel difference values ​​of the matching blocks, the pixel that best matches each first pixel can be determined, thereby improving matching accuracy. This improves stability, ensuring accurate matching of corresponding pixels even under conditions of significant deformation or image changes. Furthermore, by determining the matching pixels, registration between the deformed reference frame image and the current frame image can be achieved, ensuring consistency in aspects such as facial pose, thus providing a better foundation for subsequent image denoising tasks. In summary, the above process helps ensure pixel-level correspondence between the deformed reference frame image and the current frame image, improving the accuracy and stability of image registration, thereby producing better results in subsequent image denoising applications.

[0105] S202. The pixel value of each first pixel point is weighted and fused with the pixel value of the corresponding matching pixel point to obtain the fused pixel value of the first pixel point.

[0106] In some embodiments of this application, the implementation of weighted fusion of the pixel value of each first pixel with the pixel value of the corresponding matching pixel in S202 to obtain the fused pixel value of the first pixel may include S2021 to S2023:

[0107] S2021. For each first pixel in the current frame image, if the difference between the pixel difference value of the matching block of the first pixel and the pixel difference value of the matching block of the corresponding matching pixel is less than or equal to a first preset threshold, the fusion weight is increased to obtain the adjusted fusion weight; wherein, the fusion weight represents the fusion strength between the pixel value of the first pixel and the pixel value of the corresponding matching pixel.

[0108] In this embodiment of the application, if the difference between the pixel difference value of the matching block of the first pixel and the pixel difference value of the matching block of the corresponding matching pixel is less than or equal to a first preset threshold, then the fusion weight is increased to obtain the adjusted fusion weight, which enhances the fusion when the pixel values ​​are relatively consistent, so as to maintain better consistency.

[0109] S2022, when the difference between the pixel difference value of the matching block of the first pixel point and the pixel difference value of the matching block of the corresponding matching pixel point is greater than or equal to the second preset threshold, the fusion weight is reduced to obtain the adjusted fusion weight; wherein, the second preset threshold is greater than or equal to the first preset threshold.

[0110] In this embodiment, if the difference between the pixel difference value of the matching block of the first pixel and the pixel difference value of the matching block of the corresponding matching pixel is greater than or equal to a second preset threshold (the second preset threshold should be greater than or equal to the first preset threshold), then the fusion weight is reduced to obtain the adjusted fusion weight. This reduces fusion when the pixel values ​​are relatively inconsistent, in order to avoid excessive smoothing or distortion.

[0111] S2023. Based on the adjusted fusion weight, the pixel value of the first pixel and the pixel value of the corresponding matching pixel are fused to obtain the fused pixel value of the first pixel.

[0112] In this embodiment, the electronic device first calculates the SAD value of the matching block for the current frame pixel P0(x0,y0) and the corresponding reference frame pixel P1'(x1,y1). The matching block is a rectangular area centered on the two pixels, typically a window of fixed size. Then, the electronic device calculates the SAD value. The SAD value is calculated by summing the absolute differences in the grayscale values ​​of each pixel within the two matching blocks. The smaller the SAD value, the more similar the two matching blocks are, and the better the matching effect. Further, the electronic device uses a fusion weight w to calculate the fused pixel value Pout = (1.0-w)*P0 + 0.5*w*(P1-P0). This formula indicates that the fused pixel value Pout is a linear combination of the current frame pixel P0 and the reference frame pixel P1', and the fusion weight w determines their relative contribution. When w is close to 1, the pixel value of P1' is dominant; when w is close to 0, the pixel value of P0 is dominant.

[0113] It is understandable that by calculating the SAD value and then using it to calculate the fusion weight, pixel values ​​are fused based on the similarity of the matching blocks, thereby improving the accuracy of matching and fusion. This process helps the deformed reference frame image to better match the current frame image.

[0114] Understandably, dynamically adjusting the fusion weights based on pixel differences and preset thresholds allows for more precise control of the fusion process, adapting to different image regions and situations. Enhancing fusion when pixel values ​​are relatively consistent helps maintain detail and consistency. Conversely, reducing fusion when pixel values ​​are relatively inconsistent can prevent over-smoothing or distortion.

[0115] Understandably, on the one hand, calculating the fusion weights using the grayscale difference (SAD) of the matching blocks can reduce artifacts or discontinuities that may occur during the transition between two images, thus improving image quality. On the other hand, the fusion process ensures that the deformed reference frame image and the current frame image are consistent in the temporal domain, which helps improve the consistency between the two images. The fusion weights are determined based on the SAD; the smaller the SAD, the stronger the fusion, meaning that more well-matched pixels will be retained in the final fused image, contributing to a smooth transition. Furthermore, with appropriate weighted fusion, temporal distortion of the image can be reduced, making the deformed reference frame image closer to the current frame image. In summary, temporal fusion helps to merge the current frame image and the deformed reference frame image together to obtain a higher quality and more consistent image.

[0116] S203. Determine the denoised image based on the fused pixel values ​​of each first pixel in the current frame image.

[0117] In some embodiments of this application, the time-domain fusion process includes: a first time-domain fusion process and a second time-domain fusion process.

[0118] In some embodiments of this application, temporal fusion processing is performed on the deformed reference frame image and the current frame image to obtain a denoised image of the current frame image, including S301 to S303:

[0119] S301. Perform a first temporal fusion process on the face region in the deformed reference frame image and the current frame image to obtain the first noise reduction region.

[0120] In this embodiment, a first temporal fusion process is performed on the face region in the deformed reference frame image and the current frame image. The face region is the region previously obtained based on facial keypoint pairing and mesh generation. In this step, the goal of fusion is to reduce noise in the face region. This may include removing some noise or other temporal interference introduced by the deformation.

[0121] S302. Perform a second temporal fusion process on the non-face regions in the deformed reference frame image and the current frame image to obtain a second noise reduction region; wherein, the preset search range in the first temporal fusion process is less than or equal to the preset search range in the second temporal fusion process; and the fusion weight in the first temporal fusion process is greater than or equal to the fusion weight in the second temporal fusion process.

[0122] In this embodiment, a second temporal fusion process is performed on the non-face regions (i.e., regions outside the face region) in the deformed reference frame image and the current frame image. The goal of this process is to reduce noise in the non-face regions while preserving other information in the image. Non-face regions may include background and other objects. The preset search range of the second temporal fusion process should be less than or equal to the preset search range of the first temporal fusion process, meaning that the processing of non-face regions is more localized. The fusion weight should be larger in the first temporal fusion process because the quality of the face region is more important. In the second temporal fusion process, the fusion weight can be smaller because the requirements for non-face regions are lower.

[0123] S303. Determine the denoised image of the current frame image based on the first denoising region and the second denoising region.

[0124] In this embodiment, a denoised image of the current frame is determined based on a first denoising region and a second denoising region. This is obtained by combining the results of a first temporal fusion process and a second temporal fusion process. The final denoised image should be an effective combination of the deformed reference frame image and the current frame image to reduce noise in both face and non-face regions while preserving other important information of the image. This process helps reduce noise introduced by deformation and temporal interference, improving the quality and clarity of the final denoised image.

[0125] The method for denoising facial images proposed in this application will be explained below in a specific embodiment.

[0126] Figure 4 A schematic diagram illustrating the framework of an optional face image noise reduction method provided in this application embodiment, as shown below. Figure 4As shown, the framework of the face image denoising method includes: a network pairing module 41, a deformation registration module 42, and a denoising processing module 43. Input data includes: a previous frame image with a face 401, previous frame face landmarks 402, current frame face landmarks 403, and current frame image with a face 404. First, the face landmarks 402 and 403 of the previous frame are input to the mesh pairing module 41 to obtain face landmark pairs between the current frame image with a face and the previous frame image with a face. Then, the facial keypoint pair and the previous frame image 401 containing a face are input into the deformation registration module 42 to perform deformation registration (MorphingTransform) on the previous frame image containing a face, resulting in a deformed previous frame image 405 with the same facial pose (direction, rotation, and angle of the face in the image) as the current frame image containing a face. Subsequently, the deformed previous frame image 405 and the current frame image 404 containing a face are input into the noise reduction processing module 43 to perform noise reduction processing on the facial region of the current frame image containing a face, such as temporal noise reduction (TNR). The final output image 406 is the noise-reduced image corresponding to the current frame image containing a face. This output image is updated and cached, and the reference image is updated for use in the noise reduction processing of the next frame image containing a face.

[0127] Specifically, the noise reduction process for each frame requires four inputs: a) the current frame image, b) the facial landmark information of the current frame, c) the previous frame image (reference frame image), and d) the landmark information of the previous frame image. The facial landmark information of the current frame is obtained from the current frame image by a facial landmark detection algorithm, which is usually calculated at the front end of the video stream. The previous frame image is updated after the calculation of the previous frame. The previous frame image is updated after each noise reduction calculation, thus completing an iterative and convergent noise reduction process. The landmark information of the previous frame image is updated from the facial landmarks of the previous frame.

[0128] The facial landmark information of the current frame and the previous frame image form a facial landmark pair, which is input together with the previous frame image into the facial deformation module (for deformation registration processing). After processing, the deformed facial image of the previous frame is obtained. At this time, the facial pose in the previous frame image can be completely matched with the current frame.

[0129] The deformation module (deformation registration module) is a deformation algorithm based on a triangular mesh. First, the face region is triangulated (Delaunay) based on facial key points to form a triangular mesh. Both the current frame and the previous frame need to be delaunay digitized. In order to ensure the consistency of the correspondence and that the deformed face images can be accurately aligned, the meshes must be completely identical. Then, the deformation algorithm will completely map the facial key points of the previous frame image to the current frame image. The remaining points are interpolated using the triangle interior region interpolation algorithm to complete the deformation of the entire face region. After deformation, the face regions of the previous frame image and the current frame image can completely overlap.

[0130] The temporal noise reduction (TNR) module (noise reduction processing module) performs noise reduction by averaging corresponding pixels in two adjacent frames of the video. The input to TNR is the current frame and the previous frame after morphing transformation. The TNR module fuses pixels at the same locations in the current and previous frames based on brightness differences. If the difference is large, the fusion ratio is reduced; if the difference is small, the fusion ratio is increased. For the face region, since morphing alignment has already been performed, the threshold for judging the fusion strength can be increased accordingly, thus increasing the fusion strength in the face region and improving the temporal noise reduction effect.

[0131] Furthermore, after TNR denoising is completed, the denoised current frame image will be cached and updated, becoming the reference image of the previous frame when denoising the image at the next moment, while updating the facial key point information.

[0132] Figure 4 The face image denoising method shown includes two main algorithms in its process: deformation registration processing and denoising processing. The two algorithms are explained below.

[0133] Figure 5 A schematic diagram of an optional deformation registration process provided in an embodiment of this application is shown below. Figure 5 As shown, the method includes S51 to S54:

[0134] S51, First Delaunay Triangulation Process.

[0135] In this embodiment, the facial key points 402 of the previous frame are processed by the first Delaunay triangulation to obtain the second grid image corresponding to the previous frame image (current reference frame image).

[0136] S52, Second Delaunay Triangulation Treatment.

[0137] In this embodiment, the facial key points 403 of the current frame are processed by the second Delaunay triangulation to obtain the first grid image corresponding to the current frame image.

[0138] S53, Network Pairing Processing.

[0139] In this embodiment of the application, the second grid image and the second grid image are subjected to network pairing processing to obtain facial key point pairs.

[0140] S54, Deformation Registration Processing.

[0141] In this embodiment, the image 401 of the previous frame containing a face and the facial key point pair are subjected to deformation registration processing to obtain the deformed image 405 of the previous frame (current reference frame image).

[0142] In this embodiment of the application, the deformation registration process may specifically include the following steps:

[0143] 1) Cache the first frame of image information.

[0144] In this embodiment of the application, the first frame of the video at the start of video acquisition does not need to be processed by deformation registration and noise reduction (temporal noise reduction). Instead, a face key point detection algorithm is called to detect face key points, and then the first frame image and the information of the face key points in that image are cached.

[0145] 2) Matching of facial landmarks between consecutive frames.

[0146] In this embodiment of the application, processing begins from the second frame image. The image to be processed is called the current frame (current frame image), and the previous frame cached image is called the reference frame. The facial key point information of the reference frame image has been cached. The facial key point detection algorithm needs to be called to detect the current frame image to obtain the key point information of the current frame. The facial key point information of the reference frame and the current frame image form a facial key point pair.

[0147] 3) Mesh splitting of facial key points between consecutive frames.

[0148] In this embodiment, the facial key points of the current frame and the reference frame are split into triangular meshes, and the point cloud is used as the vertices of the triangles. Figure 6 This is a schematic diagram of an optional grid division of a face region provided in an embodiment of this application, as shown below. Figure 6 As shown, after the face region is divided into triangular meshes, the face region surrounding the keypoints is divided into meshes composed of multiple triangles. It should be noted that the current frame and the reference frame must use the same triangle composition, the same number of triangles (they must correspond one-to-one), and the same triangle division method.

[0149] 4) Reference frame deformation

[0150] In this embodiment, deformation registration processing is performed on the face region of the reference frame. Specifically, the following steps are taken: First, the vertices of the triangular mesh (i.e., the key points of the face) of the face region of the reference frame are moved to the corresponding triangular mesh vertices of the face region of the current frame. Then, the position of each pixel inside the triangle of the reference frame is interpolated according to the three vertices of the triangle to obtain the (x, y) coordinates before deformation. Finally, the pixel values ​​before deformation are mapped to the pixels inside the deformed triangle to complete the mapping of each pixel value of the deformed image, and finally the deformed face image (the deformed reference frame image) is obtained.

[0151] Figure 7 A schematic diagram illustrating an optional noise reduction process provided in an embodiment of this application, such as... Figure 7 As shown, the deformed previous frame image 405 and the current frame image 404 with a face are input to the noise reduction processing module 43 to perform temporal noise reduction processing on the current frame image 404 with a face, and obtain the output image 406 (noise-reduced image). The reference frame image (the previous frame image with a face) is updated based on the output image 406 so that it can be used for noise reduction processing of the next frame image with a face.

[0152] In this embodiment of the application, the temporal domain noise reduction process may specifically include the following steps:

[0153] 1) Face region block matching.

[0154] In this embodiment, taking a pixel at a certain position P0(x0, y0) within the face region of the current frame as an example, its corresponding point at the same position in the reference frame is P0'(x0, y0). The matching block size is set to NxN. The radius R of the search range region is found in the reference frame. The search region on the corresponding reference frame is then defined as follows: x0-R->x0+R, y0-R->y0+R. Each pixel within this region of the reference frame is traversed, and a matching block (NxN in size) centered on each pixel is extracted. The grayscale difference SAD is calculated between this block and the matching block centered on point P0 in the current frame (also NxN in size). The pixel P1'(x1, y1) with the smallest SAD difference within the search region of the reference frame is then selected.

[0155] 2) Temporal fusion.

[0156] In this embodiment, the current frame pixel P0(x0,y0) and the reference frame pixel P1'(x1,y1) are weighted and fused in the previous step. The grayscale difference SAD between the matching blocks of P0 and P1' is used as the weight reference. The smaller the SAD, the greater the fusion weight (there is a baseline value that is adjusted upwards and downwards). It can be expressed by the formula: Pout=(1.0-w)*P0+0.5*w*(P1-P0); where the smaller the SAD and the larger the w, the greater the fusion strength of P0 and P1.

[0157] 3) Adjusting face region parameters.

[0158] In this embodiment, temporal denoising of the face region can be completed by traversing and processing all pixels of the face region according to the previous steps. Since the deformation alignment operation has already been completed in this region, the intensity of the parameters in the previous steps can be adjusted. Compared with traditional temporal denoising processing, the search region radius R in the face region can be appropriately reduced, which can reduce the amount of computation without affecting the effect; the fusion weight in the face region can be appropriately increased, because the reference frame has already undergone deformation alignment operation, and the fusion can be more aggressive.

[0159] 4) Cache the results of temporal domain noise reduction processing.

[0160] In this embodiment of the application, after the temporal denoising process is completed, the denoised current frame image will be cached and updated, and will become the reference image of the previous frame when the image is denoised at the next moment. At the same time, the information of the facial key points, which are also used as the reference frame, will be updated.

[0161] In this embodiment, facial key point information is used as an important reference input. A facial deformation registration algorithm is used to align the facial regions of consecutive frames in the video. Then, temporal noise reduction of the facial regions is performed. Compared with traditional temporal noise reduction, the matching degree of the facial regions of consecutive frames is higher, which can effectively improve the matching accuracy of facial region pixels, improve the temporal noise reduction effect, reduce the ghosting phenomenon that is easy to produce in traditional temporal noise reduction, and reduce the computational complexity of temporal noise reduction.

[0162] In this embodiment, the temporal denoising principle can be divided into two parts: image registration (matching) and temporal fusion. General methods use optical flow and image block matching algorithms for image registration, without special processing for the face region. This embodiment incorporates a deformation registration algorithm based on facial key points within the face area. After deformation, the face region is basically registered and aligned. Then, image block matching is used for fine registration. This reduces the search range of block matching, lowers computational complexity, and significantly improves the matching accuracy of the face region, thereby enhancing the temporal denoising effect.

[0163] In this embodiment, facial key point information is used to perform deformation registration of the facial regions in consecutive frames of the video, so that the facial regions in consecutive frames overlap in spatial position, thereby assisting in higher accuracy of temporal noise reduction fusion matching of the video facial region and achieving better temporal noise reduction effect in the facial region.

[0164] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps; or steps from different embodiments may be combined into a new technical solution.

[0165] Based on the foregoing embodiments, Figure 8 A schematic diagram of an optional face image noise reduction device provided in an embodiment of this application is shown below. Figure 8 As shown, the face image noise reduction device 80 includes an acquisition unit 81, a processing unit 82, and a noise reduction unit 83; wherein,

[0166] The acquisition unit 81 is used to acquire the current frame image containing a human face;

[0167] The processing unit 82 is configured to perform deformation registration processing on the face region of the current reference frame image based on the face key points of the current frame image and the current reference frame image to obtain a deformed reference frame image; wherein, the current reference frame image is used to provide face key points for the current frame image to perform the deformation registration processing; the deformed reference frame image is matched with the face pose of the current frame image;

[0168] The noise reduction unit 83 is used to perform noise reduction processing on the current frame image based on the deformed reference frame image to obtain a noise-reduced image of the current frame image.

[0169] In some embodiments of this application, the processing unit 82 is further configured to perform facial key point pairing on the facial key points of the current frame image and the current reference frame image to obtain facial key point pairs; based on the facial key point pairs, to perform grid division on the facial regions of the current frame image and the current reference frame image respectively to obtain a first grid image corresponding to the current frame image and a second grid image corresponding to the current reference frame image; wherein, the number of grids in the first grid image and the second grid image is the same; and to perform pixel interpolation on the second grid image based on the first grid image to obtain the deformed reference frame image.

[0170] In some embodiments of this application, the processing unit 82 is further configured to map the grid vertices in the second grid image to the corresponding grid vertices in the first grid image to obtain an intermediate grid region; interpolate the internal pixels of each grid in the intermediate grid region to obtain the internal pixels of each grid in the intermediate grid region; and map the pixel values ​​of each grid in the second grid image to the corresponding internal pixels in the intermediate grid region to obtain the deformed reference frame image.

[0171] In some embodiments of this application, when the facial key points of the current frame image and the current reference frame image are divided into triangular meshes, the facial key points of the current frame image and the current reference frame image are located at the vertices of the triangular mesh; or, when the facial key point information of the current frame image and the current reference frame image is divided into quadrilateral meshes, the facial key points of the current frame image and the current reference frame image are anchor points close to the vertices of the mesh.

[0172] In some embodiments of this application, the noise reduction unit 83 is used to perform temporal fusion processing on the deformed reference frame image and the current frame image to obtain a noise-reduced image of the current frame image, so as to realize the noise reduction processing of the current frame image.

[0173] In some embodiments of this application, the noise reduction unit 83 is further configured to, for each first pixel in the current frame image, determine a matching pixel corresponding to the first pixel among each second pixel in the deformed reference frame image; perform weighted fusion of the pixel value of each first pixel with the pixel value of the corresponding matching pixel to obtain a fused pixel value of the first pixel; and determine the noise-reduced image based on the fused pixel values ​​of each first pixel in the current frame image.

[0174] In some embodiments of this application, the noise reduction unit 83 is further configured to, for each first pixel in the current frame image, determine a target pixel in the deformed reference frame image corresponding to the position of the first pixel; and determine the matching pixel corresponding to the first pixel based on the pixel difference value of the matching block centered on the first pixel and the pixel difference value of the matching block corresponding to each second pixel in the preset search range centered on the target pixel in the deformed reference frame image.

[0175] In some embodiments of this application, the noise reduction unit 83 is further configured to, for each first pixel in the current frame image, increase the fusion weight when the difference between the pixel difference value of the matching block of the first pixel and the pixel difference value of the matching block of the corresponding matching pixel is less than or equal to a first preset threshold, to obtain an adjusted fusion weight; wherein the fusion weight represents the fusion strength between the pixel value of the first pixel and the pixel value of the corresponding matching pixel; or, decrease the fusion weight when the difference between the pixel difference value of the matching block of the first pixel and the pixel difference value of the matching block of the corresponding matching pixel is greater than or equal to a second preset threshold, to obtain the adjusted fusion weight; wherein the second preset threshold is greater than or equal to the first preset threshold; and, based on the adjusted fusion weight, fuse the pixel value of the first pixel and the pixel value of the corresponding matching pixel to obtain the fused pixel value of the first pixel.

[0176] In some embodiments of this application, the temporal fusion processing includes: a first temporal fusion processing and a second temporal fusion processing; the denoising unit 83 is further configured to perform the first temporal fusion processing on the face regions in the deformed reference frame image and the current frame image to obtain a first denoised region; perform the second temporal fusion processing on the non-face regions in the deformed reference frame image and the current frame image to obtain a second denoised region; wherein, the preset search range in the first temporal fusion processing is less than or equal to the preset search range in the second temporal fusion processing; the fusion weight in the first temporal fusion processing is greater than or equal to the fusion weight in the second temporal fusion processing; and the denoised image of the current frame image is determined based on the first denoised region and the second denoised region.

[0177] It should be noted that the division of modules or units in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical units, or have two or more units integrated into one unit. The integrated units described above can be implemented in hardware, as software functional units, or a combination of software and hardware.

[0178] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0179] Figure 9 The following is a schematic diagram of an optional electronic device provided as an embodiment of this application, such as... Figure 9 As shown, this application embodiment provides an electronic device 90, including:

[0180] Memory 91 is used to store executable instructions;

[0181] The processor 92 is used to execute executable instructions stored in the memory 91 to implement the face image noise reduction method described above.

[0182] It is understood that the memory 91 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 91 of the systems and methods described in this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0183] The processor 92 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 92 or by instructions in software form. The processor 22 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 91, and the processor 92 reads the information in memory 91 and, in conjunction with its hardware, completes the steps of the above method.

[0184] It is understood that the embodiments described in this application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), DSP devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or combinations thereof. For software implementation, the technology described in this application can be implemented through modules (e.g., procedures, functions, etc.) that perform the functions described in this application. Software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0185] This application provides a computer storage medium storing executable instructions. When the executable instructions are executed, they cause a processor to perform a noise reduction method for a face image as described in one or more of the above embodiments.

[0186] In some embodiments, the computer-readable storage medium may be applied to the electronic device of this embodiment, and when the computer program is executed by at least one processor, it implements the corresponding processes implemented by the electronic device in the various methods of this embodiment. For the sake of brevity, these will not be described in detail here.

[0187] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the face image noise reduction method provided in the above method embodiments.

[0188] In some embodiments, the computer program product can be applied to the electronic device in this embodiment, and the computer program instructions cause the computer to execute the corresponding processes implemented by the electronic device in the various methods of this embodiment. For the sake of brevity, these will not be described in detail here.

[0189] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0190] It should be noted that the descriptions of the storage medium and device embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0191] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0192] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0193] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0194] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0195] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0196] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0197] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0198] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0199] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0200] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0201] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for denoising a face image, characterized in that: The method comprises: Get the current frame image with the face; According to the facial key points of the current frame image and the current reference frame image, a deformation registration process is performed on the facial region of the current reference frame image to obtain a deformed reference frame image; wherein the current reference frame image is used to provide facial key points for the deformation registration process of the current frame image; the deformed reference frame image matches the facial posture of the current frame image; According to the deformed reference frame image, the current frame image is subjected to denoising processing to obtain a denoised image of the current frame image.

2. The method according to claim 1, characterized in that The step of performing deformation registration processing on the face region of the current reference frame image according to the face key points of the current frame image and the current reference frame image to obtain a deformed reference frame image includes: Pairing facial key points of the current frame image and the current reference frame image to obtain a facial key point pair; According to the facial key point pairs, the facial regions of the current frame image and the current reference frame image are respectively divided into grids to obtain a first grid image corresponding to the current frame image and a second grid image corresponding to the current reference frame image; wherein the number of grids in the first grid image and the second grid image is the same; Based on the first grid image, pixel interpolation is performed on the second grid image to obtain the deformed reference frame image.

3. The method according to claim 2, characterized in that The step of performing pixel interpolation on the second grid image based on the first grid image to obtain the deformed reference frame image includes: Mapping mesh vertices in the second mesh image to mesh vertices corresponding to the first mesh image to obtain an intermediate mesh area; Interpolating the internal pixel points of each grid in the middle grid area to obtain the internal pixel points of each grid in the middle grid area; The pixel values ​​of the pixel points of each grid in the second grid image are mapped to the internal pixel points of the corresponding middle grid area to obtain the deformed reference frame image.

4. The method according to claim 2 or 3, characterized in that: In the case where a triangular mesh is used to divide the facial key points of the current frame image and the current reference frame image, the facial key points of the current frame image and the current reference frame image are located at the vertex positions of the triangular mesh; or When the facial key points of the current frame image and the current reference frame image are divided into quadrilateral grids, the facial key points of the current frame image and the current reference frame image are anchor points close to grid vertices.

5. The method according to any one of claims 1 to 4, characterized in that: The step of performing noise reduction processing on the current frame image according to the deformed reference frame image to obtain a noise-reduced image of the current frame image includes: The deformed reference frame image and the current frame image are subjected to time domain fusion processing to obtain a denoised image of the current frame image, so as to achieve denoising processing on the current frame image.

6. The method according to claim 5, characterized in that The step of performing time domain fusion processing on the deformed reference frame image and the current frame image to obtain a denoised image of the current frame image includes: For each first pixel point in the current frame image, determining a matching pixel point corresponding to the first pixel point among each second pixel point in the deformed reference frame image; Performing weighted fusion on the pixel value of each of the first pixels and the pixel value of the corresponding matching pixel to obtain a fused pixel value of the first pixel; The denoised image is determined according to the fused pixel values ​​of each of the first pixels in the current frame image.

7. The method according to claim 6, characterized in that The step of determining, for each first pixel point in the current frame image, a matching pixel point corresponding to the first pixel point in each second pixel point in the deformed reference frame image includes: For each first pixel point in the current frame image, determining a target pixel point corresponding to the position of the first pixel point in the deformed reference frame image; The matching pixel point corresponding to the first pixel point is determined according to the pixel difference value of the matching block centered on the first pixel point and the pixel difference value of the matching block corresponding to each second pixel point within a preset search range centered on the target pixel point in the deformed reference frame image.

8. The method according to claim 6, characterized in that The step of performing weighted fusion on the pixel value of each of the first pixels and the pixel value of the corresponding matching pixel to obtain the fused pixel value of the first pixels includes: For each first pixel point in the current frame image, when the difference between the pixel difference value of the matching block of the first pixel point and the pixel difference value of the matching block of the corresponding matching pixel point is less than or equal to a first preset threshold, the fusion weight is increased to obtain an adjusted fusion weight; wherein the fusion weight represents the fusion strength of the pixel value of the first pixel point and the pixel value of the corresponding matching pixel point; or, When the difference between the pixel difference value of the matching block of the first pixel point and the pixel difference value of the matching block of the corresponding matching pixel point is greater than or equal to a second preset threshold, reducing the fusion weight to obtain the adjusted fusion weight; wherein the second preset threshold is greater than or equal to the first preset threshold; Based on the adjusted fusion weight, the pixel value of the first pixel point and the pixel value of the corresponding matching pixel point are fused to obtain the fused pixel value of the first pixel point.

9. The method according to any one of claims 5 to 8, characterized in that: The time domain fusion processing includes: a first time domain fusion processing and a second time domain fusion processing; The step of performing time domain fusion processing on the deformed reference frame image and the current frame image to obtain a denoised image of the current frame image includes: Performing the first time domain fusion processing on the deformed reference frame image and the face region in the current frame image to obtain a first noise reduction region; Performing the second time domain fusion processing on the deformed reference frame image and the non-face area in the current frame image to obtain a second noise reduction area; wherein the preset search range in the first time domain fusion processing is less than or equal to the preset search range in the second time domain fusion processing; and the fusion weight in the first time domain fusion processing is greater than or equal to the fusion weight in the second time domain fusion processing; The denoised image of the current frame image is determined according to the first denoised area and the second denoised area.

10. A denoising device for a face image, characterized in that: The device comprises an acquisition unit, a processing unit and a noise reduction unit; wherein, The acquisition unit is used to acquire the current frame image with a human face; The processing unit is used to perform deformation registration processing on the face area of ​​the current reference frame image according to the face key points of the current frame image and the current reference frame image to obtain a deformed reference frame image; wherein the current reference frame image is used to provide face key points for the deformation registration processing of the current frame image; the deformed reference frame image matches the face posture of the current frame image; The denoising unit is used to perform denoising processing on the current frame image according to the deformed reference frame image to obtain a denoised image of the current frame image.

11. An electronic device, characterized in that: include: A memory for storing executable instructions; The processor is used to implement the facial image denoising method described in any one of claims 1 to 9 when executing the executable instructions stored in the memory.

12. A computer-readable storage medium, characterized in that: The storage medium stores executable instructions, and when the executable instructions are executed by one or more processors, the method for denoising a facial image as described in any one of claims 1 to 9 is implemented.

Images