A method, apparatus, and computer equipment for enhancing facial image quality in video conferencing based on single-frame images.

Abstract

This application relates to a method, apparatus, and computer device for enhancing facial image quality in video conferencing based on single-frame images. The method includes acquiring a target facial image; performing degradation processing and noise perturbation on the target facial image to obtain a first low-quality facial image and a second low-quality facial image; inputting the first and second low-quality facial images into a pre-constructed convolutional neural network model, and simultaneously training the model using a preset target loss function to enhance the facial image, outputting a first enhanced facial image and a second enhanced facial image; when the convolutional neural network model converges, this model is used as the target model, which is then used to enhance the single-frame facial image to be enhanced. This application effectively ensures the stability of consecutive video frames and enhances the consistency of facial restoration under different noise levels.

Description

Technical Field

[0001] This application relates to the field of video conferencing image technology, and in particular to a method, apparatus and computer equipment for enhancing facial image quality in video conferencing based on a single frame image. Background Technology

[0002] In video conferencing scenarios, the clarity of faces in the video can decrease due to factors such as network bandwidth, conferencing equipment, and the conferencing environment, thus affecting the user experience.

[0003] Existing face enhancement methods enhance the face in each video frame individually and then paste it back into the original frame. This method relies on the image quality and stability of the original video and is difficult to solve the temporal stability problem caused by the inconsistency of noise between frames.

[0004] Regarding the aforementioned technologies, the inventors discovered that existing face image quality enhancement methods suffer from problems such as temporal stability issues in single-frame face image quality enhancement, which can affect the face image quality in video conferencing. Summary of the Invention

[0005] To reduce the impact of temporal stability issues in single-frame face enhancement on the face quality of video conferencing, this application provides a method, apparatus, and computer device for enhancing face quality in video conferencing based on single-frame images.

[0006] In one aspect, this application provides a method for enhancing facial image quality in video conferencing based on a single frame image.

[0007] This application is achieved through the following technical solution:

[0008] A method for enhancing facial image quality in video conferencing based on a single frame image includes the following steps:

[0009] Acquire face images with a resolution that meets the preset resolution conditions to obtain the target face image;

[0010] The target face image is degraded to obtain a first low-quality face image; and

[0011] The first low-quality face image is subjected to noise perturbation to obtain a second low-quality face image;

[0012] The first low-quality face image and the second low-quality face image are input into a pre-constructed convolutional neural network model, and face enhancement training is performed simultaneously using a preset target loss function. The first enhanced face image and the second enhanced face image are output. The target loss function is used to supervise the face reconstruction quality and temporal face consistency under different noise levels.

[0013] When the convolutional neural network model converges, the current convolutional neural network model is used as the target model, which is used to perform image enhancement on the single-frame face image to be enhanced.

[0014] In a preferred embodiment, this application can be further configured such that the step of degrading the target face image includes...

[0015] Randomly select at least one degradation method from Gaussian blur or mean blur, Gaussian noise or Poisson noise, compressed noise and random upsampling and downsampling, and randomly sort the selected degradation methods to obtain the first image degradation sequence;

[0016] The target face image is degraded using the first image degradation sequence.

[0017] In a preferred embodiment, this application can be further configured to use the following formula when performing degradation processing on the target face image.

[0018] LQ1=[RS(RC(Blur,Noise,JPEG,Resize))] HQ ;

[0019] In the formula, LQ1 represents the first low-quality face image, HQ represents the target face image, Blur represents Gaussian blur or mean blur, Noise represents Gaussian noise or Poisson noise, JPEG represents compressed noise, Resize represents random upsampling or downsampling, RC represents randomly selecting at least one degradation method, and RS represents randomly sorting the selected degradation methods.

[0020] In a preferred embodiment, this application can be further configured such that the step of noise perturbation of the first low-quality face image includes...

[0021] Fine-tuning is performed on each degradation method in Gaussian blur or mean blur, Gaussian noise or Poisson noise, compressed noise, and random upsampling and downsampling.

[0022] Randomly select at least two degradation methods from the following: fine-tuned Gaussian blur or mean blur, fine-tuned Gaussian noise or Poisson noise, fine-tuned compressed noise, and fine-tuned random upsampling and downsampling, and randomly sort the selected degradation methods to obtain the second image degradation sequence.

[0023] The first low-quality face image is subjected to noise perturbation using the second image degradation sequence.

[0024] In a preferred embodiment, this application can be further configured such that, when performing noise perturbation on the first low-quality face image, the following formula is used.

[0025] LQ2=[RS(RC(Blur+α,Noise+β,JPEG+γ,Resize+δ))] LQ1 ;

[0026] In the formula, LQ2 represents the second low-quality face image, LQ1 represents the first low-quality face image, Blur represents Gaussian blur or mean blur, Noise represents Gaussian noise or Poisson noise, JPEG represents compressed noise, Resize represents random upsampling or downsampling, RC represents randomly selecting at least one degradation method, RS represents randomly sorting the selected degradation methods, and α, β, γ and δ all represent small variations in noise in different frames of the simulated video.

[0027] In a preferred embodiment, this application can be further configured such that: the target loss function includes a face reconstruction loss function for supervising the quality of face reconstruction, the expression of which includes,

[0028]

[0029] in, Let G(x) represent the face reconstruction loss function, x represent the first low-quality face image or the second low-quality face image, G(x) represent the output of the convolutional neural network model, HQ represent the target face image, L1 represent the difference pixels between the first enhanced face image or the second enhanced face image and the target face image, and L... p λ represents the difference between different layers of features of the first enhanced face image or the second enhanced face image and the target face image, i.e., the perceptual loss, where λ1 represents the weight coefficient of the perceptual loss, and L... g λ represents the GAN loss, and λ2 represents the weight coefficient of the GAN loss.

[0030] In a preferred embodiment, this application can be further configured such that: the target loss function includes a face consistency loss function, the expression of which includes,

[0031]

[0032] G(LQ1)==G(LQ2);

[0033] in, Let G(LQ1) represent the face consistency loss function, G(LQ2) represent the first enhanced face image, G(LQ2) represent the second enhanced face image, and L1' represent the difference pixels between the first enhanced face image and the second enhanced face image. p ' represents the difference between different layers of features in the first enhanced face image and the second enhanced face image, i.e., the consistency loss, λ1' represents the weight coefficient of the consistency loss, and Lid λ represents the face recognition loss of the first enhanced face image and the second enhanced face image, and λ3 represents the weight coefficient of the face recognition loss.

[0034] In a preferred embodiment, this application can be further configured such that: the convolutional neural network model adopts a UNet-Like structure, the UNet-Like structure including,

[0035] The face feature extraction module is used to perform multi-scale feature extraction on the first low-quality face image and the second low-quality face image respectively to obtain the first feature face image and the second feature face image;

[0036] The face reconstruction module is used to perform multi-scale reconstruction on the first feature face image and the second feature face image to obtain the first enhanced face image and the second enhanced face image.

[0037] Secondly, this application provides a video conferencing face image quality enhancement device based on a single frame image.

[0038] This application is achieved through the following technical solution:

[0039] A video conferencing face quality enhancement device based on a single frame image includes,

[0040] The target image module is used to acquire face images with a resolution that meets the preset resolution conditions to obtain the target face image;

[0041] The sample image module is used to degrade the target face image to obtain a first low-quality face image; and to apply noise perturbation to the first low-quality face image to obtain a second low-quality face image.

[0042] The training module is used to input the first low-quality face image and the second low-quality face image into a pre-constructed convolutional neural network model, and simultaneously perform face enhancement training using a preset target loss function, outputting a first enhanced face image and a second enhanced face image, wherein the target loss function is used to supervise the face reconstruction quality and temporal face consistency under different noise levels.

[0043] The target model module is used to take the convolutional neural network model as the target model when the convolutional neural network model converges. The target model is used to perform image enhancement on the single-frame face image to be enhanced.

[0044] Thirdly, this application provides a computer device.

[0045] This application is achieved through the following technical solution:

[0046] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described methods for enhancing the face quality in video conferencing based on a single-frame image.

[0047] In summary, compared with the prior art, the beneficial effects of the technical solution provided in this application include at least the following:

[0048] A target face image is obtained by acquiring face images with a resolution that meets a preset resolution condition. This target face image is used as the training target for the model, making the training results tend to be high-quality face images. The target face image is then degraded to obtain a first low-quality face image to simulate the noise changes in a single frame. The first low-quality face image is then subjected to noise perturbation to obtain a second low-quality face image to simulate the noise changes in another single frame. The noise changes in the second low-quality face image are different from those in the first low-quality face image. The first and second low-quality face images are then input into a pre-constructed convolutional neural network model, and face enhancement training is performed simultaneously using a preset target loss function. The face reconstruction quality and temporal face consistency under different noise levels are supervised to obtain the target model. The model then performs image enhancement on the single frame face image to be enhanced, thereby enhancing the model's ability to restore faces consistently when dealing with different noise levels. This ensures the stability of consecutive frames in the video and reduces the impact of temporal stability issues caused by single-frame face image quality enhancement on the face image quality of video conferencing. Attached Figure Description

[0049] Figure 1 The main flowchart of a video conferencing face quality enhancement method based on a single-frame image is provided as an exemplary embodiment of this application.

[0050] Figure 2 The following is a structural block diagram of a convolutional neural network model for a video conferencing face quality enhancement method based on a single-frame image, which is another exemplary embodiment of this application.

[0051] Figure 3 This application provides a structural block diagram of a video conferencing face quality enhancement device based on a single-frame image, which is another exemplary embodiment of the present application. Detailed Implementation

[0052] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of this application.

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0055] Existing methods for enhancing facial image quality fall into two main categories;

[0056] (1) Enhance the face of each video frame separately and then paste it back into the original frame. This method relies on the image quality and stability of the original video and is difficult to solve the temporal stability problem caused by the inconsistency of noise between frames.

[0057] (2) Temporal face image enhancement is adopted. A temporal module is introduced into the model structure to fuse face features across multiple frames, thereby achieving temporal stability of the face. However, this method relies on the clarity of the face image during model training and has high hardware computing power requirements, making it difficult to guarantee the real-time performance of video conferencing scenarios.

[0058] Therefore, this application proposes a method for enhancing face image quality in video conferencing based on single-frame images. During the model training process, for single-frame video images, the method simulates noise changes (noise, blur, encoding / decoding, upsampling, etc.) between different frames in the video to enhance the model's consistency in face restoration when dealing with different noises, thereby ensuring the temporal stability of the video frames and improving face image quality in video conferencing.

[0059] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0060] Reference Figure 1 This application provides a method for enhancing face image quality in video conferencing based on a single frame image. The main steps of the method are described below.

[0061] S1: Acquire face images whose resolution meets the preset resolution conditions to obtain the target face image;

[0062] S2: Degrade the target face image to obtain a first low-quality face image; and

[0063] S3: Noise perturbation is applied to the first low-quality face image to obtain a second low-quality face image;

[0064] S4: Input the first low-quality face image and the second low-quality face image into a pre-constructed convolutional neural network model, and simultaneously perform face enhancement training using a preset target loss function, outputting a first enhanced face image and a second enhanced face image, wherein the target loss function is used to supervise the face reconstruction quality and temporal face consistency under different noise levels;

[0065] S5: When the convolutional neural network model converges, the current convolutional neural network model is used as the target model, which is used to perform image enhancement on the single-frame face image to be enhanced.

[0066] Specifically, facial images that meet a preset resolution requirement, such as 1024x1024 resolution, are acquired to obtain HQ (High Quality Image), which is the target facial image. HQ images are typically high-resolution facial images captured by a local camera, or local high-resolution facial images that have undergone some minor noise reduction or beautification.

[0067] The acquired target face image is degraded through processes such as image blurring, compression, downsampling, and so on, to obtain the first low-quality face image LQ1, or LQ (Low Quality Image), and then the image LQ1 is constructed.

[0068] Next, noise perturbation is applied to the first low-quality face image, such as fine-tuning the image blur processing parameters, to obtain the second low-quality face image LQ2. This is done by further perturbating the original degraded (degraded) image with noise to construct image LQ2.

[0069] The first low-quality face image and the second low-quality face image are then input into a pre-constructed convolutional neural network model, such as a VGGNet model or a ResNet model, and face enhancement training is performed simultaneously using a preset target loss function. Face enhancement training is also performed on LQ1 and LQ2 at the same time, with shared model weight parameters. The output is the first enhanced face image and the second enhanced face image. The target loss function is used to supervise the face reconstruction quality and temporal face consistency under different noise levels.

[0070] When the model converges, the current convolutional neural network model is output as the target model to perform image enhancement on the single-frame face image to be enhanced.

[0071] After the model converges, a single face image quality enhancement model is obtained. By inputting a low-quality face image LQ, this single face image quality enhancement model can obtain G(LQ) with a quality similar to the original HQ, which meets the requirements of practical real-time applications.

[0072] By simultaneously inputting two face images of different qualities into the same model for optimization, after training, for face videos, only one frame of face image needs to be input into a single model to obtain the enhanced face image output.

[0073] In one embodiment, the step of degrading the target face image includes,

[0074] Randomly select at least one degradation method from Gaussian blur or mean blur, Gaussian noise or Poisson noise, compressed noise and random upsampling and downsampling, and randomly sort the selected degradation methods to obtain the first image degradation sequence;

[0075] The target face image is degraded using the first image degradation sequence.

[0076] Low-quality face images in conference video scenarios can typically be simulated by one or more of the following: blurring, noise, random upsampling and downsampling, JPEG compression, etc. By randomly selecting one or more degradation methods and randomly sorting the degradation methods, a first low-quality face image is constructed as one of the face pairing data.

[0077] In one embodiment, the following formula is used to degrade the target face image:

[0078] LQ1=[RS(RC(Blur,Noise,JPEG,Resize))] HQ ;

[0079] In the formula, LQ1 represents the first low-quality face image, HQ represents the target face image, Blur represents Gaussian blur or mean blur, Noise represents Gaussian noise or Poisson noise, JPEG represents compressed noise, Resize represents random upsampling and downsampling, RC (Random-Choose) represents randomly selecting at least one degradation method, and RS represents randomly sorting the selected degradation methods.

[0080] By performing the above face image degradation operations on HQ, we can construct low-quality face images LQ1 with a wide variety of degradation types, which can be used for model training to enhance the model's generalization ability.

[0081] In one embodiment, the step of noise perturbation of the first low-quality face image to simulate changes in face noise over time includes,

[0082] Fine-tuning is performed on each degradation method in Gaussian blur or mean blur, Gaussian noise or Poisson noise, compressed noise, and random upsampling and downsampling.

[0083] Randomly select at least two degradation methods from the following: fine-tuned Gaussian blur or mean blur, fine-tuned Gaussian noise or Poisson noise, fine-tuned compressed noise, and fine-tuned random upsampling and downsampling, and randomly sort the selected degradation methods to obtain the second image degradation sequence.

[0084] The first low-quality face image is subjected to noise perturbation using the second image degradation sequence.

[0085] The degraded parameters in LQ1 are fine-tuned to construct LQ2, which serves as another type of face pairing data. LQ2 and LQ1 have independent random noise locations and slightly varying noise intensities, enabling them to simulate changes in face noise across different frames in a time series.

[0086] In one embodiment, the following formula is used to perform noise perturbation on the first low-quality face image:

[0087] LQ2=[RS(RC(Blur+α,Noise+β,JPEG+γ,Resize+δ))] LQ1 ;

[0088] In the formula, LQ2 represents the second low-quality face image, LQ1 represents the first low-quality face image, Blur represents Gaussian blur or mean blur, Noise represents Gaussian noise or Poisson noise, JPEG represents compressed noise, Resize represents random upsampling or downsampling, RC represents randomly selecting at least one degradation method, RS represents randomly sorting the selected degradation methods, and α, β, γ and δ all represent small variations in noise in different frames of the simulated video.

[0089] By performing the above face image degradation operations on LQ1, we can construct LQ2, a low-quality face image with a wide variety of degradation types, for model training to enhance the model's generalization ability.

[0090] In one embodiment, the target loss function includes a face reconstruction loss function for supervising the quality of face reconstruction, the expression of which includes,

[0091]

[0092] in, Let G(x) represent the face reconstruction loss function, x represent the first low-quality face image or the second low-quality face image, G(x) represent the output of the convolutional neural network model, HQ represent the target face image, L1 represent the difference pixels between the first enhanced face image or the second enhanced face image and the target face image, and L1 represents the first or second norm. p λ1 represents the difference between different layers of features in the first or second enhanced face image and the target face image, i.e., the perceptual loss. It is typically obtained by extracting features from different layers of the two images using VGG16 or VGG19 and then subtracting them. λ1 represents the weight coefficient of the perceptual loss, and L... g λ represents the GAN loss, and λ2 represents the weight coefficient of the GAN loss.

[0093] The face reconstruction loss supervises the face reconstruction quality, aiming to make the model output G(LQ1)\G(LQ2) consistent with the high-quality face image HQ.

[0094] In one embodiment, the target loss function includes a face consistency loss function, the expression of which includes,

[0095]

[0096] G(LQ1)==G(LQ2);

[0097] in, Let G(LQ1) represent the face consistency loss function, G(LQ2) represent the first enhanced face image, G(LQ2) represent the second enhanced face image, and L1' represent the difference pixels between the first enhanced face image and the second enhanced face image. p ' represents the difference between different layers of features in the first enhanced face image and the second enhanced face image, i.e., the consistency loss, L1' and L... p The goal is to make the model's reconstruction results for LQ1 and LQ2 as similar as possible, where λ1' represents the weighting coefficient of the consistency loss, L... id The face recognition ID loss represents the first enhanced face image and the second enhanced face image. Typically, face recognition is performed on the two face images using networks such as ArcFace, and the consistency of the output vectors is compared. The purpose is to control the face output by the model under different noise levels to be as consistent as possible. λ3 represents the weight coefficient of the face recognition loss.

[0098] Face consistency loss is used to supervise the reconstruction of identical faces from two different noise levels, thus maintaining temporal face consistency.

[0099] In one embodiment, the convolutional neural network model adopts a UNet-Like structure, which includes:

[0100] The face feature extraction module is used to perform multi-scale feature extraction on the first low-quality face image and the second low-quality face image respectively to obtain the first feature face image and the second feature face image;

[0101] The face reconstruction module is used to perform multi-scale reconstruction on the first feature face image and the second feature face image to obtain the first enhanced face image and the second enhanced face image.

[0102] Specifically, refer to Figure 2 The UNet-Like architecture consists of a first convolutional layer, a second convolutional layer (stride=2), and a third convolutional layer. The first convolutional layer has three layers and is used to extract shallow features. The second convolutional layer has three layers with different dimensions, and these three layers with different dimensions are used to extract features at three different scales (large, medium, and small). The third convolutional layer includes an upsampling layer and a convolutional layer. The third convolutional layer has three layers with different dimensions and is used to reconstruct features at different scales.

[0103] In this embodiment, the first convolutional layer and the second convolutional layer constitute a face feature extraction module (encoding layer), which is used to perform multi-scale feature extraction on the first low-quality face image LQ1 and the second low-quality face image LQ2; the first convolutional layer and the third convolutional layer constitute a face reconstruction module (decoding layer), which is used for multi-scale face image reconstruction to obtain the first enhanced face image G(LQ1) and the second enhanced face image G(LQ2).

[0104] First, the input first and second low-quality face images are processed through a first convolutional layer to extract shallow features. These shallow features flow in two directions: one is to the layer before the first convolutional layer of the face reconstruction module, and the other is to the second convolutional layer of the face feature extraction module for small-scale feature extraction. The extracted small-scale features are then passed to the next medium-scale second convolutional layer for further feature extraction. Simultaneously, the extracted small-scale features are also passed to the third convolutional layer of the face reconstruction module at the same dimension. The extracted medium-scale features are then passed to the next large-scale second convolutional layer for further feature extraction. Simultaneously, the extracted medium-scale features are also passed to the third convolutional layer of the face reconstruction module at the same dimension. The three convolutional layers extract large-scale features, which are then passed to the third convolutional layer of the face reconstruction module in the same dimension. The first third convolutional layer fuses shallow features with features obtained from the second convolutional layer to obtain the first reconstructed feature. The second third convolutional layer fuses the first reconstructed feature with features obtained from the second convolutional layer to obtain the second reconstructed feature in the next dimension. The third third convolutional layer fuses the second reconstructed feature with features obtained from the second convolutional layer to obtain the third reconstructed feature in the next dimension. Finally, the third reconstructed feature passes through two first convolutional layers to complete the reconstruction of the entire face image in a single frame, resulting in the first enhanced face image G(LQ1) and the second enhanced face image G(LQ2).

[0105] In summary, a video conferencing face quality enhancement method based on single-frame images acquires face images with a preset resolution to obtain target face images, which are then used as the training target for the model, making the training results tend towards high-quality face images. The target face image is then degraded to obtain a first low-quality face image to simulate noise variations in a single frame. Furthermore, the first low-quality face image is subjected to noise perturbation to obtain a second low-quality face image to simulate noise variations in another single frame. The noise variations in the second low-quality face image are not significantly different from those in the first low-quality face image. Similarly, the first low-quality face image and the second low-quality face image are input into a pre-constructed convolutional neural network model, and face enhancement training is performed simultaneously using a preset target loss function. The face reconstruction quality and temporal face consistency under different noise conditions are supervised to obtain the target model to enhance the single-frame face image to be enhanced, so as to enhance the model's ability to restore the face in a consistent manner when dealing with different noise, thereby ensuring the stability of the video frames and reducing the impact of the temporal stability problem of single-frame face image quality enhancement on the face image quality of video conferencing, thus improving the face image quality in video conferencing.

[0106] This application presents a video conferencing face image quality enhancement scheme based on single-frame images. It simulates the jump in temporal noise between consecutive frames of faces in a video conference by changing the location and intensity of noise in the same high-definition face image, thereby improving the stability and robustness of the single-frame face image quality enhancement effect.

[0107] This application proposes a joint loss function and training scheme for face reconstruction and face consistency. By simulating video temporal noise and constructing a joint loss function, a face reconstruction loss function is built. The quality of face reconstruction G(LQ1) and G(LQ2) is supervised to approximate HQ images, and the consistency of G(LQ1) and G(LQ2) is supervised to ensure reconstruction stability. The optimization yields a single-frame face image quality enhancement model that meets the face quality and stability requirements in video conferencing. This allows the scheme to maintain face stability in the video while restoring face clarity, thus improving visual effects.

[0108] This application is based on the image2image method. By inputting a single low-quality face image into the target model of this application, a single-frame enhanced face image with a quality similar to that of a high-definition face image can be obtained, which can meet the requirements of practical real-time applications.

[0109] Compared to existing technologies, this application has lower hardware requirements, eliminating the need to process two frames of images simultaneously. It incorporates a temporal module and fuses facial features from multiple frames into the model structure, resulting in lower computational complexity. Furthermore, this application processes only a single frame, reducing the time required for introducing the temporal module and fusing facial features from multiple frames, and improving memory usage.

[0110] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0111] Reference Figure 3 This application also provides a video conferencing face quality enhancement device based on a single-frame image, which corresponds one-to-one with the video conferencing face quality enhancement method based on a single-frame image described in the above embodiments. This single-frame image-based video conferencing face quality enhancement device includes...

[0112] The target image module is used to acquire face images with a resolution that meets the preset resolution conditions to obtain the target face image;

[0113] The sample image module is used to degrade the target face image to obtain a first low-quality face image; and to apply noise perturbation to the first low-quality face image to obtain a second low-quality face image.

[0114] The training module is used to input the first low-quality face image and the second low-quality face image into a pre-constructed convolutional neural network model, and simultaneously perform face enhancement training using a preset target loss function, outputting a first enhanced face image and a second enhanced face image, wherein the target loss function is used to supervise the face reconstruction quality and temporal face consistency under different noise levels.

[0115] The target model module is used to take the convolutional neural network model as the target model when the convolutional neural network model converges. The target model is used to perform image enhancement on the single-frame face image to be enhanced.

[0116] For specific limitations regarding a video conferencing face quality enhancement device based on a single-frame image, please refer to the limitations of a video conferencing face quality enhancement method based on a single-frame image mentioned above, which will not be repeated here.

[0117] The various modules in the aforementioned video conferencing face image quality enhancement device based on single-frame images can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0118] In one embodiment, a computer device is provided, which may be a server. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements any of the above-described methods for enhancing facial image quality in video conferencing based on single-frame images.

[0119] In one embodiment, a computer-readable storage medium is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-described methods for enhancing the face quality in video conferencing based on a single-frame image.

[0120] In one embodiment, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, implements any of the above-described methods for enhancing face quality in video conferencing based on a single-frame image.

[0121] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. When executed, the computer program may include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0122] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A method for enhancing facial image quality in video conferencing based on a single frame image, characterized in that, Includes the following steps, Acquire face images with a resolution that meets the preset resolution conditions to obtain the target face image; The target face image is degraded to obtain a first low-quality face image; as well as The first low-quality face image is subjected to noise perturbation to obtain a second low-quality face image; The first low-quality face image and the second low-quality face image are input into a pre-constructed convolutional neural network model, and face enhancement training is performed simultaneously using a preset target loss function. The output is a first enhanced face image and a second enhanced face image. The target loss function is used to supervise the face reconstruction quality and temporal face consistency under different noise levels. The target loss function includes a face reconstruction loss function for supervising face reconstruction quality, and the expression of the face reconstruction loss function includes... in, Let G(x) represent the face reconstruction loss function, x represent the first low-quality face image or the second low-quality face image, G(x) represent the output of the convolutional neural network model, HQ represent the target face image, L1 represent the difference pixels between the first enhanced face image or the second enhanced face image and the target face image, and L... p λ represents the difference between different layers of features of the first enhanced face image or the second enhanced face image and the target face image, where the difference between the different layers of features is the perceptual loss, λ1 represents the weight coefficient of the perceptual loss, and L g λ represents the GAN loss, and λ2 represents the weight coefficient of the GAN loss. The target loss function includes a face consistency loss function, and the expression of the face consistency loss function includes, in, Let G(LQ1) represent the face consistency loss function, G(LQ2) represent the first enhanced face image, G(LQ2) represent the second enhanced face image, and L1' represent the difference pixels between the first enhanced face image and the second enhanced face image. p ' represents the difference between different layers of features in the first enhanced face image and the second enhanced face image, i.e., the consistency loss, λ1' represents the weight coefficient of the consistency loss, and L id λ3 represents the face recognition loss of the first enhanced face image and the second enhanced face image, and λ3 represents the weight coefficient of the face recognition loss. When the convolutional neural network model converges, the current convolutional neural network model is used as the target model, which is used to perform image enhancement on the single-frame face image to be enhanced.

2. The video conferencing face quality enhancement method based on a single-frame image according to claim 1, characterized in that, The steps for degrading the target face image include: Randomly select at least one degradation method from Gaussian blur or mean blur, Gaussian noise or Poisson noise, compressed noise and random upsampling and downsampling, and randomly sort the selected degradation methods to obtain the first image degradation sequence; The target face image is degraded using the first image degradation sequence.

3. The video conferencing face quality enhancement method based on a single-frame image according to claim 2, characterized in that, The following formula is used to degrade the target face image. LQ1=[RS(RC(Blur,Noise,JPEG,Resize))] HQ ; In the formula, LQ1 represents the first low-quality face image, HQ represents the target face image, Blur represents Gaussian blur or mean blur, Noise represents Gaussian noise or Poisson noise, JPEG represents compressed noise, Resize represents random upsampling or downsampling, RC represents randomly selecting at least one degradation method, and RS represents randomly sorting the selected degradation methods.

4. The video conferencing face quality enhancement method based on a single-frame image according to claim 2, characterized in that, The step of noise perturbation of the first low-quality face image includes: Fine-tuning is performed on each degradation method in Gaussian blur or mean blur, Gaussian noise or Poisson noise, compressed noise, and random upsampling and downsampling. Randomly select at least two degradation methods from the following: fine-tuned Gaussian blur or mean blur, fine-tuned Gaussian noise or Poisson noise, fine-tuned compressed noise, and fine-tuned random upsampling and downsampling, and randomly sort the selected degradation methods to obtain the second image degradation sequence. The first low-quality face image is subjected to noise perturbation using the second image degradation sequence.

5. The video conferencing face quality enhancement method based on a single-frame image according to claim 4, characterized in that, When applying noise perturbation to the first low-quality face image, the following formula is used: LQ2=[RS(RC(Blur+α,Noise+β,JPEG+γ,Resize+δ))] LQ1 ; In the formula, LQ2 represents the second low-quality face image, LQ1 represents the first low-quality face image, Blur represents Gaussian blur or mean blur, Noise represents Gaussian noise or Poisson noise, JPEG represents compressed noise, Resize represents random upsampling or downsampling, RC represents randomly selecting at least one degradation method, RS represents randomly sorting the selected degradation methods, and α, β, γ and δ all represent small variations in noise in different frames of the simulated video.

6. The method for enhancing face image quality in video conferencing based on a single frame image according to claim 1, characterized in that, The convolutional neural network model adopts the UNet-Like structure, which includes: The face feature extraction module is used to perform multi-scale feature extraction on the first low-quality face image and the second low-quality face image respectively to obtain the first feature face image and the second feature face image; The face reconstruction module is used to perform multi-scale reconstruction on the first feature face image and the second feature face image to obtain the first enhanced face image and the second enhanced face image.

7. A video conferencing face quality enhancement device based on a single frame image, characterized in that, A method for enhancing face quality in video conferencing based on a single frame image as described in any one of claims 1 to 6, comprising: The target image module is used to acquire face images with a resolution that meets the preset resolution conditions to obtain the target face image; The sample image module is used to degrade the target face image to obtain a first low-quality face image; and to apply noise perturbation to the first low-quality face image to obtain a second low-quality face image. The training module is used to input the first low-quality face image and the second low-quality face image into a pre-constructed convolutional neural network model, and simultaneously perform face enhancement training using a preset target loss function, outputting a first enhanced face image and a second enhanced face image, wherein the target loss function is used to supervise the face reconstruction quality and temporal face consistency under different noise levels. The target model module is used to take the convolutional neural network model as the target model when the convolutional neural network model converges. The target model is used to perform image enhancement on the single-frame face image to be enhanced.

8. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

Images