Face image enhancement device and method applied to video conference, equipment and storage medium
By using the devices and methods of image acquisition module, feature extraction module, degradation removal module and face repair module in video conferencing, the problems of coherence and consistency of face repair in video conferencing are solved, and a more stable video face repair effect is achieved.
Patent Information
- Application Number
- CN202411988129.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-05-02
AI Technical Summary
In the prior art, when face repair in video conferencing scenarios, it is difficult to ensure the consistency and consistency of the repaired face in the video, especially when expressions or actions change or when timing jump noises exist.
Using an apparatus and method including an image acquisition module, a feature extraction module, a degradation removal module and a face repair module, the final enhanced image is obtained by fusing shallow feature extraction and pre-enhanced feature information of the previous frame image by fusion of shallow feature extraction and timing fusion and multi-scale feature extraction, and then reconstruction and feature recovery based on the feature vector of the smallest scale to obtain the final enhanced image.
The coherence and consistency of faces in video conferencing is improved, and time-series fusion and multi-scale feature processing are effectively removed to achieve more stable video face repair.
Smart Images
Figure CN119919974A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a facial image enhancement device, method, equipment and storage medium for video conferencing. Background Art
[0002] Face restoration technology based on deep learning aims to restore poor quality and polluted face images into clear high-quality images. This technology compares low-quality and high-quality face images to learn the conversion rules between them. But in real life, it is difficult to find low-quality and high-quality images of the same scene, so researchers usually simulate low-quality images by adding blur, noise or resizing to high-quality images. Then, these paired images are input into the convolutional neural network for training, so that the neural network can learn to convert low-quality images into high-quality images.
[0003] Most current face restoration technologies are based on processing single-frame face images. When face restoration is used in video conferencing scenarios, this method means processing each frame of the video separately and fitting the face restoration results of each frame to the original position. However, a video is composed of a series of continuous images (frames), and there are subtle differences between each frame, especially when the facial expression or movement changes or there is timing jitter noise in the scene. Therefore, it is difficult to ensure the coherence and consistency of the restored face in the video by frame-by-frame processing. Summary of the invention
[0004] To this end, the embodiments of the present application provide a facial image enhancement device, method, equipment and storage medium applied to video conferencing, which improves the continuity of faces in video conferencing.
[0005] In a first aspect, the present application provides a facial image enhancement device for use in video conferencing.
[0006] This application is achieved through the following technical solutions:
[0007] A face image enhancement device for video conferencing, comprising:
[0008] An image acquisition module is configured to acquire a sequence of images to be enhanced;
[0009] The feature extraction module is configured to extract shallow feature information from the i-th frame image in the image sequence to be enhanced, fuse the shallow feature information of the i-th frame image with the pre-enhancement feature information of the introduced i-1-th frame image to obtain time-series fused feature information, and extract multiple feature vectors of different scales from the time-series fused feature information;
[0010] The degradation removal module is configured to introduce multiple feature vectors of different scales output by the feature extraction module, start reconstruction based on the feature vector of the smallest scale, and obtain temporally stable pre-enhancement feature information of the i-th frame image;
[0011] The face restoration module is configured to introduce the pre-enhanced feature information of the i-th frame image of the degradation removal module, start reconstruction based on the minimum-scale feature vector, and obtain a final enhanced image.
[0012] In a preferred example of the present application, it can be further set as follows:
[0013] The feature extraction module is configured to extract features from the i-th frame image in the image sequence to be enhanced to obtain shallow feature information of the i-th frame image;
[0014] The pre-enhanced feature information of the i-1th frame image introduced from the degradation removal module is fused into the shallow feature information of the i-th frame image by optical flow alignment to obtain the temporal fusion feature information of the i-th frame image;
[0015] The temporal fusion feature information of the i-th frame image is subjected to convolution processing with a preset step length to obtain feature information of multiple different scales.
[0016] In a preferred example of the present application, it can be further configured that the feature extraction module is configured as follows:
[0017] Use the optical flow algorithm to extract the optical flow information between the pre-enhanced feature information of the i-1th frame image and the shallow feature information of the i-th frame image;
[0018] Based on the optical flow information, spatially transform the shallow feature information of the i-th frame image and the pre-enhanced feature information of the i-1-th frame image to obtain aligned shallow feature information of the i-th frame image;
[0019] The aligned shallow feature information of the i-th frame image is concatenated with the pre-enhanced feature information of the i-1-th frame image to obtain the temporal fusion feature information of the i-th frame image.
[0020] In a preferred example of the present application, it can be further set that the feature extraction module is configured to: perform three convolution operations on the time series fusion feature information according to a preset step size to obtain a feature vector of a first scale, a feature vector of a second scale and a feature vector of a third scale respectively.
[0021] In a preferred example of the present application, it can be further configured that the degradation removal module is configured to receive the feature vector of the third scale of the i-th frame image output by the feature extraction module, perform a first feature recovery, and combine the first feature recovery result with the feature vector of the second scale introduced from the feature extraction module to obtain a complete reconstructed feature vector of the second scale;
[0022] Performing a second feature recovery on the reconstructed feature vector of the second scale, combining the second feature recovery result with the feature vector of the first scale introduced from the feature extraction module, to obtain a complete reconstructed feature vector of the first scale;
[0023] The reconstructed feature vector of the first scale is subjected to a third feature recovery, and the third feature recovery result is combined with the temporal fusion feature information introduced from the feature extraction module to obtain pre-enhanced feature information.
[0024] In a preferred example of the present application, it can be further configured that the face restoration module is configured as follows:
[0025] Receiving the feature vector of the third scale of the i-th frame image output by the feature extraction module, performing a first feature recovery, combining the first feature recovery result with the reconstructed feature vector of the second scale introduced from the degradation removal module, and obtaining an enhanced feature vector of the second scale;
[0026] Performing a second feature recovery on the enhanced feature vector of the second scale, combining the second feature recovery result with the reconstructed feature vector of the first scale introduced from the degradation removal module, to obtain an enhanced feature vector of the first scale;
[0027] The enhanced feature vector of the first scale is restored for the third time, and the result of the third feature restoration is combined with the pre-enhanced feature information introduced from the degradation removal module to output the final enhanced image of the i-th frame image.
[0028] In a preferred example of the present application, it can be further configured that the feature extraction module, the degradation removal module and the face restoration module are trained based on the VGG16 model or the VGG19 model.
[0029] In a second aspect, the present application provides a facial image enhancement method applied to video conferencing.
[0030] This application is achieved through the following technical solutions:
[0031] A face image enhancement method for video conferencing, comprising:
[0032] Obtaining an image sequence to be enhanced for a video conference;
[0033] Inputting the i-th frame image of the image sequence to be enhanced into the feature extraction module to extract shallow feature information of the i-th frame image, and fusing the shallow feature information of the i-th frame image with the pre-enhancement feature information of the introduced i-1-th frame image to obtain time-series fused feature information, and extracting multiple feature vectors of different scales from the time-series fused feature information;
[0034] Reconstruction is started from the smallest-scale feature vector, and a feature vector of the same scale as the reconstruction result is introduced into the reconstruction result of each scale, so as to obtain the temporally stable pre-enhancement feature information of the i-th frame image;
[0035] Feature recovery is performed starting from the feature vector of the smallest scale, and pre-enhanced feature information of the same scale as the feature recovery result is introduced into the feature recovery result of each scale to obtain the final enhanced image of the i-th frame image.
[0036] On the third aspect, the present application is realized through the following technical solutions:
[0037] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned method for enhancing facial images applied to video conferencing are implemented.
[0038] In a fourth aspect, the present application provides a computer-readable storage medium.
[0039] This application is achieved through the following technical solutions:
[0040] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned face image enhancement method applied to video conferencing are implemented.
[0041] In summary, compared with the prior art, the technical solution provided in the embodiment of the present application has at least the following beneficial effects:
[0042] The facial image enhancement device proposed in the present application includes an image acquisition module, which is configured to acquire a sequence of images to be enhanced; a feature extraction module, which is configured to extract shallow feature information from the i-th frame image in the sequence of images to be enhanced, fuse the shallow feature information of the i-th frame image with the pre-enhanced feature information of the introduced i-1-th frame image to obtain temporal fusion feature information, and extract multiple feature vectors of different scales from the temporal fusion feature information; a degradation removal module, which is configured to introduce multiple feature vectors of different scales output by the feature extraction module, start reconstruction based on the feature vector of the smallest scale, and obtain temporally stable pre-enhanced feature information of the i-th frame image; a face restoration module, which is configured to introduce the pre-enhanced feature information of the i-th frame image of the degradation removal module, start reconstruction based on the feature vector of the smallest scale, and obtain the final enhanced image. The features of the previous frame image are introduced in the feature extraction module, so that the feature alignment of faces in different frames can be achieved; a dual module of the degradation removal module and the face restoration module is constructed to achieve further temporally stable video face restoration under the premise of removing temporal noise. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 A schematic diagram of the structure of a face image enhancement device for video conferencing provided in one embodiment of the present application;
[0044] Figure 2 A schematic diagram of a flow chart of a facial image enhancement method applied to a video conference provided in an embodiment of the present application;
[0045] Description of reference numerals:
[0046] Image acquisition module 10, feature extraction module 20, degradation removal module 30, face restoration module 40. DETAILED DESCRIPTION
[0047] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make modifications to the present embodiment without any creative contribution as needed, but such modifications are protected by the patent law as long as they are within the scope of the claims of the present application.
[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0049] In addition, the term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application, unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.
[0050] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with basically the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on quantity and execution order.
[0051] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0052] The embodiments of the present application are further described in detail below in conjunction with the drawings in the specification.
[0053] like Figure 1 As shown, a face image enhancement device for video conferencing provided by the first exemplary embodiment of the present application includes:
[0054] The image acquisition module 10 is configured to acquire the image sequence to be enhanced. The image acquisition module 10 establishes a connection with the video conferencing system. Once the connection is established, the image acquisition module 10 acquires the image data of the video conference of a preset duration in real time at a certain frame rate (e.g., 30 frames per second). The image sequence to be enhanced is composed of continuous image frames. P i lq Represents the i-th frame image in the image sequence to be enhanced.
[0055] The feature extraction module 20 is configured to extract shallow feature information from the i-th frame image in the image sequence to be enhanced, fuse the shallow feature information of the i-th frame image with the pre-enhanced feature information of the introduced i-1-th frame image to obtain time-series fusion feature information, and extract multiple feature vectors of different scales from the time-series fusion feature information. Among them, shallow feature information refers to local features in the image, including edge features, corner features, texture features, etc. The pre-enhanced feature information of the i-1-th frame image is the pre-enhanced feature information of the clean face after preliminary degradation removal processing by the degradation removal module 30. The feature extraction module 20 can improve the coherence of the face in the video by fusing the shallow feature information of the current frame image (i.e., the i-th frame image) with the pre-enhanced feature information of the i-1-th frame image. The feature extraction module 20 can extract features by the convolution layer in the convolutional neural network.
[0056] The degradation removal module 30 is configured to introduce multiple feature vectors of different scales output by the feature extraction module 20, and reconstruct scale by scale starting from the feature vector of the smallest scale to obtain the temporally stable pre-enhanced feature information of the i-th frame image. Specifically, the input end of the degradation removal module 30 is connected to the output end of the feature extraction module 20, and is used to receive the feature information of the i-th frame image extracted by the feature extraction module 20, and perform multiple upsampling and convolution operations on the feature information, and output the temporally stable pre-enhanced feature information of the i-th frame image containing a clean temporal face. At the same time, the pre-enhanced feature information of the i-th frame image will be transmitted back to the feature extraction module 20, and the feature extraction module 20 processes the i+1-th frame image and is fused with the shallow feature information of the i+1-th frame image. In addition, when the degradation removal module 30 reconstructs the feature information obtained from the feature extraction module 20 size by size, it will transmit the pre-enhanced feature information of each size generated in the middle to the face restoration module 40.
[0057] The face restoration module 40 is configured to introduce the pre-enhanced feature information of the i-th frame image of the degradation removal module, reconstruct multiple feature vectors of different scales of the i-th frame image of the feature extraction module, and obtain a final enhanced image.
[0058] It should be noted that the structure of the face restoration module 40 is the same as that of the degradation removal module 30 .
[0059] In one embodiment, the feature extraction module is configured to extract features from the i-th frame image in the image sequence to be enhanced to obtain shallow feature information of the i-th frame image;
[0060] The pre-enhanced feature information of the i-1th frame image introduced from the degradation removal module is fused into the shallow feature information of the i-th frame image by optical flow alignment to obtain the temporal fusion feature information of the i-th frame image;
[0061] The temporal fusion feature information of the i-th frame image is convolved with a preset step length to obtain multiple feature vectors of different scales. The pre-enhanced feature information of the i-1th frame image output from the degradation removal module contains the clean face features after preliminary processing, and the pre-enhanced feature information of the i-1th frame image is fused (Warp) into the shallow feature information of the i-th frame image through optical flow alignment.
[0062] In one embodiment, the feature extraction module is configured to: use an optical flow algorithm to extract optical flow information between the pre-enhanced feature information of the i-1th frame image and the shallow feature information of the i-th frame image; based on the optical flow information, spatially transform the shallow feature information of the i-th frame image and the pre-enhanced feature information of the i-1th frame image to obtain the aligned shallow feature information of the i-th frame image; feature splicing the aligned shallow feature information of the i-th frame image and the pre-enhanced feature information of the i-1th frame image to obtain the temporal fusion feature information of the i-th frame image.
[0063] In this embodiment, the optical flow algorithm used in the feature extraction module can be the Lucas-Kanade method or the Horn-Schunck method. In the actual calculation process, parameters such as window size and number of iterations can be set according to the optical flow algorithm used. The optical flow information includes the moving direction and moving distance for indicating the pixel point from the pre-enhanced feature information of the i-1 frame image to the shallow feature information of the i-1 frame image. Based on the moving direction and moving distance of the above-mentioned pixel points, the pixel points in the shallow feature information of the i-1 frame image are displaced by an interpolation algorithm to align them with the pixel points in the pre-enhanced feature information of the i-1 frame image. The aligned shallow feature information of the i-1 frame image and the pre-enhanced feature information of the i-1 frame image are stacked on the channel to form a new feature map, which is the time-series fusion feature information of the i-1 frame image. It should be noted that after obtaining the time-series fusion feature information, the time-series fusion feature information can also be enhanced by a convolution layer with the same size as the convolution layer in the feature extraction module.
[0064] In one embodiment, the feature extraction module is configured to: perform three convolution operations on the time series fusion feature information according to a preset step size to obtain a feature vector of a first scale, a feature vector of a second scale, and a feature vector of a third scale, respectively.
[0065] In actual implementation, the number of network channels can be deepened on the basis of reducing the size of the feature map through a convolution layer with a step size of 2, so as to extract high-level deep features in the face image. Specifically, a convolution layer is used to convolve the temporal fusion feature information to obtain a feature vector of the first scale; a convolution layer is used to convolve the feature vector of the first scale to obtain a feature vector of the second scale; a convolution layer is used to convolve the feature vector of the second scale to obtain a feature vector of the third scale.
[0066] In one embodiment, the degradation removal module is configured to receive the feature vector of the third scale of the i-th frame image output by the feature extraction module, perform a first feature recovery, and combine the first feature recovery result with the feature vector of the second scale introduced from the feature extraction module to obtain a complete reconstructed feature vector of the second scale;
[0067] Performing a second feature recovery on the reconstructed feature vector of the second scale, combining the second feature recovery result with the feature vector of the first scale introduced from the feature extraction module, to obtain a complete reconstructed feature vector of the first scale;
[0068] The reconstructed feature vector of the first scale is subjected to a third feature recovery, and the feature recovery result is combined with the temporal fusion feature information introduced from the feature extraction module to obtain the pre-enhanced feature information.
[0069] In one embodiment, the face restoration module is configured to: receive the feature vector of the third scale of the i-th frame image output by the feature extraction module, perform a first feature restoration, and combine the first feature restoration result with the reconstructed feature vector of the second scale introduced from the degradation removal module to obtain an enhanced feature vector of the second scale;
[0070] Performing a second feature recovery on the enhanced feature vector of the second scale, and combining the second feature recovery result with the reconstructed feature vector of the first scale introduced from the degradation removal module to obtain an enhanced feature vector of the first scale;
[0071] The enhanced feature vector of the first scale is subjected to a third feature recovery, and the third feature recovery result is combined with the pre-enhanced feature information introduced from the degradation removal module to output the final enhanced image.
[0072] In one embodiment, the feature extraction module, the degradation removal module and the face restoration module of the face enhancement device are trained based on the VGG16 model or the VGG19 model.
[0073] It should be noted that the model training of the face enhancement device is divided into two stages. In the first stage, the feature extraction module and the degradation removal module are trained. After the feature extraction module and the degradation removal module converge, the model parameters of the feature extraction module and the degradation removal module are frozen; in the second stage, the face restoration module is trained. After the face restoration module converges, the model parameters of the face restoration module are frozen.
[0074] First, construct the data for training the model. Specifically, after obtaining the high-definition face image in the conference video scene, the corresponding low-quality face image is usually simulated by one or more of blur, noise, random up and down sampling, JPEG compression, etc., so the low-quality face image can be constructed by the following formula:
[0075] P lq -1=[Random-Shuffle(Random-Choose(Blur,Noise,JPEG,Resize)]P hq ,
[0076] Where P hq represents the high-definition face image captured by the local video conferencing system, P lq -1 indicates a low-quality face image after random noise processing, Blur indicates Gaussian blur or mean blur on the image, Noise indicates Gaussian noise or Poisson noise on the image, JPEG indicates compressed noise on the image, Resize indicates random up-sampling and down-sampling of the image, Random-Choose indicates randomly selecting one or more of the above noises, and Random-Shuffle indicates randomly sorting the above selected noises. Through the above face image degradation methods, a variety of low-quality face images P with degradation can be constructed. lq -1.
[0077] In order to simulate the change of face noise in time series, the low-quality face image P lq -1 to construct the low-quality face image P of the next frame lq -2:
[0078] P lq -2=[Random-Shuffle(Random-Choose(Blur+a,Noise+b,JPEG+c,Resize+d)]P hq ,
[0079] Where a, b, c, and d represent the small changes added to the noise, which are used to simulate the small changes in noise in different frames in the video. lq -1 and P lq-2 represents the image degradation process between the previous and next frames in the same time sequence. Similarly, we construct a low-quality face image sequence P lq -1,P lq -2, ..., P lq -n, used as training data for training the model.
[0080] The training data is input into the VGG16 model to let the VGG16 model learn. The purpose of the first stage is to restore the low-quality face image sequence that has been polluted by noise to obtain temporally stable face features. During the training process, the first loss function is constructed. The first loss function is L 1 Norm:
[0081]
[0082] Among them, denoisedP ld represents the degraded face image output by the model, P hq Represents the high-definition face image captured by the local video conferencing system. By comparing the loss function of the enhanced face image output by the model and the high-definition face image, back-propagating the gradient, and updating the model parameters, when the loss function stabilizes and no longer decreases, it means that the model has converged. At this time, freeze the model parameters of the feature extraction module and the degradation removal module.
[0083] The purpose of the second stage is to further restore the facial details of the clean face after the degradation removal. First of all, it is necessary to ensure the stability of the degradation removal module. Therefore, the convolution parameters of the feature extraction module and the degradation removal module are frozen, and only the convolution parameters of the face restoration module are optimized through the loss function.
[0084]
[0085] Where P hq represents the high-definition face image captured by the local video conferencing system, enhancedP lq Represents the enhanced face image output by the model, L 1 Represents the norm, representing the difference between the pixels of the two images, L p Represents perceptual loss, which means that VGG16 extracts features from different layers and makes a difference. 1 Represents the weight coefficient of perceptual loss, L g represents the GAN loss (Generative Adversarial Network), λ 2 is the weight coefficient of the loss. The above three loss functions supervise the generation quality of face images. Back propagate the gradient and update the model parameters. When the loss function is stable and no longer decreases, freeze the convolution parameters of the face restoration module. At this point, the trained face image enhancement model can be used for face image enhancement processing.
[0086] The method of obtaining a face image enhancement device by training the VGG19 model is similar to the process of the above-mentioned VGG16 model, and will not be repeated here.
[0087] like Figure 2 As shown, the embodiment of the present application also provides a face image enhancement method applied to video conferencing, and the specific method includes:
[0088] S10: Obtaining a sequence of images to be enhanced for a video conference;
[0089] S20: Inputting the i-th frame image of the image sequence to be enhanced into the feature extraction module to extract shallow feature information of the i-th frame image, and fusing the shallow feature information of the i-th frame image with the introduced pre-enhancement feature information of the i-1-th frame image to obtain time series fusion feature information, and extracting multiple feature vectors of different scales from the time series fusion feature information;
[0090] S30: Reconstructing from the smallest-scale feature vector, and introducing a feature vector of the same scale as the reconstruction result into the reconstruction result of each scale, so as to obtain temporally stable pre-enhancement feature information of the i-th frame image;
[0091] S40: Starting from the feature vector of the smallest scale, feature recovery is performed, and pre-enhanced feature information of the same scale as the feature recovery result is introduced into the feature recovery result of each scale, so as to obtain a final enhanced image of the i-th frame image.
[0092] For the specific limitations of the facial image enhancement method applied to video conferencing provided in this embodiment, please refer to the embodiment of the facial image enhancement device applied to video conferencing above, which will not be repeated here.
[0093] An embodiment of the present application provides a computer device, which may include a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the processor executes the steps of the face image enhancement method applied to video conferencing as in any of the above embodiments.
[0094] The working process, working details and technical effects of the computer device provided in this embodiment can be found in the above embodiment of the face image enhancement method applied to video conferencing, which will not be described in detail here.
[0095] The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the face image enhancement method applied to video conferencing as in any of the above embodiments are implemented. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive and / or a memory stick, etc. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
[0096] The working process, working details and technical effects of the computer-readable storage medium provided in this embodiment can be found in the embodiment of the facial image enhancement method applied to video conferencing above, and will not be described in detail here.
[0097] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0098] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0099] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system described in the present application is divided into different functional units or modules to complete all or part of the functions described above.
Claims
1. A facial image enhancement device for video conferencing, characterized in that: include: An image acquisition module is configured to acquire a sequence of images to be enhanced; The feature extraction module is configured to extract shallow feature information from the i-th frame image in the image sequence to be enhanced, fuse the shallow feature information of the i-th frame image with the pre-enhancement feature information of the introduced i-1-th frame image to obtain time-series fused feature information, and extract multiple feature vectors of different scales from the time-series fused feature information; The degradation removal module is configured to introduce multiple feature vectors of different scales output by the feature extraction module, and reconstruct from the feature vector of the smallest scale to obtain temporally stable pre-enhancement feature information of the i-th frame image; The face restoration module is configured to introduce the pre-enhanced feature information of the i-th frame image of the degradation removal module, and reconstruct it starting from the feature vector of the smallest scale to obtain the final enhanced image.
2. The facial image enhancement device for video conferencing according to claim 1, characterized in that: The feature extraction module is configured to extract features from the i-th frame image in the image sequence to be enhanced to obtain shallow feature information of the i-th frame image; The pre-enhanced feature information of the i-1th frame image introduced from the degradation removal module is fused into the shallow feature information of the i-th frame image by optical flow alignment to obtain the temporal fusion feature information of the i-th frame image; The temporal fusion feature information of the i-th frame image is subjected to convolution processing with a preset step length to obtain feature information of multiple different scales.
3. The facial image enhancement device for video conferencing according to claim 2, characterized in that: The feature extraction module is configured as follows: Use the optical flow algorithm to extract the optical flow information between the pre-enhanced feature information of the i-1th frame image and the shallow feature information of the i-th frame image; Based on the optical flow information, spatially transform the shallow feature information of the i-th frame image and the pre-enhanced feature information of the i-1-th frame image to obtain aligned shallow feature information of the i-th frame image; The aligned shallow feature information of the i-th frame image is concatenated with the pre-enhanced feature information of the i-1-th frame image to obtain the temporal fusion feature information of the i-th frame image.
4. The facial image enhancement device for video conferencing according to claim 1, characterized in that: The feature extraction module is configured to perform three convolution operations on the time series fusion feature information according to a preset step size to obtain a feature vector of a first scale, a feature vector of a second scale, and a feature vector of a third scale, respectively.
5. The facial image enhancement device for video conferencing according to claim 4, characterized in that: The degradation removal module is configured to receive the feature vector of the third scale of the i-th frame image output by the feature extraction module, perform a first feature recovery, and combine the first feature recovery result with the feature vector of the second scale introduced from the feature extraction module to obtain a complete reconstructed feature vector of the second scale; Performing a second feature recovery on the reconstructed feature vector of the second scale, combining the second feature recovery result with the feature vector of the first scale introduced from the feature extraction module, to obtain a complete reconstructed feature vector of the first scale; The reconstructed feature vector of the first scale is subjected to a third feature recovery, and the third feature recovery result is combined with the temporal fusion feature information introduced from the feature extraction module to obtain pre-enhanced feature information.
6. The facial image enhancement device for video conferencing according to claim 5, characterized in that: The face restoration module is configured as follows: Receiving the feature vector of the third scale of the i-th frame image output by the feature extraction module, performing a first feature recovery, combining the first feature recovery result with the reconstructed feature vector of the second scale introduced from the degradation removal module, and obtaining an enhanced feature vector of the second scale; Performing a second feature recovery on the enhanced feature vector of the second scale, combining the second feature recovery result with the reconstructed feature vector of the first scale introduced from the degradation removal module, to obtain an enhanced feature vector of the first scale; The enhanced feature vector of the first scale is restored for the third time, and the result of the third feature restoration is combined with the pre-enhanced feature information introduced from the degradation removal module to output the final enhanced image of the i-th frame image.
7. The facial image enhancement device for video conferencing according to any one of claims 1 to 6, characterized in that: The feature extraction module, the degradation removal module and the face restoration module are trained based on the VGG16 model or the VGG19 model.
8. A facial image enhancement method for video conferencing, characterized in that: The method comprises: Obtaining an image sequence to be enhanced for a video conference; Inputting the i-th frame image of the image sequence to be enhanced into the feature extraction module to extract shallow feature information of the i-th frame image, and fusing the shallow feature information of the i-th frame image with the pre-enhancement feature information of the introduced i-1-th frame image to obtain time-series fused feature information, and extracting multiple feature vectors of different scales from the time-series fused feature information; Reconstruction is started from the smallest-scale feature vector, and a feature vector of the same scale as the reconstruction result is introduced into the reconstruction result of each scale, so as to obtain the temporally stable pre-enhancement feature information of the i-th frame image; Feature recovery is performed starting from the feature vector of the smallest scale, and pre-enhanced feature information of the same scale as the feature recovery result is introduced into the feature recovery result of each scale to obtain the final enhanced image of the i-th frame image.
9. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to claim 8.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to claim 8 are implemented.