Face super-resolution method and system based on visual language prior, and computer equipment

Abstract

The invention discloses a human face super-resolution method based on visual language prior, and the method comprises the following steps: 1, sending a low-resolution human face image into a pre-trained visual-language large model, and extracting visual-language multivariate representation; 2, constructing a visual language prior assisted face super-resolution network, and fusing visual language prior information; and step 3, sending the low-resolution face image and the visual language multivariate representation extracted in the step 1 into the network in the step 2 to obtain a super-resolution result, and obtaining a recovered high-quality face image. Compared with the existing mainstream face super-resolution method (FSRNet, DIC, SISN, SFMNet, FaceFormer, WFEN), the face image recovered by the method provided by the invention is better in both objective evaluation index and subjective visual quality.

Description

Technical Field

[0001] The present invention belongs to the field of computer image technology, and specifically relates to a face super-resolution method, system and computer equipment based on visual language prior. Background Art

[0002] Face super-resolution technology aims to restore high-resolution face images from low-resolution face images. In low-cost cameras and non-ideal imaging environments, the captured face images are often of poor quality, which not only affects the visual experience, but may also have an adverse impact on downstream tasks such as face recognition and attribute analysis. Face super-resolution technology improves the quality of face images and thus enhances the performance of these downstream tasks. Therefore, its importance has become increasingly prominent in the past few decades.

[0003] In recent years, deep learning technology has developed rapidly, and face super-resolution methods based on deep learning have made significant progress. These methods often design effective network structures to directly learn the mapping from low-resolution face images to high-resolution face images. However, due to the difference in dimensions between high-resolution and low-resolution face images, a low-resolution face image may correspond to multiple different high-resolution faces. This feature increases the difficulty of the face super-resolution task. To this end, a strategy of introducing additional prior knowledge to optimize the solution space has been proposed. For example, some studies have captured the semantic structure information of the face by utilizing face visual priors, such as face parsing maps and heat maps, to improve the reconstruction effect.

[0004] Due to the limitations of the imaging environment, the captured face images are usually low-resolution. Low-resolution face images cannot provide a good visual experience and affect the performance of downstream tasks. Existing face super-resolution methods mainly design efficient network structures to learn the mapping from low-resolution face images to high-resolution face images. However, due to the large difference in the spatial dimensions of high-resolution and low-resolution face images, it is difficult to directly learn the mapping between the two. In this regard, there are methods that introduce face prior knowledge (such as face parsing graphs and heat maps) to constrain the solution space and improve the quality of face images. However, these methods still have obvious defects: First, they only use a single visual prior, mainly focusing on visual perception, while ignoring non-visual language text information, resulting in incomplete scene representation, which interferes with the performance of face image reconstruction. Summary of the invention

[0005] The present invention is aimed at the existing technical problems and proposes a face super-resolution method, system and computer device based on visual language prior. The present invention is implemented through the following technical solutions:

[0006] A face super-resolution method based on visual language prior includes the following steps:

[0007] Step 1: Send the low-resolution face image to the pre-trained visual-language large model to extract the visual-language multivariate representation;

[0008] Step 2: Construct a face super-resolution network assisted by visual language priors and integrate visual language prior information;

[0009] Step 3: Send the low-resolution face image and the multi-dimensional representation of the visual language extracted in step 1 to the network in step 2 to obtain a super-resolution result and a restored high-quality face image.

[0010] Furthermore, step one specifically includes the following steps:

[0011] The low-resolution face image is input into the basic large models SAM and DAM, and the face semantic segmentation map and depth map are extracted as visual prior information.

[0012] Furthermore, step one also includes the following steps:

[0013] The low-resolution image is input into the BLIP2 model to generate the corresponding text caption; to generate the description, the low-resolution face image and a specific prompt "provide useful visible features or observations about the face image" are combined and input into the ChatGPT-4 model to produce a detailed text description of the face image.

[0014] Furthermore, in step 2, the low-resolution face image and the visual language multi-representation are respectively sent to the feature extraction layer, the CLIP encoder and the image encoder to extract features, and the visual feature F0 and the visual language multi-representation feature E are obtained. C ,E D ,F S ,F D ;The extracted features are fed into a series of visual language prior fusion blocks and basic blocks;

[0015]

[0016] in and denote the i-th visual language prior fusion block and basic block respectively, and F i It is a feature that combines visual language prior information; the feature F processed by L layers L Input to a feature reconstructor implemented by a convolutional layer to produce the final super-resolution image I SR ; Introduce L1 loss as a constraint during training,

[0017]

[0018] Among them, I HR is the corresponding high-resolution face image.

[0019] Furthermore, in step 2, the visual language fusion module is specifically:

[0020] Given low-resolution facial features and visual language prior F C ,F D ,F S ,F D , the visual language prior fusion block feeds the low-resolution face features into four parallel attention mechanisms, namely SegA semantic segmentation attention, DepA depth attention, CapA title attention and DesA description attention, to achieve effective interaction with the four visual language priors; the depth map and semantic segmentation map are processed by SegA and DepA, and the spatial attention perceived by the face structure is learned through a series of cascaded convolutional layers to enhance the face features. CapA first applies global average pooling to the features of the title, and then performs convolution and sigmoid activation to generate global title attention. Compared with the title, the description of the image provides a more detailed and comprehensive view of the face image. Therefore, the image description is used to construct the query Q, while the face features are used to form the key K and value V, and their fusion is achieved through the cross attention mechanism, namely DesA. After integrating these four prior information, the four sets of resulting features are merged with the face features, and jump connections are introduced to generate the final output.

[0021] The present invention also relates to a human face super-resolution method system based on visual language prior, comprising a computer module, which applies the human face super-resolution method based on visual language prior.

[0022] The present invention also relates to a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and is characterized in that the processor implements the steps of the above method when executing the computer program.

[0023] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, wherein the computer program

[0024] When the program is executed by a processor, the steps of the above method are implemented.

[0025] Beneficial Effects

[0026] The present invention proposes a face super-resolution algorithm based on visual language prior, which not only utilizes multiple visual priors, but also integrates more advanced language priors, fully utilizing the complementary advantages of visual language priors to more completely represent face images, thereby effectively improving the quality of face images. By comparing the face super-resolution performance of this model with other methods, the model designed by the present invention has the best performance. Compared with the existing mainstream face super-resolution methods (FSRNet, DIC, SISN, SFMNet, FaceFormer, WFEN), the face images restored by the present invention perform better in both objective evaluation indicators and subjective visual quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 This is the face super-resolution model diagram of the present invention.

[0029] Figure 2 Schematic diagram of the visual language prior fusion block in the present invention.

[0030] Figure 3 This is a comparison chart of the 8x face super-resolution results in the present invention.

[0031] Figure 4 This is a comparison chart of the 16x face super-resolution results in the present invention.

[0032] Figure 5 This is a comparison result of the identity distance between the restored face image and the high-resolution face image in the present invention.

[0033] Figure 6 This is a comparison chart of the visual quality of different models in the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0035] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0036] The following is combined with Figures 1 to 6 The implementation manner is further described in detail.

[0037] The present invention utilizes language-visual priors to improve the restoration quality of face images. In this section, the specific details and design concepts of the proposed framework will be elaborated in detail.

[0038] For a given low-resolution face image I LR The present invention designs an effective reconstruction method to restore the corresponding high-quality face image. The reconstruction method of the present invention comprises the following steps:

[0039] Step 1: Feed the low-resolution face image into the pre-trained vision-language model to extract the vision-language multivariate representation

[0040] The workflow of the framework proposed by the present invention is as follows Figure 1 As shown. Input low-resolution face image I LR After that, it is first sent to the visual language model to extract the multi-dimensional representation of the visual language. The specific process is as follows:

[0041] Visual prior: semantic segmentation map and depth map. The face semantic segmentation map can accurately capture the high-level semantic details of the face structure, while the depth map provides accurate depth information of the face. Directly input low-resolution face images into these pre-trained models to extract the face semantic segmentation map and depth map as visual prior information.

[0042] Language prior: text title and description. The text title provides a macro perspective for the face image, summarizing its overall and abstract semantic content, while the text description goes further and provides a more detailed and comprehensive description of facial features. Unlike the structure and depth information depicted by the semantic mask and depth map in the visual prior, the title and description provide higher-level semantic information, among which the description contains comprehensive and detailed text content. The present invention inputs the low-resolution image into the BLIP2 (image text pre-training model) model and generates the corresponding text title. At the same time, in order to generate the description, combined with the low-resolution face image and a specific prompt, the model is required to provide useful visible features or observations about the face image, which are then input into the ChatGPT-4 model to produce a detailed text description of the face image.

[0043] Step 2: Construct a face super-resolution network assisted by visual language priors and integrate visual language prior information

[0044] Considering that visual-linguistic multivariate representations contain different information, their effects on face super-resolution are also different. Specifically, the text title summarizes the overall information of the face, while the description carefully depicts the specific content and features of the face. The semantic segmentation map contains facial structure information, and the depth map provides facial depth information. These prior information complement each other, and their comprehensive application can achieve a more comprehensive facial image representation, thereby effectively promoting facial image reconstruction. Therefore, the framework proposed in the present invention integrates these prior information to enhance the performance of face super-resolution.

[0045] Specifically, the low-resolution face image and the visual language multi-representation are extracted from the feature extraction layer, CLIP (visual language contrast pre-training large model) encoder and image encoder to obtain the visual feature F0 and the visual language multi-representation feature E C ,E D ,F S ,F D Subsequently, these extracted features are fed into a series of visual-linguistic prior fusion blocks and basis blocks (L groups in total) to fully exploit the potential of visual-linguistic priors.

[0046]

[0047] in, and denote the functions of the i-th visual language prior fusion block and the basic block, respectively, and F i It is a feature that combines visual language prior information. Finally, the feature F processed by L layers L Input to a feature reconstructor implemented by a convolutional layer to produce the final super-resolution image I SR In order to ensure that the method can recover satisfactory results, L1 loss is introduced as a constraint during training.

[0048]

[0049] Among them, I HR is the corresponding high-resolution face image.

[0050] The visual language fusion module is specifically:

[0051] Most existing face super-resolution technologies focus on the field of visual perception, improving image quality by incorporating visual priors, but often ignore the importance of language and text features, which limits the integrity of image representation and the performance of face super-resolution. The present invention constructs a comprehensive visual language prior system to more richly characterize the features of face images, thereby significantly improving face super-resolution performance. In order to effectively combine visual and language elements and give full play to the potential and complementarity of visual language priors, the present invention designs a visual language prior fusion block (VLPFB), such as Figure 2 shown.

[0052] Given low-resolution facial features and visual language prior E C ,E D ,F S ,F D , the goal of the visual-linguistic prior fusion block (VLPFB) is to integrate these elements and use the complementary information between them to improve the effect of face super-resolution. The initial step of VLPFB is to feed the features of low-resolution faces into four parallel and distinctive attention mechanisms - named SegA (Semantic Segmentation Attention), DepA (Deep Attention), CapA (Caption Attention) and DesA (Description Attention) - to achieve effective interaction with the four visual-linguistic priors. Specifically, the depth map and semantic segmentation map, as key elements reflecting the structure and spatial pixel information of the face, are processed by SegA and DepA, which learn spatial attention to the perception of face structure through a series of cascaded convolutional layers to enhance the face features. The title, as information that provides a global overview of the face image, is processed by CapA. CapA first applies global average pooling to the features of the title, and then performs convolution and sigmoid activation to generate global title attention. Compared with the title, the description of the image provides a more detailed and comprehensive perspective on the face image. Therefore, the query Q is constructed using the description of the image, while the facial features are used to form the key K and value V, and their fusion is achieved through the cross-attention mechanism, namely DesA. After integrating these four prior information, the four sets of result features are merged with the facial features, and jump connections are introduced to generate the final output. In the design of VLPFB, the present invention adopts a targeted fusion strategy based on the characteristics of various prior information, realizes sensitive fusion of prior information, and effectively promotes the performance improvement of face super-resolution.

[0053] Step 3: Send the low-resolution face image and the multi-dimensional representation of the visual language extracted in step 1 to the network in step 2 to obtain a super-resolution result and restore a high-quality face image.

[0054] Effect verification

[0055] 1. Comparison of objective evaluation indicators:

[0056] Table 1 shows the comparison results of objective evaluation indicators between the present invention and existing methods. In the ×16-fold face super-resolution task, the PSNR of the present invention is 25.95dB, which is 0.43dB higher than the second best method WFEN. In the ×8-fold face super-resolution task, the present invention also achieved the best objective evaluation indicators. FSRNet and DIC are committed to estimating facial visual priors and incorporating these priors into the reconstruction of face images, but their performance improvement is limited. SISN captures facial information by introducing an interleaved attention mechanism, and its performance is better than DIC and FSRNet. FaceFormer combines transformer and convolutional neural network, and its performance is better than SISN and DIC. WFEN and SFMNet use transformer and Fourier transform to capture the global receptive field, which further outperform SISN and FaceFormer. Although these methods all perform well to some extent, they mainly focus on a single visual prior and pixel-level image representation, without fully exploring a more comprehensive visual language prior. The present invention integrates visual language prior into the face super-resolution task, which can more completely represent the face image and thus effectively improve the quality of the face image.

[0057] Table 1 Comparison of objective evaluation indicators of existing face super-resolution methods

[0058]

[0059] 2. Comparison of subjective visual quality: Figure 3 and Figure 4 The visual quality comparison results of different methods in ×8 and ×16 face super-resolution tasks are shown. SRCNN is unable to reconstruct key facial components. Although FSRNet, DIC, and SISN can restore clear facial contours, they perform poorly in reconstructing key facial details such as eyes and mouth in the ×8 face super-resolution task. In the more difficult ×16 face super-resolution task, the performance of these methods further deteriorates and the restoration results are unsatisfactory. For the ×16 face super-resolution task, FaceFormer, SFMNet, and WFEN often cannot accurately restore the key parts of the face and there are distortions. Although in some specific cases (such as Figure 5 ), they can restore some facial features, but the generated images are often too smooth and lack necessary details. In contrast to these methods, the face super-resolution method with visual language prior designed in the present invention provides a more comprehensive image representation by utilizing visual language prior to restore facial images, thereby being able to restore facial images that are more visually pleasing and have realistic and natural details.

[0060] 3. Face identity distance comparison: In addition to improving the quality of low-resolution face images, face super-resolution methods should also maintain face identity information. That is, super-resolution face images should share the same identity information with their corresponding high-resolution face images. Therefore, a further comparison of identity distances was performed. The face images restored by different super-resolution methods and the corresponding high-resolution face images were input into the pre-trained face recognition model DeepFace to extract face identity features. Then, the cosine distance between these features was calculated as a measure of identity distance. The comparison results are shown in Figure 2. Figure 5 The identity distance of the present invention is lower than that of the comparison method, which shows that the present invention can better preserve the identity information of the face image, thus helping to improve the accuracy and efficiency of the face recognition task.

[0061] 4. Effectiveness analysis of VLPFB: When analyzing the effectiveness of the visual language prior fusion block (VLPFB), we took a step-by-step approach to evaluate its contribution. First, the base model Model 1 is created, which does not contain VLPFB and any visual language prior information. Then Model 2 is introduced, which fuses the visual language prior information through simple splicing and convolution. Model 2 has improved performance compared to Model 1, which verifies the positive role of visual language prior information in face reconstruction, although the improvement is not significant. In order to further verify the advantages of VLPFB, the splicing in Model 2 is replaced by VLPFB to form the model of the present invention. As shown in Table 2, the present invention has achieved the best performance in performance, and the improvement is obvious, which fully proves the effectiveness of VLPFB. Figure 6 The visual quality comparison between different models is shown. The present invention can retain more details when restoring face images, especially in terms of the clarity and accuracy of key face features such as eyes, which is significantly improved compared with Model 1 and Model 2. In summary, VLPFB can not only effectively integrate visual language prior information, but also significantly improve the quantitative evaluation indicators and visual quality of face images, which shows its importance and practicality in face super-resolution tasks.

[0062] Table 2 Effectiveness analysis of VLPFB

[0063]

[0064] The face super-resolution of the present invention is committed to recovering a high-resolution face image from a given low-resolution face image. In dealing with this ill-posed and challenging problem, the existing technologies are either committed to developing efficient network structures to learn the mapping from low-resolution faces to high-resolution faces, or to constrain the solution space by introducing prior knowledge. Although existing methods have made significant progress, they mainly focus on visual perception and ignore language and text features, resulting in incomplete scene representation, which in turn affects the reconstruction effect of face images. With the rise of large models, they have demonstrated excellent capabilities in content generation. Unlike the visual priors used in existing methods, language (text) knowledge describes a deeper level of understanding and higher-level abstraction, which can be regarded as a language prior to supplement the deficiencies of visual perception.

Claims

1. A face super-resolution method based on visual language prior, characterized in that: The steps include: Step 1: Send the low-resolution face image to the pre-trained visual-language large model to extract the visual-language multivariate representation; Step 2: Construct a face super-resolution network assisted by visual language priors and integrate visual language prior information; Step 3: Send the low-resolution face image and the multi-dimensional representation of the visual language extracted in step 1 to the network in step 2 to obtain a super-resolution result and a restored high-quality face image.

2. The face super-resolution method based on visual language prior according to claim 1, characterized in that: Step 1 specifically includes the following steps: The low-resolution face image is input into the basic large models SAM and DAM, and the face semantic segmentation map and depth map are extracted as visual prior information.

3. The face super-resolution method based on visual language prior according to claim 1, characterized in that: Step 1 also includes the following steps: The low-resolution image was input into the BLIP2 model, and the corresponding text caption was generated; to generate the description, the low-resolution face image and a specific prompt were combined to provide useful visible features or observations about the face image, which were input into the ChatGPT-4 model to produce a detailed text description of the face image.

4. The face super-resolution method based on visual language prior according to claim 1, characterized in that: In step 2, the low-resolution face image and the visual language multi-representation are sent to the feature extraction layer, CLIP encoder and image encoder respectively to extract features, and the visual feature F0 and the visual language multi-representation feature E are obtained. C ,E D ,F S ,F D ;The extracted features are fed into a series of visual language prior fusion blocks and basic blocks; in and denote the i-th visual language prior fusion block and basic block respectively, and F i It is a feature that combines visual language prior information; the feature F processed by L layers L The input is fed into a feature reconstructor implemented by a convolutional layer to produce the final super-resolution image I SR ; Introduce L1 loss as a constraint during training, Among them, I HR is the corresponding high-resolution face image.

5. The face super-resolution method based on visual language prior according to claim 4, characterized in that: In step 2, the visual language fusion module is specifically: Given low-resolution facial features and visual language prior E C ,E D ,F S ,F D The visual language prior fusion block feeds the low-resolution face features into four parallel attention mechanisms, namely SegA semantic segmentation attention, DepA depth attention, CapA caption attention and DesA description attention, to achieve effective interaction with the four visual language priors; the depth map and semantic segmentation map are processed by SegA and DepA, and the spatial attention perceived by the face structure is learned through cascaded convolutional layers to enhance the face features; CapA first applies global average pooling to the features of the title, and then performs convolution and sigmoid activation to generate global title attention; DesA uses image description to construct the query Q, while using facial features to form the key K and value V, and realizes their fusion through the cross attention mechanism; then these four sets of resulting features are merged with the facial features to generate the final output.

6. A face super-resolution system based on visual language prior, characterized in that: It comprises a computer module, which applies the face super-resolution method based on visual language prior as described in any one of claims 1 to 5.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method of any one of claims 1 to 7 are implemented.

Images