A multi-modal face restoration and expression recognition system and method based on facial action unit semantic guidance

Abstract

The present application belongs to the technical field of image restoration and computer vision, and specifically relates to a multi-modal face restoration and expression recognition system and method based on facial action unit semantic guidance. The system extracts multi-scale features through visual coding and detects AU activation probability using a graph neural network; a semantic conversion module converts numerical probability into interpretable biomechanical structured text; a multi-modal reasoning module fuses visual and text information and introduces the common sense reasoning ability of a multi-modal large language model to improve the performance of the student network through knowledge distillation; finally, a conditional generation module realizes image restoration guided by semantic features. The present application uses facial action units as a biomechanical intermediary and converts multi-modal reasoning ability into restoration constraints, solving the defects of existing technology in physiological distortion, recognition dependence on image quality and isolation of the two tasks, achieving collaborative enhancement of face restoration and expression recognition, and ensuring that the restored result is visually clear and consistent with the physiological laws of facial muscles.

Description

Technical Field

[0001] This invention belongs to the field of image restoration and computer vision technology, specifically relating to a multimodal face restoration and expression recognition system and method based on semantic guidance of facial action units. Background Technology

[0002] Facial image restoration and expression recognition are two key tasks in computer vision, with wide applications in security monitoring, telemedicine, and human-computer interaction. Current technologies typically treat these two tasks as independent problems. In image restoration, existing methods are mainly based on generative adversarial networks (GANs) or diffusion models, aiming to recover visually clear facial features and textures from low-quality, occluded, or noisy images. However, these methods largely rely on pixel-level visual feature learning, lacking explicit modeling of the physiological structure of the human face. This can easily introduce deformations that violate muscle movement patterns during the restoration process, such as generating a "fake smile" image with raised corners of the mouth but no corresponding tightening around the eyes, leading to physiological distortions in the restoration results.

[0003] In facial expression recognition, deep learning methods for high-quality images have matured, but their performance drops sharply when image quality is severely degraded. Although some studies have attempted to introduce facial action units (AUs) as intermediate representations for expression analysis to improve robustness, AU detection itself is prone to failure under extremely damaged conditions. Furthermore, these methods are mainly limited to mining low-level visual features and fail to effectively utilize the rich psychological and emotional common sense knowledge inherent in AUs.

[0004] In summary, the main shortcomings of the existing technology are: 1) the restoration process is disconnected from biomechanical constraints, which may lead to anatomically unreasonable results; 2) the recognition module is highly dependent on image quality; 3) the restoration and recognition tasks are carried out in isolation, without forming a synergistic enhancement effect. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multimodal face restoration and expression recognition system and method based on semantic guidance of facial action units. The objective of this invention is to provide a novel solution capable of uniformly addressing low-quality face restoration and recognition problems, ensuring that the results are both visually and physiologically reliable.

[0006] The first aspect of this invention provides a multimodal face restoration and expression recognition system based on facial action unit semantic guidance, comprising the following core processing modules working in concert: a visual encoding and AU detection module, a semantic conversion and enhancement module, a multimodal reasoning and guided generation module, and a conditional generation module;

[0007] The visual encoding and AU detection module extracts multi-scale features from the face image to be restored and outputs multi-scale visual feature representations. At the same time, it uses graph neural networks to model the correlation of facial action units and outputs AU activation probability vectors that represent the probability of facial muscle movement.

[0008] The semantic conversion and enhancement module, connected to the visual encoding and AU detection module, converts the AU activation probability vector into structured guiding text containing definite actions and possible actions through predefined biomechanical mapping rules, realizing the conversion of numerical features into high-level interpretable semantic features.

[0009] The multimodal reasoning and guidance generation module is connected to the visual encoding and AU detection module and the semantic conversion and enhancement module, respectively. It performs multimodal fusion of multi-scale visual feature representation, AU activation probability vector and structured guidance text, and uses multimodal reasoning ability to complete potential action units and infer muscle movement trends to generate semantic guidance feature vectors and expression labels.

[0010] The conditional generation module is connected to the visual encoding and AU detection module and the multimodal reasoning and guided generation module, respectively. It receives multi-scale visual feature representation and semantic guided feature vector, and injects the semantic guided feature vector as a conditional signal into the convolutional generation network to achieve guided image inpainting and outputs a restored face image that conforms to the physiological laws of facial muscle movement.

[0011] The system performs multi-task joint optimization using pixel reconstruction loss, AU consistency loss based on biomechanical priors, and expression classification loss.

[0012] As a further optimization of the above system, the visual coding and AU detection module includes cascaded multi-scale visual coding units and facial motion unit detection branches.

[0013] The multi-scale visual coding unit adopts a Transformer architecture based on a shifted window self-attention mechanism, which contains four feature extraction blocks in sequence. Each feature extraction block consists of a downsampling layer, a window multi-head self-attention layer, and a shifted window multi-head self-attention layer in sequence, which are used to extract multi-scale visual feature representations from local texture to global structure.

[0014] The facial action unit detection branch is connected to the outputs of the second and third feature extraction blocks in the multi-scale visual coding unit to output the AU activation probability vector.

[0015] As a further optimization of the above system, the facial action unit detection branch includes a learnable graph network and a fully connected layer. The nodes of the graph network correspond to predefined facial action units, and the edge weights are initialized and dynamically updated through a trainable dependency matrix. The graph network realizes message passing and feature aggregation between nodes through a graph attention mechanism. Then, through the linear mapping and non-linear activation processing of the fully connected layer, the features are transformed into AU activation probability vectors.

[0016] As a further optimization of the above system, the facial action unit consists of 44 standard facial action units covering the forehead, eye area, nose, corner of mouth and jaw area; the graph structure network maps the real-value scores of nodes to continuous values ​​in the range of 0 to 1 through the Sigmoid activation function, and arranges all node outputs in a fixed order to form an AU activation probability vector of length 44.

[0017] As a further optimization of the above system, the semantic conversion and enhancement module includes a semantic rule mapping library and a text generation unit. The semantic rule mapping library stores the mapping relationship between a single facial action unit and its corresponding biomechanical semantic phrase, the mapping relationship between a combination of facial action units and its corresponding biomechanical semantic phrase, and the mapping relationship between a combination of facial action units and a description of a complex expression or muscle coordination pattern. This enables the mapping from a single facial action unit or a combination of facial action units to a corresponding interpretable semantic description. The text generation unit receives the AU activation probability vector, executes independent preset judgment rules on a single facial action unit or a combination of facial action units to divide the activation set, and generates structured guiding text based on the mapping relationship in the semantic rule mapping library.

[0018] As a further optimization of the above system, the text generation unit adopts a three-segment judgment mechanism, which performs independent judgment rules on individual facial action units and combinations of facial action units respectively: For an individual facial action unit, when its AU activation probability is greater than or equal to the high threshold, it is classified into the "definitely activated" set; when its AU activation probability is less than or equal to the low threshold, it is classified into the "inactive" set; when its AU activation probability is between the low threshold and the high threshold, it is classified into the "potentially activated" set. For combinations of facial action units, when the activation probability of all AUs in the combination is greater than or equal to the high threshold, it is classified into the "definitely activated" set; when the combination meets the following three conditions at the same time, it is classified into the "potentially activated" set: ① There are no AUs in the combination with an activation probability less than or equal to the low threshold; ② The number of AUs in the combination with an activation probability greater than or equal to the high threshold is not less than a preset proportion (e.g., 60%) of the total number of AUs in the combination; ③ There are some AUs in the combination whose activation probability is between the low threshold and the high threshold; if either the "definitely activated" or "potentially activated" condition is not met, it is classified into the "inactive" set.

[0019] As a further optimization of the above system, the text generation unit generates structured guidance text based on a predefined biomechanical semantic description grammar, combined with the mapping relationship of the semantic rule mapping library and the aforementioned activation set division results: For individual facial action units and combinations of facial action units belonging to the "definite activation" set, they are mapped to corresponding biomechanical semantic phrases, generating text fragments in the format of "definite action: + action phrase list"; for individual facial action units and combinations of facial action units belonging to the "possible activation" set, they are mapped to corresponding biomechanical semantic phrases and uncertainty modifiers are added, generating text fragments in the format of "possible action: + action phrase list"; facial action units and combinations of facial action units belonging to the "inactive" set do not participate in text generation; the above two types of text fragments are concatenated in a preset order to form a complete structured guidance text; if the AU activation probability of all facial action units is lower than a preset high threshold, then a descriptive text representing a neutral expression is directly output.

[0020] As a further optimization of the above system, the multimodal reasoning and guided generation module includes a multimodal reasoning sub-network, a student text encoder, and a guided feature fusion layer in the reasoning stage;

[0021] The multimodal inference subnetwork converts the AU activation probability vector into the AU semantic embedding vector through a semantic projector, and maps the multi-scale visual feature representation into the AU visual embedding vector through a visual projection layer. After concatenating the two types of embedding vectors in the channel dimension, the feature fusion layer is used to obtain the multimodal fusion feature. The multimodal inference subnetwork is set with two parallel output heads. The expression classification head outputs expression labels through a fully connected layer and a Softmax function, and the semantic guidance head outputs the AU visual semantic fusion guidance vector through a fully connected layer.

[0022] The student text encoder encodes structured guided text into fixed-length text semantic embedding vectors through a character-level embedding layer, a bidirectional gated recurrent unit, and a fully connected projection layer.

[0023] The guided feature fusion layer concatenates the AU visual semantic fusion guidance vector and the text semantic embedding vector in the feature dimension, and compresses them into a semantic guided feature vector of a preset dimension through linear mapping, which serves as the control signal for the conditional generation module.

[0024] As a further optimization of the above system, the multimodal reasoning and guided generation module adopts a two-layer architecture of teacher-guided learning during the training phase. On the basis of the reasoning phase architecture, a multimodal large language model and a chain-thinking reasoning control unit are added. Through the knowledge distillation mechanism, students can learn the common sense reasoning ability and semantic expression accuracy of the large model through the network.

[0025] As a further optimization of the above system, the multimodal large language model is a Transformer-based vision-language joint encoding architecture that integrates a visual encoder and a text encoder to achieve alignment of images and text in a unified semantic space and output teacher semantic embedding vectors and guiding natural language descriptions.

[0026] The chain-thinking reasoning control unit constructs and injects multi-step reasoning instructions to control the multimodal large language model to carry out two-level reasoning. The first-level reasoning completes the AU states that were not detected due to occlusion or degradation. The second-level reasoning infers the expression category based on the complete AU state and explains the reasoning.

[0027] The semantic embedding vectors of the text output by the student text encoder are aligned with the semantic embedding vectors of the teacher output by the multimodal large language model by mean square error to achieve knowledge distillation; the multimodal inference subnetwork is optimized end-to-end under the common constraints of pixel reconstruction loss, AU consistency loss and expression classification loss.

[0028] As a further optimization of the above system, the conditional generation module adopts a step-by-step upsampling decoding structure with a decoding generation network with an embedded conditional modulation mechanism as its core. It includes three upsampling units, convolutional blocks corresponding to the upsampling units, conditional adaptive normalization sub-units, and output convolutional layers.

[0029] After each upsampling unit restores the resolution of the feature map, the local texture information is extracted by the corresponding convolutional block, and then the corresponding conditional adaptive normalization subunit performs affine modulation with the semantically guided feature vector as the control signal. The feature map after three levels of modulation is mapped to a three-channel face image by the output convolutional layer to obtain the restored face image.

[0030] The conditional adaptive normalization subunit uses semantically guided feature vectors as control signals to generate channel scale coefficients and bias terms at the corresponding scale in real time, and performs affine modulation on the feature map of the current layer; the output of the output convolutional layer is processed by a normalized nonlinear function to obtain the final restoration result.

[0031] A second aspect of this invention provides a multimodal face reconstruction and expression recognition method based on semantic guidance of facial action units, comprising the following steps:

[0032] S1: Obtain the face image to be restored, perform multi-scale feature extraction on the face image to be restored, obtain multi-scale visual feature representation, and at the same time use graph neural network to model the correlation of facial action units, outputting AU activation probability vector representing the probability of facial muscle movement.

[0033] S2: The AU activation probability vector is semantically transformed and enhanced through predefined biomechanical mapping rules to generate structured guiding text containing definite actions and possible actions, realizing the transformation of numerical features into high-level interpretable semantic features;

[0034] S3: Multimodal fusion of multi-scale visual feature representation, AU activation probability vector and structured guidance text, using multimodal reasoning capabilities to complete potential action units and infer muscle movement trends, generating semantic guidance feature vectors and expression labels;

[0035] S4: Receives multi-scale visual feature representation and semantically guided feature vector, injects the semantically guided feature vector as a conditional signal into the convolutional generative network to carry out guided image inpainting, and outputs a restored face image that conforms to the physiological laws of facial muscle movement.

[0036] The method employs multi-task joint optimization through pixel reconstruction loss, AU consistency loss based on biomechanical priors, and expression classification loss to achieve synergistic enhancement of face restoration and expression recognition.

[0037] As a further optimization of the above method, this method can be implemented using any of the aforementioned multimodal face restoration and expression recognition systems based on facial action unit semantic guidance.

[0038] Beneficial effects

[0039] The present invention constructs a closed-loop collaborative framework of "visual perception-semantic reasoning-physiologically guided reconstruction". By introducing facial action units as a biomechanical mediator, it successfully transforms the common sense reasoning ability of large language models into structured constraints of the generation process, overcoming the physiological distortion problem of restoration results caused by the lack of anatomical priors in existing generative models. At the same time, by using semantic conversion and enhancement modules to transform low-level numerical probabilities into high-level interpretable text, it not only enhances the robustness of the system under extremely low-quality image conditions and enables the recognition module to get rid of its dependence on single pixel features, but also achieves a synergistic effect of mutual promotion between restoration quality and recognition accuracy through multi-task joint optimization and knowledge distillation mechanisms. This ensures that the output face image is visually clear, identity-preserving, and strictly follows the physiological and mechanical laws of facial muscle movement, providing technical support for multimodal emotion computing and high-quality visual reconstruction that combines perceptual realism and logical credibility. Attached Figure Description

[0040] Figure 1 This is a diagram showing the overall architecture of the multimodal face restoration and expression recognition system of the present invention.

[0041] Figure 2 This is a schematic diagram of the visual encoding and AU detection module architecture.

[0042] Figure 3 This is a schematic diagram of the semantic transformation and enhancement module architecture.

[0043] Figure 4 This is a schematic diagram of the architecture of the multimodal reasoning and guided generation module in the reasoning stage.

[0044] Figure 5 This is a schematic diagram of the architecture of the multimodal reasoning and guided generation module during the training phase.

[0045] Figure 6 This is a schematic diagram of the architecture of the conditional generation module.

[0046] Figure 7 The image shown is the face to be restored used in this embodiment.

[0047] Figure 8 This is a visualization of the multi-scale visual feature representation extracted by the visual encoding and AU detection modules in the embodiment.

[0048] Figure 9 The image shown is the restored face image output by the conditional generation module in this embodiment. Detailed Implementation

[0049] This invention proposes a face image restoration and expression recognition system that integrates semantic priors of facial action units with multimodal large model reasoning. It constructs a closed-loop framework of "visual perception-semantic reasoning-physiological guided reconstruction". By using the facial action unit as a biomechanical representation as a bridge, the common sense reasoning ability of multimodal large models is injected into the generative restoration process, and consistency constraints are used to achieve collaborative optimization of restoration and recognition tasks.

[0050] The system proposed in this invention is as follows Figure 1 As shown, it includes the following four core processing modules that work together: visual encoding and AU detection module, semantic conversion and enhancement module, multimodal reasoning and guided generation module, and conditional generation module.

[0051] The visual encoding and AU detection module performs multi-scale feature extraction on the face image to be restored and outputs multi-scale visual feature representations; and uses graph neural networks to model the correlation of each facial action unit (AU) and outputs AU activation probability vectors representing the probability of facial muscle movements.

[0052] Semantic Conversion and Enhancement Module: The numerical AU activation probability vector is converted into a structured guiding text containing "determined actions" and "possible actions" described in natural language through predefined biomechanical mapping rules, thus transforming numerical features into high-level interpretable semantic features with biomechanical meaning.

[0053] Multimodal reasoning and guidance generation module: The multi-scale visual feature representation and AU activation probability vector output by the visual encoding and AU detection module are fused with the structured guidance text generated by the semantic transformation and enhancement module in a multimodal manner. The chain-like thinking reasoning ability of the multimodal big model is used to complete the potential action units and infer the muscle movement trend, thereby generating semantic guidance feature vectors and final expression labels.

[0054] Conditional generation module: Receives multi-scale visual feature representations from the visual encoding and AU detection modules, as well as semantically guided feature vectors from the multimodal reasoning and guided generation modules. Injects the guided parameters as conditional signals into the convolutional generation network to achieve guided image inpainting and outputs restored face images that conform to physiological laws.

[0055] The system utilizes pixel reconstruction loss, biomechanical prior-based AU consistency loss, and expression classification loss for multi-task joint optimization to ensure that the restored face is visually clear while possessing the rationality of physiological structure and the accuracy of expression expression, and also ensures the accuracy of expression recognition.

[0056] (1) Visual encoding and AU detection module

[0057] The structure of the visual encoding and AU detection module is as follows: Figure 2 As shown, the system includes a cascaded multi-scale visual coding unit and a facial action unit detection branch. The multi-scale visual coding unit adopts a Transformer architecture based on a shifted window self-attention mechanism (Swin Transformer), which sequentially contains four feature extraction blocks. Each feature extraction block consists of a downsampling layer, a window multi-head self-attention layer, and a shifted window multi-head self-attention layer, used to extract and output a multi-scale visual feature representation from the input face image, ranging from local texture to global structure.

[0058] The facial action unit detection branch is connected to the outputs of the second and third feature extraction blocks in the multi-scale visual coding unit. It comprises a learnable graph structure network where nodes correspond to predefined facial action units, and edge weights are initialized and dynamically updated using a trainable dependency matrix. This branch computes the association weights between nodes through a graph attention mechanism, enabling message passing and feature aggregation between nodes. Finally, it performs linear mapping and non-linear activation processing (such as through fully connected layers and their built-in Sigmoid activation function) via fully connected layers, transforming the features into AU activation probability vectors representing the probabilities of facial muscle movements.

[0059] Specifically, in some embodiments, the multi-scale visual coding unit receives an aligned and normalized image of the face to be restored as input. The input image has a fixed spatial resolution and a three-channel (RGB) structure, for example, it can be set to 128×128 or 224×224 pixels. The attention window size and the number of attention heads in the windowed multi-head self-attention layer and the shifted windowed multi-head self-attention layer are preset hyperparameters. For example, in a typical embodiment, the attention window size can be set to 8×8 or 16×16 pixels, and the number of attention heads can be set to 4 or 8. The facial motion unit consists of several standard facial motion units covering the forehead, periorbital area, nasal alae, corners of the mouth, and jaw region, totaling 44 units. This set may include, but is not limited to, the following representative action units: AU1 (inner eyebrow lift), AU2 (outer eyebrow lift), AU4 (brow pull-down / brow gathering), AU6 (periocular levator muscle contraction), AU7 (eyelid tightening), AU9 (nasal alar wrinkling), AU10 (upper lip lift), AU12 (corner of mouth lift), AU14 (cheek contraction), AU15 (corner of mouth pull-down), AU17 (chin lift), AU20 (lip protrusion), AU23 (lip compression), AU24 (lip closure), etc.

[0060] In the readout phase, the graph-structured network outputs a real-valued score for each facial action unit node, which is then mapped to a continuous value between 0 and 1 using an element-wise sigmoid activation function, representing the confidence level of the facial action unit's activation in a probabilistic form. The outputs of all nodes are arranged in a fixed order to form an AU activation probability vector of length 44. Simultaneously, the graph-structured network operates on the output features of the second and third feature extraction blocks of the multi-scale visual coding unit through global pooling operations to obtain a set of global feature vectors. These vectors are concatenated along the channel dimension and projected onto the node feature space via a linear mapping unit to obtain the initial node vector representation. This vector is then copied and expanded along the action unit dimension to form an initial node feature tensor of size "batch size × number of action units × node feature dimension". The graph structure consists of a fixed number of nodes, each corresponding to a predefined facial action unit. The connections between nodes are represented by a learnable adjacency matrix, the size of which is equal to "number of action units × number of action units", and can be randomly initialized using a uniform or normal distribution. The graph-structured network performs a linear transformation on the node features of each layer, constructs a concatenated vector of any pair of node features, and performs an inner product with a set of trainable attention parameters. After nonlinear activation, unnormalized attention weights are obtained. These attention weights are then combined with a learnable adjacency matrix, and the softmax function is used to normalize the feature contributions within the node's neighborhood, resulting in weighted coefficients for message passing. Node updates are achieved by weighted summation of the features of neighboring nodes, thereby propagating and aggregating contextual information related to facial muscle biomechanical constraints between layers.

[0061] (2) Semantic conversion and enhancement module

[0062] The structure of the semantic transformation and enhancement module is as follows: Figure 3As shown, it includes a predefined semantic rule mapping library from AU to semantics and a text generation unit. The semantic rule mapping library stores the mapping relationship between a single facial action unit and its corresponding biomechanical semantic phrase, the mapping relationship between a combination of facial action units and its corresponding biomechanical semantic phrase, and the mapping relationship between a combination of facial action units and a description of a complex expression or muscle coordination pattern. This includes mappings from individual AUs (such as AU1: inner eyebrow lift, AU2: outer eyebrow lift, AU4: eyebrow droop, AU6: periorbital levator muscle contraction, AU7: mild eyelid tension, AU9: nasal alar wrinkling, AU10: upper lip lift, AU12: corner of mouth upturn, AU15: corner of mouth downturn, AU17: chin upturn, AU20: lips protrude, AU23: lips tighten, AU24: lips tightly closed, etc.) to corresponding biomechanical phrases, as well as mappings from typical combination patterns (such as AU6+AU12 representing "synergistic contraction of the zygomaticus major and orbicularis oculi muscles, corresponding to an enjoyable smile", AU1+AU2+AU5 representing "joint lifting of the upper eyelid and eyebrow area, corresponding to a tendency of surprise", AU4+AU15 representing "eyebrow droop and corner of mouth downturn, corresponding to suppressed or negative emotions, etc.) to combined descriptive texts. Through this rule base, when the model detects the activation of a certain AU or AU combination at the numerical level, it can directly look up the corresponding biomechanical keyword description in the table, realizing an interpretable mapping from action unit probability vectors to high-level structured semantic features.

[0063] The text generation unit receives the AU activation probability vector output by the facial motion unit's probe branch at its input end. Internally, the text generation unit employs a three-segment decision mechanism, performing activation level determination on individual facial motion units and combinations of facial motion units separately. For an individual facial motion unit, when its AU activation probability is greater than or equal to a high threshold (e.g., 0.6), it is assigned to the "definitely activated" set; when the activation probability is less than or equal to a low threshold (e.g., 0.4), it is assigned to the "inactive" set; and when the activation probability is between the low and high thresholds, it is assigned to the "potentially activated" set. For combinations of facial motion units, when the activation probabilities of all AUs in the combination are greater than or equal to the high threshold, it is assigned to the "definitely activated" set; when the combination simultaneously meets the following conditions: ① there are no AUs with activation probabilities less than or equal to the low threshold; ② the number of AUs with activation probabilities greater than or equal to the high threshold is not less than a preset proportion (e.g., 60%) of the total number of AUs in the combination; ③ some AUs have activation probabilities between the low and high thresholds, it is assigned to the "potentially activated" set; when neither the "definitely activated" nor the "potentially activated" condition is met, it is assigned to the "inactive" set. The aforementioned classification rules are clear and logically rigorous, and highly compatible with the physiological characteristics of facial expressions. Especially for combinations of facial action units, compared to the intuitive classification method of "classifying all cases that are not completely certain or not completely inactive as possible activations through elimination," the possible activation determination rules adopted in this invention incorporate biomechanical priors of facial muscle synergistic movements. This effectively eliminates abnormal cases with contradictory activation probability distributions and unreasonable physiological structures, retaining possible activation cases where the overall structure has a clear activation basis and only some accompanying movements are in a state of ambiguity. This not only better reflects the true laws of facial expression movements but also provides more stable and reliable constraints for subsequent multimodal semantic guidance and image restoration processes.

[0064] Subsequently, the text generation unit maps the action units and combinations in the "definitely activated" set to corresponding deterministic biomechanical descriptive phrases based on predefined correspondences in the semantic rule mapping library. This description follows a confidence-guided biomechanical semantic description grammar, which adheres to predefined sentence templates. For action units and combinations in the "definitely activated" set, they are organized into text fragments such as "definite action: ..."; for action units and combinations in the "potentially activated" set, uncertainty modifiers such as "potentially" or "slightly" are added before the corresponding phrases, organizing them into text fragments such as "possible action: ...". When the activation probability of all action units is below a high threshold, a default description representing a near-neutral expression is generated. Action units and combinations in the "inactive" set do not participate in text generation.

[0065] The aforementioned semantic transformation and enhancement module employs a predefined natural language template grammar to fill the two sets into coherent descriptive fragments: for the "definite actions" set, it generates statements such as "definite actions: inner eyebrow raised, corners of mouth turned up, periorbital levator muscle contraction" in the form of "label + colon + comma-separated action phrase list"; for the "possible actions" set, it generates statements such as "possible actions: possible slight wrinkling of the nasal wing, possible slight lifting of the chin". When both sets exist, these fragments are connected sequentially by semicolons or periods to form a complete description. When the activation probability of all AUs is below a high threshold, a default template text is output to indicate that the overall expression is close to neutral and muscle activity is weak. The aforementioned confidence-guided biomechanical semantic description grammar can be understood as a grammatical rule of "degree of certainty + location + movement trend". Keywords from the rule base are filled into fixed sentence templates according to their level of certainty, thereby converting the underlying numerical features into structured biomechanical semantic text that can be directly understood and utilized by the multimodal inference module, providing standardized semantic input for downstream multimodal inference.

[0066] (3) Multimodal reasoning and guided generation module

[0067] The multimodal reasoning and guided generation module presents a lightweight closed-loop structure during the reasoning phase, such as... Figure 4 As shown, it mainly consists of a local multimodal inference subnetwork, a student text encoder, and a guided feature fusion layer.

[0068] The multimodal inference subnetwork receives multi-scale visual feature representations and AU activation probability vectors output from the visual encoding and AU detection modules. Specifically, the multimodal inference subnetwork first uses its internal semantic projector to perform linear transformation and non-linear activation on the AU activation probability vector, elevating it to a fixed-dimensional AU semantic embedding vector. Simultaneously, a visual projection layer maps the multi-scale visual feature representations to the same hidden dimension as the AU semantic embedding vector, obtaining the AU visual embedding vector. Subsequently, the multimodal inference subnetwork concatenates the obtained AU visual embedding vector and AU semantic embedding vector along the channel dimension, and obtains a unified multimodal fusion feature through a feature fusion layer based on linear layers and non-linear activation functions. Around this fusion feature, the multimodal inference subnetwork processes it through two parallel output heads: one is an expression classification head, which outputs the expression classification result, i.e., expression label, through a fully connected layer and a Softmax function; the other is a semantic guidance head, which maps and outputs an AU visual-semantic fusion guidance vector related to the current expression semantics through a fully connected layer.

[0069] Meanwhile, the student text encoder is responsible for encoding the structured guiding text output from the semantic transformation and enhancement module into fixed-length text semantic embedding vectors. During processing, after receiving the structured guiding text, the student text encoder first truncates or pads each text to a preset maximum length, then maps the discrete character sequence into an embedding vector sequence through a character-level embedding layer. Subsequently, bidirectional gated recurrent units are used to perform forward and backward modeling in the time dimension to obtain a hidden state sequence containing contextual information. This sequence is then processed through time-dimensional aggregation (such as average pooling) and a fully connected projection layer, ultimately mapping it to a fixed-dimensional text semantic embedding vector.

[0070] Finally, the guided feature fusion layer is responsible for uniformly encoding the aforementioned AU visual-semantic fusion guided vector and text semantic embedding vector. This fusion layer receives the concatenation result of the two types of vectors in the feature dimension (e.g., concatenating the 256-dimensional AU visual-semantic fusion guided vector and the 256-dimensional text semantic embedding vector into a 512-dimensional vector), and compresses it to a semantically guided feature vector of a preset dimension through linear mapping. This semantically guided feature vector serves as the control signal input in the conditional generation module, providing refined semantic constraints for the subsequent image restoration and expression reconstruction processes.

[0071] This closed-loop structure enables the multimodal reasoning and guided generation modules to break free from the dependence on large-scale language models during the reasoning stage, achieving lightweight and fine-grained semantic control over the subsequent image restoration and facial expression reconstruction processes.

[0072] In order to enable the aforementioned lightweight student network to acquire deep common sense reasoning ability and semantic expression accuracy, this invention introduces a pre-trained multimodal large language model as a teacher network for guided training.

[0073] During the training phase, the multimodal reasoning and guided generation modules are presented as a two-tiered architecture guided by teachers, such as... Figure 5 As shown, it includes a multimodal large language model, a chain-based reasoning control unit, a multimodal reasoning subnetwork, a student text encoder, and a guided feature fusion layer.

[0074] The pre-trained multimodal large language model adopts a Transformer-based vision-language joint encoding architecture, integrating a visual encoder and a text encoder. The visual encoder takes the face image to be restored as input, dividing the image into several local regions or patches and mapping them to a sequence of visual tokens. Through multi-layer self-attention and cross-modal attention computation, it obtains a visual feature representation that combines local texture and overall structural information. The text encoder takes the structured guiding text output by the semantic transformation and enhancement module as input, segmenting the text and mapping it to a sequence of text tokens. The visual tokens and text tokens are concatenated along the sequence dimension and then input into the Transformer network for joint modeling, achieving alignment of the image and text in a unified semantic space. The hidden state sequence output by the last Transformer layer of this network is pooled along the temporal dimension of the text-related position vectors in the hidden state sequence to obtain a fixed-dimensional teacher semantic embedding vector. Simultaneously, this hidden state sequence is fed into the language generation head, outputting guiding natural language instructions containing observation results, reasoning processes, and restoration suggestions. The aforementioned teacher semantic embedding vector serves as the semantic supervision signal for the local student network, while the aforementioned guiding natural language description serves as the semantic guidance text for subsequent reconstruction.

[0075] The chain-thinking reasoning control unit is used to construct and inject multi-step reasoning instructions to control the pre-trained multimodal large language model to perform two levels of reasoning. In the first level of reasoning, the control unit writes structured guiding text into the prompts, requiring the model to "infer AU states that were not detected due to occlusion or degradation based on common sense." The model then completes the inference by combining muscle co-activation priors. For example, when AU6 is observed to be activated and showing an overall smiling trend, it infers that AU12 is likely to be activated simultaneously; when AU1, AU2, and AU5 are all at medium to high activation levels, it infers that the forehead and upper eyelids are raised. In the second level of reasoning, the control unit requires the model to "give the most likely expression category and explain the reasoning based on the complete AU states." The large language model then makes judgments based on the pattern relationship between AU combinations and expression labels. For example, it interprets the joint strong activation of AU6 and AU12 as an enjoyable smile, and the activation of AU9 and AU10 as an expression of disgust. The intermediate reasoning process and the final decision are output in natural language form.

[0076] During training, the multimodal inference subnetwork and the student text encoder work synchronously and coupled with the aforementioned teacher model. The multimodal inference subnetwork performs end-to-end optimization under the common constraints of pixel reconstruction loss, AU consistency loss, and expression classification loss. Simultaneously, by aligning the text semantic embedding vector output by the student text encoder with the teacher semantic embedding vector of the large language model using mean squared error (MSE), distillation of the knowledge and inference patterns of the upper-layer multimodal large language model is achieved. This mechanism allows the student text encoder to learn refined text representations consistent with the semantic space of the teacher model without directly calling the large-scale language model. Finally, the guided feature fusion layer fuses the text semantic embedding vector generated by the student side with the AU visual semantic fusion guided vector output by the multimodal inference subnetwork, outputting a unified semantic guided feature vector. This vector serves as the control signal for the modulation unit in the conditional generation module, ensuring that the model can fully utilize the prior knowledge of the large model to guide complex face reconstruction tasks during training.

[0077] (4) Conditional generation module

[0078] Conditional generation module, such as Figure 6 As shown, a decoding-generative network with an embedded conditional modulation mechanism is used as the core, aiming to achieve high-fidelity face image restoration through multi-level semantic injection. The input of this module includes multi-scale visual feature representations from the visual encoding and AU detection modules, and semantically guided feature vectors output by the multimodal reasoning and guided generation modules; its output is a restored face image with the same spatial resolution as the face image to be restored.

[0079] In terms of specific architecture, this module adopts a step-by-step upsampling decoding structure. First, the module inputs the final-level feature map (i.e., the deepest feature map with the lowest spatial resolution and the highest semantic abstraction) extracted by the visual encoding and AU detection modules into the first-level upsampling unit, and restores the spatial resolution of the features through deconvolution or transposed convolution operations. Then, the feature map enters the first-level convolution block, and the local texture information is refined using convolution operations. Subsequently, the convolution-processed feature map enters the first-level conditional adaptive normalization subunit. This subunit uses the semantically guided feature vector as the control signal to generate the channel scale coefficient and bias term at the corresponding scale in real time, and performs affine modulation on the feature map of the current layer.

[0080] Following the above process, the feature map sequentially passes through the second and third level upsampling units. After each level of upsampling, it first undergoes further feature extraction through a corresponding convolutional block before entering the corresponding conditional adaptive normalization subunit for semantic modulation. This hierarchical architecture, alternating between "upsampling-convolutional block-modulation unit," continuously injects high-level semantic constraints jointly defined by AU semantics and textual reasoning into each stage of decoding and reconstruction, ensuring that the reconstruction process is subject to fine-grained semantic control.

[0081] Finally, the highest-resolution feature map, after multiple transformations and modulations, is mapped into a three-channel face image through the output convolutional layer. During the mapping process, a normalized nonlinear function is used to limit the output pixel values ​​within a preset range, resulting in the final restored image. Thanks to this embedded conditional modulation decoding architecture, this invention can significantly improve the structural integrity and expression accuracy of the restored face image under various degradation factors such as noise, blurring, compression artifacts, and large-area occlusion, through strong guidance from semantic information, ensuring that the restored image not only conforms to the original identity features but also follows the laws of physiological muscle movement.

[0082] The present invention is further illustrated below with specific embodiments. These embodiments are exemplary and intended to illustrate the problem and explain the present invention, and are not intended to be limiting. Example

[0083] (1) Obtain the image of the face to be restored, such as Figure 7 a and Figure 7 As shown in b.

[0084] (2) Input the face image to be restored into the visual encoding and AU detection module, extract multi-scale visual feature representations and obtain the AU activation probability vector. Wherein, the face image to be restored... Figure 7 a and Figure 7 The visualization results of the multi-scale visual feature representation corresponding to b are as follows: Figure 8 a and Figure 8 As shown in b; Figure 7 a and Figure 7 The AU activation probability vectors corresponding to b are AU1 and AU2, respectively.

[0085] AU1:[

[0086] 0.3450134098529816, 0.2033716291189194, 0.0378810241818428,

[0087] 0.3132055103778839, 0.4929853975772858, 0.4022902548313141,

[0088] 0.1377994567155838,0.0380948483943939,0.0883205533027649,

[0089] 0.0414999835193157,0.0400922037661076,0.4035684764385223,

[0090] 0.0370470508933067,0.0342602469027042,0.2244472503662109,

[0091] 0.0406628847122192,0.0397996604442596,0.0389281585812569,

[0092] 0.0350953638553619,0.1011773720383644,0.0373144261538982,

[0093] 0.0329572968184948,0.0845512226223946,0.0875866413116455,

[0094] 0.037956353276968,0.0345995761454105,0.0391826555132866,

[0095] 0.0385126397013664,0.0349358767271042,0.0396638512611389,

[0096] 0.0381765924394131,0.0412753187119961,0.0320580676198006,

[0097] 0.0332185812294483,0.0383368618786335,0.0392054207623005,

[0098] 0.0410782173275948,0.0354478843510151,0.0336135439574718,

[0099] 0.0350070036947727,0.0352605022490025, 0.0390623584389687,

[0100] 0.0389149263501167,0.0355447083711624

[0101] ]。

[0102] AU2:[

[0103] 0.4123561382293701,0.3674925267696381,0.0473819598555565,

[0104] 0.1325841844081879,0.6419424271583557,0.7843176174163818,

[0105] 0.4629086256027222,0.0537190819382668,0.0941281616687775,

[0106] 0.1073453202843666,0.0582178607583046,0.7985630679130554,

[0107] 0.0493217781186104,0.2845072150230408,0.1189722046256065,

[0108] 0.0629144534468651,0.2365193963050842,0.0521730892360210,

[0109] 0.0614809468388557,0.1732045561075211,0.0721983015537262,

[0110] 0.0684510245919228,0.1623071879148483,0.1584935486316681,

[0111] 0.0412751734256744,0.0389161594212055, 0.0443822257220745,

[0112] 0.0365077815949917,0.0391284301877022,0.0426917336881161,

[0113] 0.0372546650469303, 0.0408136173188686, 0.0399728603661060,

[0114] 0.0435074903070927, 0.0381258092820644, 0.0419637188315392,

[0115] 0.0376892872154713, 0.0401529295444489, 0.0392844490706921,

[0116] 0.0420176573097706, 0.0371186062693596, 0.0398750342428684,

[0117] 0.0412069149315357, 0.0384927577376366

[0118] ].

[0119] (3) Input the AU activation probability vectors AU1 and AU2 into the semantic transformation and enhancement module respectively to generate structured guidance text.

[0120] Structured guiding text 1: The overall expression is close to neutral, with weak facial muscle activity;

[0121] Structured guiding text 2: Definite action: Raise cheeks, smile; Possible actions: Narrow eyes, smiling mouth, raise eyebrows.

[0122] (4) Input the above multi-scale visual feature representation and structured guidance text into the multimodal reasoning and guidance generation module, and output semantic guidance feature vector and expression label.

[0123] Semantic guided feature vector 1 (length 256): [

[0124] -0.1367204785346985, -0.556484043598175, 0.35849297046661377,

[0125] -0.038837648928165436, -0.22848358750343323, 0.28277474641799927,

[0126] -0.13552922010421753, 1.1362121105194092, -0.044281475245952606,

[0127] 0.7402400374412537,...

[0128] ].

[0129] Semantic guided feature vector 2 (length 256): [

[0130] 0.040011025965213776, -0.21171680092811584, 1.4294369220733643,

[0131] 1.0412861108779907, 0.031179506331682205, -0.02512134611606598,

[0132] 0.31401264667510986, 0.09535691142082214, 0.6546614766120911,

[0133] -0.3071514070034027,...

[0134] ].

[0135] Emoji tag 1: Neutral;

[0136] Emoji tag 2: Happy.

[0137] (5) Input the above multi-scale visual feature representation and semantically guided feature vector into the conditional generation module to generate a restored face image that conforms to physiological laws. The generated restored face images are as follows: Figure 9 a and Figure 9 As shown in b.

[0138] The above embodiments are exemplary and are intended to illustrate the technical concept and features of the present invention, so that those skilled in the art can understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made according to the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A multimodal face reconstruction and expression recognition system based on semantic guidance of facial action units, characterized in that, The core processing modules that work together include: visual encoding and AU detection module, semantic conversion and enhancement module, multimodal reasoning and guided generation module, and conditional generation module; The visual encoding and AU detection module performs multi-scale feature extraction on the face image to be restored and outputs multi-scale visual feature representation. At the same time, it uses graph neural network to model the correlation of facial action units and outputs AU activation probability vector representing the probability of facial muscle movement. The semantic conversion and enhancement module is connected to the visual encoding and AU detection module. It converts the AU activation probability vector into structured guiding text containing definite actions and possible actions through predefined biomechanical mapping rules, thereby realizing the conversion of numerical features into high-level interpretable semantic features. The multimodal reasoning and guidance generation module is connected to the visual encoding and AU detection module and the semantic conversion and enhancement module, respectively. It performs multimodal fusion on multi-scale visual feature representation, AU activation probability vector and structured guidance text, and uses multimodal reasoning ability to complete potential action units and infer muscle movement trends to generate semantic guidance feature vectors and expression labels. The conditional generation module is connected to the visual encoding and AU detection module and the multimodal reasoning and guided generation module, respectively. It receives multi-scale visual feature representation and semantic guided feature vector, injects the semantic guided feature vector as a conditional signal into the convolutional generation network to achieve guided image restoration, and outputs a restored face image that conforms to the physiological laws of facial muscle movement. The system employs multi-task joint optimization through pixel reconstruction loss, AU consistency loss based on biomechanical priors, and expression classification loss.

2. The system according to claim 1, characterized in that, The visual coding and AU detection module includes cascaded multi-scale visual coding units and facial motion unit detection branches. The multi-scale visual coding unit adopts a Transformer architecture based on a shifted window self-attention mechanism, and contains four feature extraction blocks in sequence. Each feature extraction block consists of a downsampling layer, a window multi-head self-attention layer and a shifted window multi-head self-attention layer in sequence, which is used to extract multi-scale visual feature representations from local texture to global structure. The facial motion unit detection branch is connected to the outputs of the second and third feature extraction blocks in the multi-scale visual coding unit to output the AU activation probability vector.

3. The system according to claim 2, characterized in that, The facial action unit detection branch includes a learnable graph network and a fully connected layer. The nodes of the graph network correspond to predefined facial action units, and the edge weights are initialized and dynamically updated through a trainable dependency matrix. The graph network realizes message passing and feature aggregation between nodes through a graph attention mechanism. Then, through linear mapping and nonlinear activation processing of the fully connected layer, the features are transformed into AU activation probability vectors.

4. The system according to claim 1, characterized in that, The semantic conversion and enhancement module includes a semantic rule mapping library and a text generation unit. The semantic rule mapping library stores mapping relationships between individual facial action units and corresponding biomechanical semantic phrases, mapping relationships between combinations of facial action units and corresponding biomechanical semantic phrases, and mapping relationships between combinations of facial action units and descriptions of complex expressions or muscle coordination patterns. This is used to achieve mapping from individual facial action units and combinations of facial action units to corresponding interpretable semantic descriptions. The text generation unit receives the AU activation probability vector and employs a three-segment judgment mechanism. It applies independent activation degree judgment rules to individual facial action units and combinations of facial action units based on preset high and low thresholds, respectively, resulting in three activation sets: "definitely activated," "potentially activated," and "not activated." Specifically, the judgment rule for individual facial action units is: those with an AU activation probability greater than or equal to the high threshold are classified into the "definitely activated" set, and those with an AU activation probability less than or equal to the low threshold are classified into the "not activated" set. Threshold values ​​are categorized into the "Inactive" set, while those between the low and high thresholds are categorized into the "Possibly Active" set. The determination rule for facial motion unit combinations is as follows: combinations where all AU activation probabilities are greater than or equal to the high threshold are categorized into the "Defined Active" set; combinations that simultaneously satisfy three conditions—no AUs with activation probabilities less than or equal to the low threshold, the number of AUs with activation probabilities greater than or equal to the high threshold not less than a preset proportion of the total number of AUs in the combination, and the existence of some AUs with activation probabilities between the low and high thresholds—are categorized into the "Possibly Active" set. Combinations that do not meet the above "Defined Active" or "Possibly Active" set conditions are categorized into the "Inactive" set. The text generation unit generates corresponding structured guiding text for the "Defined Active" and "Possibly Active" sets based on the mapping relationship of the semantic rule mapping library. The "Inactive" set does not participate in text generation. When the AU activation probabilities of all facial motion units are lower than the preset high threshold, a descriptive text representing a neutral expression is output.

5. The system according to claim 1, characterized in that, The multimodal reasoning and guided generation module includes a multimodal reasoning sub-network, a student text encoder, and a guided feature fusion layer in the reasoning stage; The multimodal inference subnetwork converts the AU activation probability vector into an AU semantic embedding vector through a semantic projector, maps the multi-scale visual feature representation into an AU visual embedding vector through a visual projection layer, and concatenates the two types of embedding vectors in the channel dimension before passing them through a feature fusion layer to obtain the multimodal fusion feature. The multimodal inference subnetwork is equipped with two parallel output heads: the expression classification head outputs expression labels through a fully connected layer and a Softmax function, and the semantic guidance head outputs AU visual-semantic fusion guidance vectors through a fully connected layer. The student text encoder encodes the structured guided text into a fixed-length text semantic embedding vector through a character-level embedding layer, a bidirectional gated recurrent unit, and a fully connected projection layer. The guided feature fusion layer concatenates the AU visual semantic fusion guided vector and the text semantic embedding vector in the feature dimension, and compresses them into a semantic guided feature vector of a preset dimension through linear mapping, which serves as the control signal for the conditional generation module.

6. The system according to claim 5, characterized in that, The multimodal reasoning and guided generation module adopts a two-layer architecture of teacher-guided learning during the training phase. On the basis of the reasoning phase architecture, a multimodal large language model and a chain-thinking reasoning control unit are added. Through the knowledge distillation mechanism, students can learn the common sense reasoning ability and semantic expression accuracy of the large model through the network.

7. The system according to claim 6, characterized in that, The multimodal large language model is a Transformer-based vision-language joint encoding architecture that integrates a visual encoder and a text encoder to achieve alignment of images and text in a unified semantic space, and outputs teacher semantic embedding vectors and guiding natural language descriptions. The chain-like thinking and reasoning control unit constructs and injects multi-step reasoning instructions to control the multimodal large language model to carry out two-level reasoning. The first-level reasoning completes the AU states that were not detected due to occlusion or degradation. The second-level reasoning infers the expression category based on the complete AU states and explains the reasoning. The semantic embedding vectors of the text output by the student text encoder are aligned with the semantic embedding vectors of the teacher output by the multimodal large language model by mean square error to achieve knowledge distillation; the multimodal inference subnetwork is optimized end-to-end under the common constraints of pixel reconstruction loss, AU consistency loss and expression classification loss.

8. The system according to claim 1, characterized in that, The conditional generation module adopts a step-by-step upsampling decoding structure with a decoding generation network with an embedded conditional modulation mechanism as its core. It includes three upsampling units, convolutional blocks corresponding to the upsampling units, conditional adaptive normalization sub-units, and output convolutional layers. After each upsampling unit restores the resolution of the feature map, the local texture information is extracted by the corresponding convolutional block, and then the corresponding conditional adaptive normalization subunit performs affine modulation with the semantically guided feature vector as the control signal. The feature map after three-level modulation is mapped to a three-channel face image through the output convolutional layer to obtain the restored face image; The conditional adaptive normalization subunit uses semantically guided feature vectors as control signals to generate channel scale coefficients and bias terms at the corresponding scale in real time, and performs affine modulation on the feature map of the current layer; the output of the output convolutional layer is processed by a normalized nonlinear function to obtain the final restoration result.

9. A multimodal face reconstruction and expression recognition method based on semantic guidance of facial action units, characterized in that, Includes the following steps: S1: Obtain the face image to be restored, perform multi-scale feature extraction on the face image to be restored, obtain multi-scale visual feature representation, and at the same time use graph neural network to model the correlation of facial action units, and output AU activation probability vector representing the probability of facial muscle movement. S2: The AU activation probability vector is semantically transformed and enhanced through predefined biomechanical mapping rules to generate structured guiding text containing definite actions and possible actions, thereby realizing the transformation of numerical features into high-level interpretable semantic features; S3: Multimodal fusion of the multi-scale visual feature representation, AU activation probability vector and structured guidance text, using multimodal reasoning capabilities to complete potential action units and infer muscle movement trends, generating semantic guidance feature vectors and expression labels; S4: Receive the multi-scale visual feature representation and semantic guidance feature vector, inject the semantic guidance feature vector as a conditional signal into the convolutional generator network to carry out guided image inpainting, and output a restored face image that conforms to the physiological laws of facial muscle movement. The method employs multi-task joint optimization through pixel reconstruction loss, biomechanical prior-based AU consistency loss, and expression classification loss to achieve synergistic enhancement of face restoration and expression recognition.

10. The method according to claim 9, characterized in that, The above method is implemented using the multimodal face restoration and expression recognition system based on facial action unit semantic guidance as described in any one of claims 1-8.

Images