Three-dimensional multi-state comprehensive grade evaluation method and system based on hybrid feature fusion

Abstract

The application discloses a three-dimensional multi-state comprehensive grade evaluation method and system based on mixed feature fusion, and relates to the technical field of feature evaluation. The three-dimensional multi-state comprehensive grade evaluation method and system based on mixed feature fusion comprises the following steps: S1, preprocessing and labeling original facial data; S2, point cloud reconstruction and dense and sparse structure generation are performed to construct three-dimensional data and two-dimensional expression; S3, enhancement, feature coding, projection mapping and modal comparison constraint are performed, and a classifier supervision and loss joint optimization are combined; S4, expression branch feature extraction, attention focusing and fusion layer channel regulation are performed; and S5, integration and feature layer fusion decision are performed. The application solves the problems that the existing facial paralysis evaluation technology has deficiencies in deep feature collaborative modeling of two-dimensional multi-view images and three-dimensional point cloud structure, consistent fusion of multi-expression dynamic mode and posture disturbance and noise suppression in a complex collection environment, and the accuracy and robustness of grade evaluation cannot meet actual requirements.

Description

Technical Field

[0001] This invention relates to the field of feature evaluation technology, specifically to a three-dimensional multi-state comprehensive level evaluation method and system based on hybrid feature fusion. Background Technology

[0002] With the rapid development of medical imaging technology, 3D facial modeling technology, and computer vision algorithms, the objective assessment of facial movement disorders has gradually become an important direction for clinical diagnosis, efficacy monitoring, and rehabilitation management. Currently, clinical assessment of facial paralysis patients still mainly relies on physicians' visual observation, providing subjective scores by judging facial symmetry, muscle strength, and localized movement limitations under different facial expressions. However, due to factors such as differences in physician experience, observation angle, lighting conditions, and patient cooperation, manual assessment still has inherent limitations in terms of consistency, repeatability, and refined expression. Meanwhile, scanning equipment is becoming increasingly widespread in medical settings, capable of acquiring various types of data, including facial expression changes, geometric deformations, and localized movement characteristics. This lays the foundation for conducting higher-dimensional and more objective data-driven facial function assessments. How to obtain stable and accurate facial function assessment results while maintaining clinical interpretability remains a key focus of current research and industrial applications.

[0003] For example, invention patent CN117274497A discloses a method for 3D face reconstruction from monocular images. This method constructs a dataset of normal face images and a dataset of asymmetric expression face images, combined with an encoder-decoder neural network architecture, to achieve 3D face reconstruction from monocular images. The method first collects and preprocesses normal face images, then constructs a neural network model based on the preprocessed data and performs weakly supervised training. Subsequently, it constructs a dataset of asymmetric expression face images and performs fine-tuning on the trained neural network model to generate the final model. Further, it constructs a validation set for performance verification and evaluates the reconstruction effect of the final model on a test set. Through the above training and verification process, the final model can be used for practical 3D face reconstruction tasks. This method achieves 3D face structure estimation under monocular image conditions, improves the model's adaptability to asymmetric expressions, and has high reconstruction accuracy and application value.

[0004] For example, the invention patent with publication number CN120953482A discloses a three-dimensional facial scan data modeling intelligent interactive fusion method and system. The method includes: S1, simultaneously acquiring visible light images and near-infrared light images of the patient's face; S2, generating a three-dimensional mesh model of the patient's epidermis based on the visible light images, and extracting subcutaneous tissue structure data based on the near-infrared light images; S3, acquiring multi-angle scan images of the patient's teeth and generating a three-dimensional model of the teeth; S4, using an artificial neural network model to identify facial feature points from the three-dimensional mesh model of the epidermis and extracting the contours of the tooth region; S5, spatially registering the three-dimensional model of the teeth with the contours of the tooth region, thereby segmenting and obtaining matching three-dimensional data of the tooth region; S6, fusing the three-dimensional mesh model of the epidermis, the subcutaneous tissue structure data, and the three-dimensional data of the tooth region to generate a three-dimensional facial anatomical structure model; and performing dynamic expression simulation on the above-mentioned three-dimensional facial anatomical structure model based on a facial muscle kinematics model, outputting a dynamic prediction sequence that changes over time.

[0005] Currently, in fields such as facial 3D reconstruction, multimodal representation learning, and facial state analysis, although existing methods can achieve a certain degree of geometric reconstruction and structural representation using monocular images, 3D meshes, point cloud data, or multi-angle scan information, significant shortcomings remain in multimodal data correlation mining and multi-expression and complex state modeling. There is a lack of effective alignment and fusion mechanisms between different modal features, making it difficult to fully leverage the complementary advantages of detailed textures and spatial structures. When faced with real-world scene data involving multiple expressions, asymmetric deformations, and pose changes, the models lack robustness and stability. Existing methods often rely on single-path or single-expression information, making it difficult to form a comprehensive facial state representation, thus limiting the accuracy of tasks such as grade assessment. This makes it difficult for current technologies to meet the requirements of high-precision, multi-state, and highly robust comprehensive face assessment.

[0006] Therefore, in order to address the above problems, there is an urgent need for a three-dimensional multi-state comprehensive level evaluation method and system based on hybrid feature fusion. Summary of the Invention

[0007] Technical problems to be solved

[0008] To address the shortcomings of existing technologies, this invention provides a three-dimensional multi-state comprehensive grade assessment method and system based on hybrid feature fusion. This method solves the problems of existing facial paralysis assessment technologies in terms of deep feature collaborative modeling of two-dimensional multi-view images and three-dimensional point cloud structures, consistent fusion of dynamic patterns of multiple facial expressions, and posture perturbation and noise suppression in complex acquisition environments. As a result, the accuracy and robustness of the grade assessment cannot meet the actual needs.

[0009] Technical solution

[0010] To achieve the above objectives, this invention employs the following technical solution: a three-dimensional multi-state comprehensive level evaluation method based on hybrid feature fusion, comprising: S1, acquiring original facial data and obtaining public image data; preprocessing and labeling the original facial data; S2, reconstructing the three-dimensional facial data into point clouds and generating dense and sparse structures, combining viewpoint projection and two-dimensional image normalization processes to construct three-dimensional point cloud matrix data and two-dimensional representations; S3, performing enhancement, feature encoding, projection mapping, and modal contrast constraints on the three-dimensional point cloud matrix data and two-dimensional representations through a dual-branch structure, and combining classifier supervision and loss joint optimization; S4, extracting deep structural features from the original facial data through expression branch, expression attention focusing, and channel control mechanisms of the fusion layer, obtaining comprehensive deep features, and performing joint optimization through hierarchical supervision; S5, executing the inference process of cross-modal paths and expression fusion paths during the testing phase, performing rule-based integration and feature layer fusion decision on the prediction results, and structurally archiving the output levels.

[0011] The second aspect of this invention provides a three-dimensional multi-state comprehensive level evaluation system based on hybrid feature fusion, including: a data acquisition, preprocessing, and annotation module for acquiring raw facial data and obtaining public image data; preprocessing and annotating the raw facial data; a three-dimensional facial modeling and multi-view processing module for reconstructing point clouds and generating dense and sparse structures from the three-dimensional facial data, and constructing three-dimensional point cloud matrix data and two-dimensional representations by combining view projection and two-dimensional image normalization processes; a point cloud image hybrid feature comparison learning module for performing enhancement, feature encoding, projection mapping, and modal comparison constraints on the three-dimensional point cloud matrix data and two-dimensional representations through a dual-branch structure, and combining classifier supervision and loss joint optimization; a structural feature parsing and fusion decision module for extracting deep structural features from the raw facial data through expression branches, expression attention focusing, and channel control mechanisms of the fusion layer, obtaining comprehensive deep features, and performing joint optimization through hierarchical supervision; and a level evaluation and optimization module for executing the inference process of cross-modal paths and expression fusion paths during the testing phase, performing rule-based integration and feature layer fusion decision on the prediction results, and structurally archiving the output levels.

[0012] Beneficial effects

[0013] The present invention has the following beneficial effects:

[0014] (1) This invention extracts differential facial expression features through facial expression branches and realizes the structured integration of deep representation in the fusion layer. This invention can capture deformation patterns under different facial expressions at the same time, thereby realizing a unified expression framework across facial expressions, effectively solving the shortcomings of existing technologies in modeling multiple facial expression states.

[0015] (2) By combining point cloud global structure representation with multi-view two-dimensional image detail features, this invention can simultaneously enhance feature representation capabilities in terms of spatial morphology and surface texture, thereby achieving accurate depiction of multi-dimensional facial states and effectively solving the problem of limited recognition performance caused by insufficient single-modal information in the prior art.

[0016] (3) In this invention, the classification cross-entropy loss is jointly optimized with multiple contrastive losses, so that the three dimensions of structure alignment, feature discrimination and hierarchical classification are supervised at the same time. This constructs a multi-objective collaborative optimization path, thereby achieving the effect of stable gradient signal and sufficient feature learning during the training stage. This effectively solves the problem of slow convergence and insufficient expressive power caused by a single loss in the existing technology.

[0017] (4) This invention establishes a stable feature space across conditions through point cloud enhancement, image enhancement and multi-view projection. This invention can maintain stable recognition performance under different acquisition environments and posture changes, thereby realizing reliable inference under complex clinical or field conditions, and effectively solving the problem of performance degradation caused by sensitivity to illumination and angle in the prior art.

[0018] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0019] Figure 1 The flowchart of the three-dimensional multi-state comprehensive level evaluation method based on hybrid feature fusion of the present invention is shown below;

[0020] Figure 2 This is a structural diagram of the three-dimensional multi-state comprehensive level evaluation system based on hybrid feature fusion of the present invention;

[0021] Figure 3 This is a schematic diagram of the three-dimensional and two-dimensional collaborative and hierarchical discrimination structure of the present invention;

[0022] Figure 4 This is a graph showing the relationship between the input vector, the classifier output probability, and the cross-entropy loss of this invention.

[0023] Figure 5 This is a diagram of the adaptive fusion architecture of the facial multi-expression branch analysis and fusion layer of the present invention;

[0024] Figure 6 This is an ablation analysis result diagram showing the impact of cross-modal contrastive learning on grade prediction in this invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Please see Figures 1-6 This invention provides a technical solution: a three-dimensional multi-state comprehensive level evaluation method based on hybrid feature fusion, comprising the following steps: S1, collecting original facial data and obtaining public image data; preprocessing and labeling the original facial data; S2, reconstructing the three-dimensional facial data into point clouds and generating dense and sparse structures, and constructing three-dimensional point cloud matrix data and two-dimensional representations by combining viewpoint projection and two-dimensional image normalization processes; S3, performing enhancement, feature encoding, projection mapping, and modal contrast constraints on the three-dimensional point cloud matrix data and two-dimensional representations through a dual-branch structure, and combining classifier supervision and loss joint optimization; S4, obtaining comprehensive depth features from the original facial data through deep structural feature extraction of expression branches, expression attention focusing, and channel regulation mechanisms of the fusion layer, and performing joint optimization through hierarchical supervision; S5, executing the inference process of cross-modal paths and expression fusion paths during the testing phase, performing rule-based integration and feature layer fusion decision on the prediction results, and structurally archiving the output levels.

[0027] Specifically, the process involves collecting raw facial data and acquiring public image data. The preprocessing and annotation of the raw facial data is as follows: Raw facial data is collected, including 3D facial data, batch size, identification number data, and expression type identifiers. Public image data is acquired, including the ImageNet dataset. Each identification number corresponds to a set of 3D facial data, containing four types of facial motion data. A high-precision 3D facial acquisition device, the WS-BM321, is used to collect the 3D facial data, including four types of facial motion data: eyebrow raising, teeth showing, nose twitching, and cheek puffing. Each set of four types of facial motion data forms an array. K-fold cross-validation is performed to evaluate the stability of different partitioning methods. The K value is obtained through preliminary analysis of the dataset size and optimization based on the cross-validation results, ranging from 5 to 10. The partitioning strategy is then determined based on the cross-validation results, dividing each set of 3D facial data into a training set, a test set, and a validation set. The training set is used for parameter learning, the validation set is used for hyperparameter selection and loss weight search, and the test set is used for final performance evaluation. The 3D facial data is stored in WRL format.

[0028] The 3D facial data underwent denoising, resampling, and coordinate correction through 3D structural calibration and data format standardization to eliminate random noise and local irregularities generated during acquisition. Centroid localization, translation calibration, and scale unification algorithms mapped the 3D point cloud data to a unified standard coordinate system, improving the comparability of the data in spatial structure. Data consistency checks were performed on each array through the association verification of identity numbers and expression type identifiers, ensuring that the 3D data matched its corresponding patient information and expression category. Formatting and data integrity checks eliminated missing point clouds, duplicate identifiers, and arrays with acquisition anomalies. This provides reliable and structurally standardized input data for subsequent in-depth analysis and computation. The House-Brackmann grading system was used to rate and label the original facial data as a supervision label. Combining multi-classification and mapping algorithms, the grade level was limited to the range of 'a' to 'b'. Expression type identifiers were used to support the feature learning process of expression branches. All four expression data shared the same 'HB' grade label, meaning the supervision label for each of the four expressions was consistent with the 'HB' grade.

[0029] In this implementation plan, a high-quality raw facial data system covering identity number data, expression type identifiers, batch size, and four types of facial motion data was constructed, achieving high-fidelity acquisition and structured processing of facial 3D information. A multi-expression 3D point cloud dataset with spatial alignment, reliable structure, and accurate labels was established through 3D structure calibration, data format standardization, centroid localization, scale unification, and consistency verification. A stable training system suitable for parameter learning, hyperparameter search, and performance evaluation was formed through a K-fold cross-validation-driven data partitioning mechanism. A unified and standardized supervision signal system was established through House-Brackmann grading, providing a traceable, highly consistent, and highly standardized input foundation for subsequent feature learning, modality contrast constraints, classifier training, and fusion decision-making, ensuring the reliability, generalization, and interpretability of the entire grading evaluation process.

[0030] Specifically, the process of reconstructing 3D facial data into point clouds and generating dense / sparse structures, combined with viewpoint projection and 2D image normalization, involves the following steps: Analyzing the mesh vertices, faces, and their topology of the 3D facial data; extracting vertex coordinates from each vertex according to its spatial position to obtain a set of point cloud coordinates; creating a 3D facial model from the input point cloud data; and representing the 3D facial model as dense and sparse point clouds using a combination of uniform filtering based on mesh vertex resampling and farthest point sampling. The number of point clouds in each array is uniformly sampled to N points, with the value of N determined based on the size and feature density of the point cloud dataset. Finally, a clustering algorithm is applied to the original point clouds. The values ​​are obtained, ranging from 3000 to 5000, to ensure the consistency of model input and controllable computational load. After sampling, the point cloud data is stored in the form of an N×3 matrix, where N represents the number of points and 3 represents the three-dimensional coordinate components. The centroid position of the point cloud is calculated, and the entire point cloud is translated so that its centroid is translated to the origin of the coordinate system. The point cloud is uniformly scaled, and the maximum offset of the point cloud in the three coordinate axes is calculated with the centroid as the reference. The global maximum value of the maximum offset in each direction is taken as the scale reference, and its reciprocal is used as the scaling factor to linearly scale all points. By adding a minima ε to avoid the division by zero problem, the overall scale is normalized to the interior of the unit sphere to obtain the three-dimensional point cloud matrix data, making point cloud data of different three-dimensional point cloud data and different acquisition scales comparable.

[0031] A 3D facial model is projected using multi-view projection technology to generate n 2D RGB images. The value of n, ranging from 5 to 8, is obtained through an information gain-based viewpoint selection algorithm based on the number of viewpoints and spatial coverage. This enriches the image modal data and improves the 2D network's ability to capture geometric details. The 2D RGB images are then resized to a uniform c×c pixel size to fit the subsequent convolutional neural network structure. Image normalization is performed, including pixel value normalization and channel normalization. Statistical values ​​from the ImageNet dataset are used as the normalized mean and variance to standardize the pixel distribution of the normalized images, thereby improving network training stability and convergence speed.

[0032] In this implementation plan, this step constructs a spatially consistent, resolution-uniform, and topologically standardized 3D point cloud matrix by performing structural analysis, point cloud reconstruction, and scale normalization on the 3D facial data. Data scale alignment and controllable computational complexity are achieved through dense and sparse point cloud resampling strategies. A standardized 3D representation system is constructed across individuals and acquisition conditions through centroid correction and unit sphere normalization. A 2D RGB image is generated through multi-view projection and image normalization is completed, constructing a 2D representation form complementary to the 3D modality. This provides a structurally unified, scale-consistent, and information-rich data foundation for subsequent feature encoding, cross-modal contrast constraints, and deep network training, ensuring that the model can fully capture 3D geometric structure and 2D texture features, achieving stable and efficient multimodal feature learning.

[0033] Specifically, the process of performing enhancement, feature encoding, projection mapping, and modal contrast constraints on 3D point cloud matrix data and 2D representation through a dual-branch structure is as follows:

[0034] In this embodiment, as Figure 3 The diagram illustrates the 3D / 2D collaborative and graded discrimination structure, showcasing a multimodal learning framework. Point cloud and image data are used for feature extraction via a dual-branch structure, and cross-modal contrastive learning achieves intermodal fusion and consistency learning. Through joint optimization of the point cloud and image branches, key features for facial paralysis assessment can be learned from 2D images and 3D point clouds. Furthermore, cross-modal and intramodal losses further enhance the complementarity and synergy between the two modalities. Finally, a classifier outputs the predicted facial paralysis grade, ensuring the accuracy and consistency of the facial paralysis grade assessment.

[0035] In the training set, the dual-branch structure includes a point cloud branch and an image branch: the point cloud branch performs data augmentation operations on the 3D point cloud matrix data, including: random rotation, random translation, adding noise, and randomly dropping points, for each point cloud data. Two different augmentation operations are performed on the same point cloud data to obtain two augmented point cloud data, which are used for subsequent intramodal instance discrimination and comparative learning. Using a graph convolutional neural network as the backbone, augmented point cloud data is input. Multi-scale local structural features are extracted through graph convolution operations based on a k-nearest neighbor graph. A global representation is formed through multi-layer feature concatenation, and then converged into two fixed-length m-dimensional global feature vectors using global max pooling and global average pooling. This process can be denoted as the point cloud feature extraction mapping operator. The global feature vectors are then dimensionality-reduced through a fully connected layer, mapping them to an e-dimensional embedding space to obtain two projected mapping vectors. The single-array contrast loss is calculated as follows: the similarity between different augmented views of the same array is calculated and used as the positive array similarity; subsequently, the similarity between the array and other arrays within the batch under their respective augmented views is calculated and used as the negative array similarity; then, the positive and negative array similarities are combined into a softmax-form normalized ratio. By taking the negative logarithm of the positive array similarity, the two augmented features of the same array are made closer in the embedding space while maintaining distinction from other arrays, thus obtaining the single-array contrast loss, which is used to constrain instance consistency learning under point cloud modalities. The specific calculation formula is as follows:

[0036] ;

[0037] in, This represents the single-array contrastive loss, used to measure the projection consistency of the same array under different augmented views, and serves as a core metric for learning stable structural representations. , Representing two projection mapping vectors, it is possible to characterize the key geometric patterns and semantic orientation information of an array in a low-dimensional space; This indicates the batch size, which directly affects the number of negative arrays, thus impacting the separability of feature distributions and the richness of gradient updates in contrastive learning. The temperature coefficient is selected through validation set optimization, with a value ranging from 0.03 to 0.2. It is used to adjust the smoothness of the similarity distribution, focusing on global structure perception or local fine-grained discrimination ability under high temperature and low temperature conditions, respectively. Cosine similarity is a measure that measures the difference in the angle between vectors in the normalized space, which helps to form an angle-sensitive distinguishing boundary in the projected space. This represents the index of arrays other than array i, used to represent the point cloud features corresponding to the negative array.

[0038] Based on the single-array contrastive loss, the contrastive results of all arrays are summarized and averaged within the batch range to form the intra-modal instance resolution loss, which is used to constrain the stability and robustness of point cloud features under augmentation perturbations. The intra-modal instance resolution loss, as one of the key training objectives of the point cloud branch, participates in parameter updates together with the cross-modal instance resolution loss and the classifier loss.

[0039] The image branch performs data augmentation on 2D RGB images, including color perturbation and random flipping operations, to obtain augmented image data, thereby enhancing the model's robustness to changes in illumination and pose. Using a ResNet50 network as the backbone, the augmented image data is input, and semantic features are extracted through multi-layer convolution, pooling, and residual blocks, outputting an m-dimensional global feature vector. This global feature vector is then reduced to e-dimensionality via a fully connected layer to obtain the image projection vector. The two projection mapping vectors are then element-wise averaged across their corresponding dimensions to obtain the projection prototype vector for the point cloud modality. A category-aware projection prototype construction method is used to adjust the construction of the projection prototype vector based on the distribution characteristics of different categories. Finally, based on... The cross-modal contrast loss is defined using the projected prototype vector and the image projection vector: The similarity between the projected prototype vector and the image projection vector of the same array is calculated as the positive cross-modal array similarity; then, the similarity between the projected prototype vector and the image projection vectors of other arrays in the batch is calculated as the negative cross-modal array similarity; finally, both positive and negative array similarities are substituted into the normalized ratio in the softmax form under temperature scaling. By taking the negative logarithm of the positive array similarity, the features of the same array in the point cloud modality and image modality are made closer in the embedding space, while remaining separated from other arrays, thus obtaining the cross-modal contrast loss, which is used to constrain the alignment and consistency learning of features from the two modalities. The specific calculation formula is as follows:

[0040] ;

[0041] ;

[0042] in, Representing the cross-modal contrastive loss, it measures the degree of matching between the point cloud projection prototype vector and the image projection vector in the shared embedding space. It is the core constraint term of cross-modal consistency learning, used to drive the two modalities to tend to align in the feature space. It represents the image projection vector, which retains the encoding information in the image that has discriminative power over facial morphology and local expression changes, and is used for similarity calculation in cross-modal contrastive learning; This represents the projection mapping vector, characterizing the stable structural features and enhanced invariance of the array, providing a foundation for constructing robust representations within point cloud modes; The projection prototype vector represents the centered feature representation of the point cloud modality in contrastive learning. By fusing the encoding results of two enhanced views, the cross-modal matching process can rely on more consistent modal representative features. This indicates the batch size, which directly affects the number of negative arrays, thus impacting the separability of feature distributions and the richness of gradient updates in contrastive learning. The temperature coefficient is selected through validation set optimization, with a value ranging from 0.03 to 0.2. It is used to adjust the smoothness of the similarity distribution, focusing on global structure perception or local fine-grained discrimination ability under high temperature and low temperature conditions, respectively. Cosine similarity is a measure that measures the difference in the angle between vectors in the normalized space, which helps to form an angle-sensitive distinguishing boundary in the projected space. This represents the index of arrays other than array i, used to represent the point cloud features corresponding to the negative array.

[0043] Based on cross-modal contrastive loss, a cross-modal instance discrimination loss is obtained by summarizing and averaging the cross-modal contrastive losses of all arrays within a batch. This loss is used to encourage close alignment between point clouds and image features of the same expression in the embedding space. A dynamic training strategy based on challenging array mining is employed: the array distribution in the contrastive learning loss is used, and the training priority adjustment rule is based on the similarity and loss value of the arrays. Specifically, by calculating the loss of each array and its distribution in the batch, arrays with larger losses or higher challenges are prioritized for training. This method does not rely on traditional decision rules such as positive and negative similarity thresholds, quantiles, and Top-m selection; instead, it optimizes array selection during the training process through the feature distribution of the data itself.

[0044] In this implementation scheme, a unified embedding space representation mechanism between 3D point cloud modalities and 2D image modalities is established by constructing a dual-branch collaborative learning structure consisting of point cloud branches and image branches, achieving consistency constraints and structural alignment of cross-modal features. Intramodal instance discrimination loss enhances the stable expressive ability of point cloud features under various enhancement perturbations, forming a point cloud modal representation with structural invariance. Cross-modal instance discrimination loss ensures a high degree of matching between the point cloud projection prototype vector and the image projection vector in the representation space, significantly improving the correlation and complementary feature utilization capabilities between the two modalities. Parallel computation of positive and negative array similarities enables the model to form a clear and separable boundary structure in the feature space, strengthening the discriminative ability of the representation. Collaborative optimization of point cloud and image features allows 3D geometric structural features and 2D texture semantic features to be integrated within a unified framework, improving the integrity, robustness, and generalization ability of the overall feature encoding. End-to-end optimization processes make cross-modal contrastive learning the core driving force for stable training, providing a structurally consistent, semantically sufficient, and highly discriminative deep feature foundation for subsequent classification and ranking discrimination modules.

[0045] Specifically, the joint optimization process combining classifier supervision and loss is as follows: the global feature vector of the point cloud is used as the input to the classifier. Any one of the global feature vectors of the point cloud can be selected and passed through two fully connected layers and a Dropout layer to output the original s-dimensional classification score; the original classification score is mapped to the predicted probability distribution of the HB level of the corresponding expression through the softmax function to obtain the classifier's predicted level; the classifier's predicted level and supervision label are input, the negative logarithm of the correct category prediction probability is taken and averaged within the batch range, and the classification cross-entropy loss is calculated to quantify the deviation between the predicted level and the true level; the classification cross-entropy loss is used as a supervision signal to optimize the classification result.

[0046] Total loss function The loss is the sum of the intra-modal instance discrimination loss, cross-modal instance discrimination loss, and classification cross-entropy loss. Backpropagation is used to jointly update the parameters of the point cloud branch, image branch, and classifier. During training, the PyTorch deep learning framework is used, with an initial learning rate of... The learning rate is adjusted using a cosine annealing strategy, the training period is 100, and the optimizer is Adam.

[0047] In this embodiment, Table 1 is an example table of the output and loss response characteristics of the point cloud branch classifier. It records in detail the original classification scores, softmax output probabilities, prediction levels, supervision labels, and calculated classification cross-entropy loss of different input point cloud feature vectors during the classification process, which is used to illustrate the role mechanism of feature vectors in the model decision-making stage. Wherein: the original classification score of input feature vector F1 is [2.1, 1.3, 0.8, 0.2], the softmax output is [0.52, 0.28, 0.15, 0.05], the prediction level is 2, consistent with the supervision label, and the classification cross-entropy loss is 0.654; the original classification score of input feature vector F2 is [1.0, 1.6, 1.2, 0.4], the softmax output is [0.29, 0.40, 0.24, 0.07], the prediction level is 3, the supervision label is 3, and the loss is 0.916; the original classification score of input feature vector F3 is [0.5, 1.4, 2.2, 1.1], and the softmax output is... The prediction level is 4, the supervision label is 4, and the loss is 0.733. The original classification score of the input feature vector F4 is [1.8, 0.9, 1.0, 1.4], the softmax output is [0.39, 0.18, 0.20, 0.23], the prediction level is 2, which is inconsistent with the supervision label 3, causing the loss to rise to 1.204. The original classification score of the input feature vector F5 is [2.4, 1.9, 1.0, 0.7], the softmax output is [0.47, 0.32, 0.13, 0.08], the prediction level is 2, which is consistent with the supervision label, and the loss is 0.756. The table shows the correspondence between point cloud feature representation, classifier output probability structure and loss feedback, which can be used to explain the response characteristics of the classification model under different feature input conditions.

[0048] Table 1. Examples of Output and Loss Response Characteristics of Point Cloud Branch Classifiers

[0049] <![CDATA[F1]]> [2.1,1.3,0.8,0.2] [0.52,0.28,0.15,0.05] 2 2 0.654 <![CDATA[F2]]> [1.0,1.6,1.2,0.4] [0.29,0.40,0.24,0.07] 3 3 0.916 <![CDATA[F3]]> [0.5,1.4,2.2,1.1] [0.10,0.22,0.48,0.20] 4 4 0.733 <![CDATA[F4]]> [1.8,0.9,1.0,1.4] [0.39,0.18,0.20,0.23] 2 3 1.204 <![CDATA[F5]]> [2.4,1.9,1.0,0.7] [0.47,0.32,0.13,0.08] 2 2 0.756

[0050] like Figure 4The graph shows the relationship between the input vector, the classifier's output probability, and the cross-entropy loss. A dual-axis broken-line structure is used to display the trend of predicted probability changes and the magnitude of loss response for different feature vectors. The solid line represents the change of predicted probability with feature vector corresponding to the supervision level, and the dashed line represents the trend of cross-entropy loss. As shown in Table 2, F1 to F3 have higher predicted probabilities and lower losses, reflecting the model's stable recognition ability for this type of feature. When the input is F4, the incorrect prediction level leads to a significant increase in loss. When the input is F5, the predicted probability rebounds and the loss decreases, indicating that the model regains strong discriminative ability in this feature range. This graph visually demonstrates the coupling relationship between feature vectors and model output behavior, providing a visual basis for analyzing the classifier's discriminative sensitivity under different feature inputs.

[0051] In this implementation scheme, a classifier is used to discriminate the global feature vectors of the point cloud, constructing a mapping mechanism from embedded representation to grade prediction, thus achieving an interpretable output expression for facial paralysis grades. Direct supervision is provided through classification cross-entropy loss, enabling the model to continuously correct the deviation between the predicted distribution and the true grade during training, strengthening the separability of feature representation and grade discrimination ability. By jointly constructing the total loss function from intra-modal instance discrimination loss, cross-modal instance discrimination loss, and classification cross-entropy loss, cross-modal consistency learning and the final classification task converge collaboratively under the same optimization objective, promoting a more stable collaborative expression structure of the point cloud branch, image branch, and classifier in the parameter space. During training, a cosine annealing learning rate strategy and the Adam optimizer are combined to achieve smooth control of gradient updates, improving training convergence efficiency and parameter optimization quality, strengthening the model's stable prediction ability under complex facial expression conditions, and providing a highly discriminative and robust classification foundation for the final grade evaluation.

[0052] Specifically, the process of extracting deep structural features from the original facial data through expression branches and focusing expression attention is as follows: Inputting 3D point cloud matrix data, data augmentation is performed on the four types of expression point clouds, including random rotation, translation, noise addition, and random point dropping operations, outputting augmented expression point cloud data to improve robustness to changes in expression posture and acquisition noise; each expression branch shares the same feature extraction backbone network built on a graph convolutional neural network, used for unified encoding and extraction of deep structural features from the augmented point cloud data; each expression branch independently processes the augmented expression point cloud data, outputting a branch global feature vector of length m through graph convolution and multi-scale feature aggregation; each expression branch connects to an expression attention layer, inputting the branch global feature vector, assigning different channel weights to the expression feature channels, using a strategy based on feature importance evaluation. By analyzing the responses of each expression category under different channels, a gradient-weighted feature normalization algorithm is used to quantify the contribution of each channel, and weights are assigned according to the contribution, with values ​​ranging from 0 to 1. The channel focuses on the most discriminative feature components for facial paralysis rating, outputting a weighted expression feature vector. Each expression branch is connected to a classifier after the expression attention layer, which consists of a fully connected layer and a Dropout layer. The input is the weighted expression feature vector, the output is an s-dimensional original classification score, which is then mapped to the probability distribution of the corresponding HB level using a softmax function to obtain the branch prediction result. For each branch prediction result, the branch cross-entropy loss between the prediction and the supervision label is calculated to form an auxiliary supervision signal for the corresponding expression. The expression attention layer adopts a variant structure of the Squeeze-and-Excitation module, performing channel compression and re-weighting operations on the feature map using a scaling factor r=16 and a kernel size k=3 to enhance the model's selective attention to expression features. During the channel weight calculation process, a fully connected layer sigmoid activation function is used to learn the channel weights, and a stabilization strategy is combined to optimize the weight allocation, ensuring that the features are stable and have strong discriminative ability during training. The specific calculation formula is as follows:

[0053]

[0054] In the formula, represents the branch cross-entropy loss, used to measure the deviation between the predicted and true levels of the j-th class of expression in independent branches, and serves as a local supervision signal to guide the feature learning direction of that expression branch; G represents the total number of categories in the classification task, taken as 4. The supervisory label is used to ensure that the loss function makes an effective gradient contribution to the correct category. The branch prediction results are used to reflect the confidence level of each level. The House-Brackmann facial paralysis level is from a to b. j represents the expression type label, which is 1-4, representing different expressions. It is used to distinguish the independent training paths of the four groups of expression features and to ensure that each expression receives targeted supervision and constraints, so as to improve the sensitivity and expressive ability of the branch level to the differences in expression features.

[0055] In this implementation scheme, an independent feature modeling mechanism for four types of facial expression branches enables fine-grained analysis of structural features under different facial movement patterns. Each expression branch forms its own feature encoding path based on a shared feature extraction backbone network, allowing the model to capture the differences in spatial geometry, muscle movement trajectories, and local deformation patterns of each expression. The expression attention layer further strengthens the response weights of key channels, focusing branch features on discrimination regions highly correlated with facial paralysis levels, thus improving sensitivity to slight facial asymmetry and subtle muscle group changes. The auxiliary supervision signal provides independent optimization objectives for each type of expression through branch cross-entropy loss, enabling the model to simultaneously learn the structural differences and hierarchical correlation features of the four types of expressions during training, improving overall expression consistency and diagnostic discrimination capabilities in multi-expression scenarios. Through this mechanism, the model achieves stronger expression-specific modeling capabilities and higher hierarchical expression stability, establishing a more reliable branch-level deep feature foundation for the final fusion decision.

[0056] Specifically, the channel control mechanism of the fusion layer obtains the comprehensive deep features as follows: the weighted facial feature vectors are concatenated along the feature dimension to form a combined feature representation, which includes the comprehensive deep features of four facial expressions; based on the channel attention mechanism, the combined features are weighted and learned to learn the relative importance of the four facial features in different channels, and the intermediate representation after channel weighting is output; the combined features are superimposed and fused according to the channel weights to obtain a comprehensive feature representation, which reflects the overall severity of facial paralysis in patients with multiple facial expressions.

[0057] The input comprehensive features are processed through fully connected layers and Dropout layers, outputting s-dimensional original classification scores. Nonlinear mapping and dimensionality transformation are applied to the fused deep features to enhance their class separability. Dropout suppresses overfitting and enhances the model's generalization ability across different facial states. After softmax mapping, the final HB level probability distribution is obtained. Softmax normalizes the output vector into probabilistic form, yielding the comprehensive prediction result. For each comprehensive prediction result, the label cross-entropy loss between the two is calculated based on the supervision label, providing a direct global supervision signal to the fusion layer to continuously enhance the discriminative ability of the comprehensive features in the decision space. The cross-entropy loss of the fusion layer is used to supervise the output, ensuring that the final comprehensive features maintain consistency and stability in level prediction even after aggregation of multi-expression and multi-scale features. The specific calculation formula is as follows:

[0058]

[0059] in, The label cross-entropy loss represents the deviation between the probability distribution output by the fusion layer and the House-Brackmann rank, and is the core optimization objective of the final decision layer; G represents the total number of categories in the classification task, which is set to 4. The supervisory label is used to ensure that the loss function makes an effective gradient contribution to the correct category. The branch prediction results represent the confidence level of each level, representing House-Brackmann facial paralysis levels a to b.

[0060] like Figure 5 The diagram shows the architecture of the facial expression branch parsing and fusion layer adaptive fusion system, illustrating the structure of the facial paralysis assessment system. Multiple expression networks work together to analyze facial expressions and predict the level of facial paralysis. Expression features are extracted using a DGCNN network, and an attention-weighted layer focuses on features with high discriminative power for paralysis level prediction. Each expression branch independently learns specific expression information, and the Meta-Expert fusion layer comprehensively analyzes the outputs of these expressions to arrive at the final prediction result. By introducing auxiliary supervision signals and a fusion loss function, efficient multi-task learning across multiple expression feature dimensions is achieved, improving the accuracy and stability of facial paralysis assessment.

[0061] In this implementation scheme, a fusion layer unifies and integrates the deep features of four types of facial expressions, achieving collaborative discrimination capabilities across expressions and multi-scale structural information. Guided by a channel attention mechanism, the combined features form a differentiated weight distribution, enabling the model to automatically highlight key channels that contribute more to grade recognition and weaken redundant or noisy components, thereby constructing a more robust joint representation of multiple facial expressions. After the integrated features enter the decision-making stage, a fully connected structure completes feature compression and category separation in high-dimensional space, allowing the fused representation to form a clearer discriminative structure at the boundary of facial paralysis grade distinction. The cross-entropy loss of the fusion layer provides global supervision constraints, ensuring that the four types of facial expression information maintain a consistent prediction direction and grade stability after superposition and fusion, strengthening the joint mapping capability between overall facial movement patterns, local asymmetric changes, and structural features. Ultimately, this mechanism enables the model to obtain a more comprehensive structural understanding and higher grade prediction reliability in multi-expression scenarios, providing a unified and robust feature decision-making foundation for the comprehensive assessment of facial paralysis severity.

[0062] Specifically, the joint optimization process through hierarchical supervision is as follows: The branch cross-entropy loss and the fusion layer cross-entropy loss are weighted and summed to obtain the total fusion loss. The branch loss weights and fusion layer loss weights are obtained through a validation set-driven hyperparameter search. The accuracy and stability of grade prediction on the validation set are evaluated by assessing the weight combinations. The branch loss weights range from 0.2 to 0.3, and the fusion layer loss weights range from 0.9 to 1.0. The total fusion loss is then used to jointly optimize the shared graph convolutional neural network backbone, expression branch attention layer and classifier, channel attention units in the fusion layer, and the fusion layer classifier parameters via backpropagation. During training, the PyTorch deep learning framework is used; the initial learning rate is... The learning rate is adjusted using a cosine annealing strategy; the training cycle is 100; the optimizer is Adam; each training batch contains a set of three-dimensional point cloud matrix data, which are four types of facial expression point cloud data. The four facial expression branches are processed in parallel, and the fusion layer module is used to fuse them to form a multi-task joint optimization process.

[0063] In this implementation scheme, a unified training loop for multi-expression, multi-branch, and multi-level structures is formed through a joint optimization mechanism that integrates the total loss. This enables the feature extraction backbone network, expression branches, attention modules, and fusion layers to converge collaboratively under the same objective. The weighted loss framework ensures that the fine-grained expression discrimination capability of local branches and the global level decision-making capability of the fusion layer are simultaneously enhanced, achieving a stable balance between local difference learning and overall level discrimination. During multi-task parallel training, the network structure is continuously and dynamically adjusted between shared expressions and expression-specific features, resulting in a final feature space with higher discrimination consistency and generalization robustness. The overall training process gradually compresses noise-sensitive dimensions and enhances key structural dimensions during optimization, enabling the model to form more explicit facial movement pattern representations. Ultimately, this constructs an automatic facial paralysis assessment model with high reliability, high consistency, and high level discrimination capability, providing a joint feature representation with good convergence, structural stability, and high prediction reliability for subsequent inference stages.

[0064] Specifically, the process of executing the inference flow of cross-modal paths and expression fusion paths during the testing phase, performing rule-based integration and feature layer fusion decisions on the prediction results, and structurally archiving the output levels is as follows: During the testing phase, the inference flow of cross-modal paths and expression fusion paths is executed: only the point cloud branch and classifier are called, and the classifier prediction level is output; the dimensional features of the classifier prediction levels of the four expressions are integrated, and a refined level mapping algorithm is applied. Based on the comprehensive multidimensional features, the prediction level of each expression is globally consistent and fused to obtain the final patient-level facial paralysis level. For example, if a patient's eyebrow raising, teeth showing, nose twitching, and cheek puffing are predicted to have levels of 2, 2, 3, and 3 respectively, the final facial paralysis level is level 2. The expression hybrid rating framework calls the expression branch and fusion layer for rule-based integration to obtain a comprehensive prediction result and output the final patient-level facial paralysis level. At the same time, a prediction result database is created to record all prediction results, providing a reliable data foundation for algorithm verification and performance improvement.

[0065] In this embodiment, Table 2 shows the classification accuracy of the cross-modal contrastive learning framework and the single-modal method. In the evaluation of four facial expressions of 22 patients, the average accuracy of the cross-modal contrastive learning framework for four facial expressions reached 41.67%, which is significantly better than typical single-modal and traditional 3D methods, such as PointMLP's 38.64% and DGCNN's 39.39%.

[0066] Table 2. Classification accuracy of cross-modal contrastive learning framework and single-modal method.

[0067] SVM+LBP 32.00 PointNN 36.64 PointNet++ 37.04 PointMLP 38.64 DGCNN 39.39 This invention 41.67

[0068] like Figure 6The figure shown is an ablation analysis result of the impact of cross-modal contrastive learning on grade prediction. It demonstrates that the ablation experiment results prove that the multimodal hybrid framework extracts deep features of different expressions through four expression branches, and then achieves intelligent fusion of multi-expression features through an expression attention layer and a channel attention mechanism. This dual-innovative design not only overcomes the sensitivity of 2D methods to lighting and pose, but also compensates for the shortcomings of traditional 3D methods that rely on handcrafted features and lack global structural modeling capabilities, achieving comprehensive feature learning and classification of multimodal data and multi-expression information.

[0069] In this implementation scheme, this step establishes a final decision-making mechanism in the inference phase that combines cross-modal structure and multi-expression fusion strategy, enabling grade prediction, result integration, and performance verification without additional manual intervention. Through robust structural representation of point cloud branches, fine-grained motion difference capture of expression branches, and the multi-expression collaborative modeling capability of the fusion layer, stable and consistent grade outputs were obtained during testing, demonstrating significantly superior predictive performance compared to single-modal methods in assessments of multiple patients. The feature alignment relationship established between global geometric representation and image semantic supplementation by the cross-modal structure gives the final prediction higher discriminative power. The multi-expression fusion path forms a more comprehensive grade response pattern in the multi-dimensional feature space of facial motion, exhibiting higher sensitivity and stability to grade changes in different facial states. The prediction result database output in this stage provides a structured and reproducible data foundation for algorithm comparison, performance analysis, and iteration, enabling the overall system to demonstrate stronger robustness, verifiability, and comprehensive evaluation capabilities in application.

[0070] like Figure 2 As shown, the second aspect of this invention provides a three-dimensional multi-state comprehensive level evaluation system based on hybrid feature fusion, including a data acquisition, preprocessing, and annotation module for acquiring raw facial data and obtaining public image data; preprocessing and annotating the raw facial data; a three-dimensional facial modeling and multi-view processing module for reconstructing point clouds and generating dense and sparse structures from the three-dimensional facial data, and constructing three-dimensional point cloud matrix data and two-dimensional representations by combining view projection and two-dimensional image normalization processes; a point cloud image hybrid feature comparison learning module for performing enhancement, feature encoding, projection mapping, and modal comparison constraints on the three-dimensional point cloud matrix data and two-dimensional representations through a dual-branch structure, and combining classifier supervision and loss joint optimization; a structural feature parsing and fusion decision module for extracting deep structural features from the raw facial data through expression branches, expression attention focusing, and channel control mechanisms of the fusion layer, obtaining comprehensive deep features, and performing joint optimization through hierarchical supervision; and a level evaluation and optimization module for executing the inference process of cross-modal paths and expression fusion paths during the testing phase, performing rule-based integration and feature layer fusion decision on the prediction results, and structurally archiving the output levels.

[0071] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0072] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A three-dimensional multi-state comprehensive level evaluation method based on hybrid feature fusion, characterized in that, include: S1, Collect raw facial data and obtain public image data; Preprocess and label the raw facial data; S2 performs point cloud reconstruction and dense / sparse structure generation on 3D facial data, and combines view projection and 2D image normalization process to construct 3D point cloud matrix data and 2D representation. S3 performs enhancement, feature encoding, projection mapping, and modality contrast constraints on 3D point cloud matrix data and 2D representation through a dual-branch structure, and combines classifier supervision and loss joint optimization. S4 extracts deep structural features from the original facial data through expression branch, expression attention focusing, and channel regulation mechanism of the fusion layer to obtain comprehensive deep features, and performs joint optimization through hierarchical supervision. S5 executes the inference process of cross-modal path and expression fusion path during the testing phase, performs rule-based integration and feature layer fusion decision on the prediction results, and performs structured archiving of the output level.

2. The three-dimensional multi-state comprehensive level evaluation method based on hybrid feature fusion according to claim 1, characterized in that: The process involves collecting raw facial data and obtaining public image data. The specific process of preprocessing and labeling the raw facial data is as follows: Raw facial data was collected, including: 3D facial data, batch size, identification number data, and expression type identifier; public image data was also acquired, including the ImageNet dataset. K-fold cross-validation was performed, and a partitioning strategy was determined based on the cross-validation results, dividing each set of 3D facial data into training, testing, and validation sets. Noise reduction, resampling, and coordinate correction were performed on the 3D facial data through 3D structure calibration and data format normalization. The 3D point cloud data was mapped to a unified standard coordinate system using centroid localization, translation calibration, and scale unification algorithms. Data consistency was checked for each array by verifying the association between the identification number and expression type label. Missing point clouds, duplicate labels, and arrays with abnormal acquisition were removed through formatting and data integrity checks. House-Brackmann rating was used to rate and label the original facial data as supervisory labels, and the rating level was limited to the range of a to b using multi-classification and mapping algorithms. Expression type labels were used to support the feature learning process of expression branches. A 3D facial database was created, and the 3D facial data was stored in the database.

3. The three-dimensional multi-state comprehensive level evaluation method based on hybrid feature fusion according to claim 1, characterized in that: The specific process of constructing 3D point cloud matrix data and 2D representation by performing point cloud reconstruction and dense / sparse structure generation on 3D facial data, combined with viewpoint projection and 2D image normalization, is as follows: The mesh vertices, faces, and topology of the 3D facial data are analyzed. Vertex coordinates are read, and all vertices are extracted into point cloud coordinate sets according to their spatial positions to obtain 3D point cloud data. A 3D facial model is created by inputting the 3D point cloud data. A combination of homogenization filtering based on mesh vertex resampling and farthest point sampling is used to represent the 3D facial model as dense and sparse point clouds. The number of point clouds in each array is uniformly sampled to N points and stored in an N×3 matrix. The centroid of the point cloud is calculated, and the entire matrix is ​​translated so that the centroid is located at the origin. The centroid is used as a reference. The maximum offset of the point cloud along the three coordinate axes is calculated. The reciprocal of the global maximum offset is taken as the scaling factor, and all points are linearly scaled to normalize the overall scale to the interior of a unit sphere, resulting in a 3D point cloud matrix data. The 3D facial model is projected using multi-view projection technology to generate n 2D RGB images. The 2D RGB images are adjusted to a uniform size and normalized, including pixel value normalization and channel normalization. Statistical values ​​from the ImageNet dataset are obtained as the normalized mean and variance to unify the pixel distribution of the normalized images.

4. The three-dimensional multi-state comprehensive level evaluation method based on hybrid feature fusion according to claim 1, characterized in that: The specific process of performing enhancement, feature encoding, projection mapping, and modal contrast constraints on the three-dimensional point cloud matrix data and two-dimensional representation through the dual-branch structure is as follows: In the training set, the dual-branch structure includes a point cloud branch and an image branch. In the point cloud branch, data augmentation operations are performed on the 3D point cloud matrix data, and two different augmented views are generated for the same array to obtain two sets of augmented point cloud data. A graph convolutional neural network is used as the backbone network. The augmented point cloud data is input, and multi-scale local structural features are extracted through graph convolution operations based on k-nearest neighbor graphs. The global feature vector of the point cloud is obtained through feature concatenation and global pooling. The global feature vector of the point cloud is reduced in dimensionality through a fully connected layer to obtain the projection mapping vector; the similarity between different augmented views of the same array is calculated as the positive array similarity, and the similarity between the array and other arrays in the batch under their respective augmented views is calculated as the negative array similarity; Then, the positive array similarity and negative array similarity are substituted into the normalized ratio in the form of softmax, and the single array contrast loss is obtained by taking the negative logarithm of the positive array similarity; the single array contrast loss of all arrays is averaged on the batch dimension to form the intra-modal instance resolution loss; In the image branch, data augmentation is performed on a two-dimensional RGB image to obtain augmented image data; Using ResNet50 network as the backbone, the system takes enhanced image data as input, extracts semantic features of the image through multi-layer convolution, pooling and residual blocks, and outputs a global feature vector of the image. The global feature vector of the image is reduced in dimensionality through a fully connected layer to obtain the image projection vector. The two projection mapping vectors are averaged element-wise in their corresponding dimensions to obtain the projection prototype vector in the point cloud modality. The construction of the projection prototype vector is adjusted according to the distribution characteristics of different categories using a category-aware projection prototype construction method. Based on the projection prototype vector and the image projection vector, the similarity between the projection prototype vector and the image projection vector in the same array is calculated as the cross-modal positive array similarity. The similarity between the projection prototype vector and the image projection vector in other arrays in the batch is calculated as the cross-modal negative array similarity. Finally, the positive and negative array similarities are combined into the normalized ratio in the softmax form under temperature scaling. The cross-modal contrast loss is obtained by taking the negative logarithm of the positive array similarity. The cross-modal instance discrimination loss is obtained by summing and averaging the cross-modal contrast losses of all arrays in the batch. A dynamic training strategy based on hard array mining is adopted: the training priority is adjusted by comparing the distribution of the number of groups in the learning loss.

5. The three-dimensional multi-state comprehensive level evaluation method based on hybrid feature fusion according to claim 1, characterized in that: The specific process of combining classifier supervision and loss joint optimization is as follows: The global feature vector of the point cloud is used as input to the classifier. It is then passed through two fully connected layers and a Dropout layer to output the original s-dimensional classification score. The original classification score is mapped to the predicted probability distribution of the HB level of the corresponding expression using the softmax function to obtain the classifier's predicted level. The classifier's predicted level and the supervision label are input, and the negative logarithm of the predicted probability of the correct category is taken and averaged within the batch range to calculate the classification cross-entropy loss. The classification cross-entropy loss is used as a supervision signal to optimize the classification result. The sum of the intra-modal instance resolution loss, cross-modal instance resolution loss and classification cross-entropy loss is calculated to obtain the total loss function. The parameters of the point cloud branch, image branch and classifier are jointly updated through backpropagation.

6. The three-dimensional multi-state comprehensive level evaluation method based on hybrid feature fusion according to claim 1, characterized in that: The specific process of extracting deep structural features from the original facial data through expression branches and focusing expression attention is as follows: The system takes a 3D point cloud matrix as input and performs data augmentation on four types of facial expression point clouds to output augmented facial expression point cloud data. Each facial expression branch shares the same feature extraction backbone network built on a graph convolutional neural network. Each facial expression branch independently processes the augmented facial expression point cloud data and outputs a branch global feature vector through graph convolution and multi-scale feature aggregation. Each facial expression branch is connected to an facial expression attention layer, which takes the branch global feature vector as input, assigns different channel weights to the facial expression feature channels, focuses on the feature components with the most discriminative power for facial paralysis rating, and outputs a weighted facial expression feature vector. Each facial expression branch is connected to a classifier after the facial expression attention layer, which takes the weighted facial expression feature vector as input and outputs an s-dimensional original classification score. The score is then mapped to the probability distribution of the corresponding HB level through a softmax function to obtain the branch prediction result. The branch cross-entropy loss between the prediction results and the supervision label is calculated.

7. The three-dimensional multi-state comprehensive level evaluation method based on hybrid feature fusion according to claim 1, characterized in that: The channel control mechanism of the fusion layer, and the specific process by which comprehensive depth features are obtained, are as follows: The system inputs 3D point cloud matrix data and performs data augmentation on four types of facial expression point clouds to output augmented facial expression point cloud data. Each facial expression branch shares the same feature extraction backbone network built on a graph convolutional neural network. Each facial expression branch independently processes the augmented facial expression point cloud data and outputs a branch global feature vector through graph convolution and multi-scale feature aggregation. Each facial expression branch is connected to an facial expression attention layer, which inputs the branch global feature vector, assigns different channel weights to the facial expression feature channels, focuses on the feature components with the most discriminative power for facial paralysis rating, and outputs a weighted facial expression feature vector. Each facial expression branch is connected to a classifier after the facial expression attention layer, which inputs the weighted facial expression feature vector and outputs an s-dimensional original classification score. This score is then mapped to the probability distribution of the corresponding HB level through a softmax function to obtain the branch prediction result. For each branch prediction result, the branch cross-entropy loss between the two is calculated based on the supervision label. The weighted facial expression feature vectors are concatenated along the feature dimension to form a combined feature representation. Based on a channel attention mechanism, channel-weighted learning is performed on the combined features to learn the relative importance of the four facial expression features in different channels, and the intermediate representation after channel weighting is output. The combined features are then superimposed and fused according to the channel weights to obtain a comprehensive feature representation. Input... The comprehensive features are processed through fully connected layers and Dropout layers to output s-dimensional original classification scores, which are then mapped to the probability distribution of the final HB level through softmax to obtain the comprehensive prediction results. For each comprehensive prediction result, the label cross-entropy loss between the two is calculated based on the supervised labels. The output is supervised using the cross-entropy loss of the fusion layer.

8. The three-dimensional multi-state comprehensive level evaluation method based on hybrid feature fusion according to claim 1, characterized in that: The specific process of joint optimization through hierarchical supervision is as follows: The branch cross-entropy loss and the fusion layer cross-entropy loss are weighted and summed using the branch loss weight and the fusion layer loss weight to obtain the total fusion loss. The total fusion loss is then used to jointly optimize the parameters of the shared graph convolutional neural network backbone, expression branch attention layer and classifier, channel attention unit in the fusion layer, and fusion layer classifier through the backpropagation algorithm.

9. The three-dimensional multi-state comprehensive level evaluation method based on hybrid feature fusion according to claim 1, characterized in that: The specific process of executing the inference flow of cross-modal path and expression fusion path during the testing phase, performing rule-based integration and feature layer fusion decision on the prediction results, and structurally archiving the output level is as follows: During the testing phase, two frameworks were developed: a cross-modal contrastive learning framework and an expression hybrid rating framework. The cross-modal contrastive learning framework executes the inference process of cross-modal paths and expression fusion paths: it only calls the point cloud branch and classifier to output the classifier's predicted level; it integrates the dimensional features of the classifier's predicted levels for the four expressions and obtains the final patient-level facial paralysis level through a precise level mapping algorithm; the expression hybrid rating framework calls the expression branch and fusion layer for rule-based integration to obtain a comprehensive prediction result, outputs the final patient-level facial paralysis level, and simultaneously creates a prediction result database to record all prediction results for use in experiments and algorithm comparisons.

10. A three-dimensional multi-state comprehensive level evaluation system based on hybrid feature fusion, applying the three-dimensional multi-state comprehensive level evaluation method based on hybrid feature fusion as described in any one of claims 1-9, comprising: The acquisition, preprocessing, and annotation module is used to acquire raw facial data and obtain public image data. Preprocess and label the raw facial data; The 3D facial modeling and multi-view processing module is used to reconstruct point clouds and generate dense and sparse structures from 3D facial data. It combines view projection and 2D image normalization processes to construct 3D point cloud matrix data and 2D representation. The point cloud image hybrid feature contrast learning module is used to perform enhancement, feature encoding, projection mapping and modal contrast constraints on 3D point cloud matrix data and 2D representation through a dual-branch structure, and combines classifier supervision and loss joint optimization. The structural feature parsing and fusion decision module is used to extract deep structural features from the original facial data through expression branches, expression attention focusing and channel regulation mechanism of the fusion layer, obtain comprehensive deep features, and perform joint optimization through hierarchical supervision; The rating evaluation and optimization module is used to execute the inference process of cross-modal path and expression fusion path during the testing phase, perform rule-based integration and feature layer fusion decision on the prediction results, and perform structured archiving of the output rating.

Images