A dual-modal facial paralysis recognition optimization method and system

Abstract

This invention discloses a dual-modal facial paralysis recognition optimization method and system. The method includes acquiring facial images and electromyography (EMG) signals to obtain first facial features and first EMG features; inputting the facial images into a first model to output facial key points and muscle texture features, and acquiring facial expression trajectory based on time series to output dynamic symmetry deviation; extracting second EMG features from the facial EMG signals; inputting the output features of the first model and the EMG signal features into a second model to obtain a total loss function and output a classification category; inputting the acquired features and the total loss function into the second model to obtain a classification probability, which is used to update the output of the second model. This invention uses an optimized convolutional neural network model to extract facial key points, combines static deviations with dynamic changes to solve the problem of fusing dynamic and static features, eliminates individual differences through DTW (Dynamic Transformation Wave) and achieves accurate recognition of the degree of facial paralysis through dual-modal data fusion based on attention mechanisms.

Description

Technical Field

[0001] This invention relates to the field of bimodal recognition technology, and in particular to an optimized method and system for bimodal facial paralysis recognition. Background Technology

[0002] Facial paralysis is a common neurological disorder characterized by partial or complete loss of facial muscle function, leading to symptoms such as facial asymmetry and expression disorders. The traditional House-Brackmmann grading system for facial nerve paralysis relies primarily on the physician's clinical experience and visual observation, which has inherent subjectivity and limitations. This has prompted neurosurgeons to seek facial paralysis identification tools utilizing computer vision and artificial intelligence technologies to assist in clinical diagnosis and treatment evaluation.

[0003] However, existing facial paralysis recognition methods have the following technical shortcomings and drawbacks: (1) They cannot capture dynamic symmetry deviations during facial expression changes, such as the degree of upward movement of the corners of the mouth when smiling, and lack the fusion of dynamic and static features; (2) They are not good at recognizing mild facial paralysis, and the static feature analysis is singular, such as insufficient muscle movement but not manifested as significant static asymmetry; (3) The analysis of dynamic expressions depends on the patient's action coordination, and the standardization of facial expression movements is insufficient, but the range and speed of movement of different patients may vary greatly, affecting the consistency of results; (4) Existing methods usually rely only on visual data and do not combine other biometrics for comprehensive evaluation, such as electromyography (EMG) signals, and lack the fusion of bimodal data. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a dual-modal facial paralysis recognition optimization method and system to solve the problems of current methods being unable to capture dynamic symmetry deviations in the process of facial expression changes, having limited static feature analysis, being inaccurate in facial paralysis expression recognition, and lacking comprehensive judgment based on multiple data.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a dual-modal facial paralysis recognition optimization method, comprising: acquiring a facial image and facial electromyography signals, performing preprocessing, and acquiring a first facial feature and a first electromyography feature;

[0008] The preprocessed facial image is input into the first model for feature extraction, outputting facial key points and muscle texture features, and obtaining the facial movement trajectory based on the time series. The movement trajectory is compared with the standard template to output the dynamic symmetry deviation.

[0009] Extracting the second electromyographic features from the preprocessed facial electromyographic signals;

[0010] The output features of the first model and the electromyographic signal features are input into the second model for feature fusion to obtain the total loss function and output the classification category.

[0011] The acquired features and total loss function are input into the second model to obtain classification probabilities, which are then used to update the output of the second model.

[0012] As a preferred embodiment of the dual-modal facial paralysis recognition optimization method described in this invention, obtaining the first facial feature includes:

[0013] Define the left and right coordinates of a pair of mirror images of the corners of the mouth on the face, and calculate the Euclidean distance between the left and right coordinates;

[0014] Define the vector angles of the left and right coordinates about the axis of symmetry, respectively, to calculate the symmetry angle deviation;

[0015] Define an arc formed by fitting discrete coordinates within a preset range of the corner of the mouth or the brow bone, and calculate the curvature of the arc;

[0016] Define the initial key points of the face when smiling or closing the eyes, as well as the key points of facial expression movements, and calculate the amplitude of expression changes to obtain the left-right change rate of symmetry;

[0017] The facial symmetry features are calculated by using the curvature of the arc and the rate of change of symmetry to the left and right as constraints, and based on the Euclidean distance between the left and right coordinates and the symmetry angle deviation.

[0018] As a preferred embodiment of the dual-modal facial paralysis recognition optimization method of the present invention, the preprocessed facial image is input into the first model for feature extraction, and the facial key points and muscle texture features are output, including: inputting the facial image into a CNN model containing an LSTM architecture and a DTW algorithm;

[0019] The model outputs the predicted coordinates of n facial key points through multiple convolutional, pooling, and fully connected layers.

[0020] Contrast and homogeneity features in muscle texture are extracted based on the gray-level co-occurrence matrix.

[0021] As a preferred embodiment of the dual-modal facial paralysis recognition optimization method described in this invention, the following is included: obtaining facial expression movement trajectories based on time series, comparing the movement trajectories with standard templates, and outputting dynamic symmetry deviations, including: inputting extracted facial key points and muscle texture features as time series data into the LSTM architecture in the first model to obtain facial expression movement trajectories;

[0022] The acquired facial expression trajectory is compared with a standard facial expression template, and time and amplitude alignment is performed using the DTW algorithm, specifically including:

[0023] Let the set of m standard templates be U = {U1, U2, ..., U...} m};

[0024] Let X = {X1, X2, ..., Xn} be the set of facial expression trajectories of n patients. n};

[0025] Let the goal of DTW be the optimal matching path P = {(i1, j1), (i2, j2), ..., (i k j k )};

[0026] Aligning U and X in time and magnitude yields the DTW distance metric, represented as:

[0027]

[0028] Where i∈m, j∈n;

[0029] Calculate the dynamic symmetry deviation between the output patient and the standard template, including:

[0030] Combining the distance metric of DTW, the similarity between the synchronicity of time series and the changes in feature amplitude is calculated, and expressed as:

[0031]

[0032] Where T represents the total number of facial paralysis patient X and standard template U at time step t;

[0033] If the similarity exceeds the set threshold, it is considered that there is a dynamic symmetry deviation of facial paralysis, and the similarity is output as the dynamic symmetry deviation value.

[0034] As a preferred embodiment of the dual-modal facial paralysis recognition optimization method of the present invention, the extraction of the second electromyographic feature of the preprocessed facial electromyographic signal includes:

[0035] The preprocessed facial electromyography signals were processed by short-time Fourier transform to obtain the time-frequency characteristic energy spectrum of facial paralysis.

[0036] The dynamic spectral difference is calculated based on the time-frequency characteristic energy spectrum of facial paralysis and used as the second electromyographic feature.

[0037] As a preferred embodiment of the dual-modal facial paralysis recognition optimization method described in this invention, the method involves: inputting the output features of the first model and the electromyographic signal features into the second model, performing feature fusion to obtain a classification loss function and outputting the classification category, specifically including:

[0038] Define the output features of the first model as visual features, represented as:

[0039] Z = {Z1, Z, ..., Z} i} i = 1, 2, ..., n

[0040] Among them, Z i This represents the i-th vector output by the first model, which includes facial key points, muscle texture features, and dynamic symmetry deviation.

[0041] Electromyographic characteristics are defined as follows:

[0042] E = {E1, E2, ..., E} i} i = 1, 2, ..., n

[0043] Among them, E i This represents the dynamic spectral difference features extracted using short-time Fourier transform.

[0044] Visual features and electromyographic features are input into an attention-based Transformer model. The model is processed through two layers of self-attention layers with different features and a feedforward network. The features are then weighted and fused, and a classification label is output.

[0045] The total loss function is obtained based on the similarity measurement function and classification loss function in the second model.

[0046] As a preferred embodiment of the dual-modal facial paralysis recognition optimization method described in this invention, the method involves: inputting the acquired features and the total loss function into a second model to obtain classification probabilities, which are then used to update the output of the second model. This includes: inputting the acquired features and the total loss function into the second model for fusion, as shown below:

[0047] F(t)=[S, RMS, ΔD, ΔE(f), L]

[0048] Where S represents the first facial feature, RMS represents the first electromyographic feature, ΔD represents the dynamic symmetry deviation, ΔE(f) represents the dynamic spectral difference, and L represents the total loss function.

[0049] The classification probability is obtained through iteration of the second model and used to update the output classification of the second model, as follows:

[0050]

[0051] Secondly, the present invention provides a dual-modal facial paralysis recognition optimization system, comprising:

[0052] The acquisition module is used to acquire facial images and facial electromyography signals, perform preprocessing, and acquire first facial features and first electromyography features;

[0053] The first feature extraction module is used to input the preprocessed facial image into the first model, extract features, output facial key points and muscle texture features, obtain facial expression movement trajectory based on time series, compare the movement trajectory with the standard template, and output dynamic symmetry deviation.

[0054] The second feature extraction module is used to extract the second electromyographic features of the preprocessed facial electromyographic signals.

[0055] The feature fusion module is used to input the output features of the first model and the electromyographic signal features into the second model, perform feature fusion, obtain the total loss function, and output the classification category.

[0056] The update module is used to input the acquired features and the total loss function into the second model to obtain the classification probability, which is then used to update the output of the second model.

[0057] Thirdly, the present invention provides an electronic device, comprising:

[0058] Memory and processor;

[0059] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the dual-modal facial paralysis recognition optimization method are implemented.

[0060] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the dual-modal facial paralysis recognition optimization method.

[0061] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention combines Euclidean distance and angular difference symmetry indices, employs an optimized convolutional neural network model to extract facial key points, and captures dynamic patterns in the process of facial expression changes through a time-series LSTM model. It combines static key point deviations with dynamic change trends to construct hybrid features, thus solving the problem of fusing dynamic and static features. A muscle texture module is introduced into the CNN model to extract more subtle changes from facial images. Comparative learning is performed using gray-level co-occurrence matrix texture features, thereby enhancing sensitivity to subtle changes and improving the ability to recognize mild facial paralysis. Dynamic temporal warping (DTW) is used to align the time and amplitude of facial expression trajectories, regularizing the movement features, eliminating individual differences, and solving the problem of movement standardization. Furthermore, a fusion model based on an attention mechanism processes heterogeneous data, constructing feature matrices of different modalities, thereby achieving accurate identification of the degree of facial paralysis through dual-modal data fusion. Attached Figure Description

[0062] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a schematic diagram of the overall process of the dual-modal facial paralysis recognition optimization method according to an embodiment of the present invention;

[0064] Figure 2 This is a schematic diagram of the overall framework structure of the dual-modal facial paralysis recognition optimization method according to an embodiment of the present invention;

[0065] Figure 3 This is a dynamic wavelet transform spectrum of electromyographic signals in a dual-modal facial paralysis recognition optimization method according to an embodiment of the present invention;

[0066] Figure 4 This is a training effect diagram of the prediction loss function in the dual-modal facial paralysis recognition optimization method according to an embodiment of the present invention. Detailed Implementation

[0067] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0068] Example 1

[0069] Reference Figure 1-Figure 2 As an embodiment of the present invention, a dual-modal facial paralysis recognition optimization method is provided, comprising:

[0070] S100: Acquire facial images and facial electromyography signals, perform preprocessing, and acquire first facial features and first electromyography features;

[0071] In this embodiment of the application, the preprocessing in step S100 includes: denoising and annotating the key points of the facial image;

[0072] Specifically, denoising is achieved by applying a Gaussian filter to the facial image I, with a mean of 0 and a variance of σ. Remove noise components.

[0073] Specifically, the annotation includes: marking the location of facial landmarks P. X ∈{P eye P nose, ..., P mouth};

[0074] Where X represents parts such as eyes, tip of nose, and corners of mouth;

[0075] Let each key point form a two-dimensional vector, represented as:

[0076] P X ={P1, P2, ..., P i} i = 1, 2, ..., n (1)

[0077] Among them, P i =(x i y i ) represents the planar coordinates of any two-dimensional key point of the corresponding facial region.

[0078] Based on the reference center of the nose tip and the reference standard scale of both eyes, the two-dimensional vector coordinates of each key point are normalized and can be represented as follows:

[0079]

[0080] In this embodiment of the application, obtaining the first facial feature in step S100 includes the following steps A1-A5:

[0081] A1: Define the left and right coordinates of a pair of mirrored points at the corners of the mouth on the face as follows:

[0082] P L，m =(x L,m y L,m )P R，m =(x R,m y R,m ), ;

[0083] And calculate the Euclidean distance between the left and right coordinates, expressed as:

[0084]

[0085] A2: Define the vector angles of the left and right coordinates about the axis of symmetry as θ. L ,θ R , used to calculate the symmetry angle deviation, is expressed as:

[0086] D θ =||θ L -θ R || (4)

[0087] Where ||·|| represents the Euclidean norm, and θ L With θ R All (c x c y) represents the coordinates of the reference center point c of the nose tip.

[0088] A3: Define the arc s formed by fitting the discrete coordinates within a preset range of the corner of the mouth or brow bone, calculate the curvature of the arc, and express it as:

[0089]

[0090] Where x′(s) and y′(s) are the first derivatives of arc s in the x and y directions, respectively, and x″(s) and y″(s) are the second derivatives of arc s in the x and y directions, respectively.

[0091] A4: Define the initial facial key points when smiling or closing your eyes. and key points of facial expressions and movements Calculate the amplitude of facial expression change AP to obtain the left-right symmetry rate of change υ, expressed as:

[0092]

[0093] v=|ΔP L -ΔP R | (7)

[0094] Where |·| represents the absolute value, ΔP L ΔP R These represent the range of change in facial expressions, left and right.

[0095] A5: Using the curvature of the arc and the rate of change of symmetry as constraints, and based on the Euclidean distance between the left and right coordinates and the symmetry angle deviation, the facial symmetry features are calculated and expressed as:

[0096] S=αD m +βD θ

[0097]

[0098] Where α and β are the weighting deviation coefficients of Euclidean distance and angle, respectively, and ε1 and ε2 represent the thresholds for curvature and expression change rate in clinical judgment of facial paralysis, respectively.

[0099] In this embodiment of the application, the preprocessing of electromyographic data in step S100 includes:

[0100] The facial muscle EMG signal acquired by the wearable sensor is x(t), and the signal obtained after passing through the bandpass filter is:

[0101]

[0102] Where h(·) represents a bandpass filter.

[0103] To further extract EMG signal features, x f The (t) signal can be decomposed into different frequency bands, and noise is removed using wavelet coefficients:

[0104] x d (t)=∑ k ∑ j ω j，k ·ψ j，k (t) (10)

[0105] Where j and k represent the number of signal frequency components and the number of basis functions, respectively, ω j，k ψ represents the corresponding wavelet coefficients. j，k (t) represents the corresponding wavelet basis function.

[0106] In this embodiment of the application, the first electromyographic feature in step S100 is specifically:

[0107] The RMS representation is obtained by extracting signal features from the root mean square (RMS) data.

[0108]

[0109] Where T represents the upper limit of time integration.

[0110] S200: Input the preprocessed facial image into the first model, extract features, output facial key points and muscle texture features, obtain the facial expression trajectory based on the time series, compare the trajectory with the standard template, and output the dynamic symmetry deviation.

[0111] In this embodiment of the application, step S200 involves inputting the preprocessed facial image into the first model for feature extraction and outputting facial key points and muscle texture features, including: inputting the facial image into a CNN model containing an LSTM architecture and a DTW algorithm;

[0112] After passing through multiple convolutional, pooling, and fully connected layers, the model outputs the predicted coordinates of n facial key points, which can be represented as:

[0113]

[0114] Contrast and homogeneity features are extracted from muscle texture features based on the gray-level co-occurrence matrix (GLCM). The GLCM calculation can be expressed as follows:

[0115]

[0116] In the formula, p(i, j, d, θ) represents the co-occurrence probability of gray levels i and j at distance d and angle θ, M and N are the total number of pixels in the horizontal and vertical directions of the image, δ(·) is the indicator function, and I(·) is the gray pixel value;

[0117] The muscle texture features extracted from contrast (CON) and homogeneity (HOMO) are represented as follows:

[0118] CON = ∑ i，j (ij) 2 p(i,j) (14)

[0119]

[0120] In this embodiment of the application, step S200 involves obtaining the facial expression trajectory based on the time series, comparing the trajectory with a standard template, and outputting the dynamic symmetry deviation, including the following steps B1-B3:

[0121] B1: The extracted facial key points and muscle texture features are input as temporal data into the LSTM architecture of the first model to obtain the facial expression trajectory;

[0122] Specifically, step B1 can be implemented by assuming that at time step t, It is the predicted value of the key point location, f t If the corresponding muscle texture features are used, then the input to the model is the facial feature vector.

[0123] Furthermore, in order to capture the dynamic patterns in facial expression changes, such as the changes in facial key points and muscle texture features when smiling or frowning, the extracted facial key points and muscle texture features are treated as temporal data and input into the LSTM model. The basic computational units are:

[0124] Input gate: i t =σ(W i ·[h t-1 X t ]+b i (16)

[0125] Forgotten Gate: f t =σ(W f ·[h t-1 X t ]+b f (17)

[0126] Output gate: o t =σ(W o ·[h t-1 X t ]+b o (18)

[0127] Update status: c t =f t ·c t-1 +i t ·tanh(Wc ·[h t-1 X t ]+b c (19)

[0128] Hidden state: h t =o t ·tanh(c t (20)

[0129] B2: The acquired facial expression trajectory is compared with a standard facial expression template, and time and amplitude alignment is performed using the DTW algorithm, specifically including:

[0130] Let the set of m standard templates be U = {U1, U2, ..., U...} m};

[0131] Let X = {X1, X2, ..., Xn} be the set of facial expression trajectories of n patients. n};

[0132] Let the goal of DTW be the optimal matching path P = {(i1, j1), (i2, j2), ..., (i k j k )};

[0133] Aligning U and X in time and magnitude yields the DTW distance metric, represented as:

[0134]

[0135] Where i∈m, j∈n;

[0136] B3: Calculate the dynamic symmetry deviation between the output patient and the standard template, including:

[0137] Combining the distance metric of DTW, the similarity between the synchronicity of time series and the changes in feature amplitude is calculated, and expressed as:

[0138]

[0139] Where T represents the total number of facial paralysis patient X and standard template U at time step t;

[0140] If the similarity ΔD exceeds the set threshold, it is considered that there is a dynamic symmetry deviation of facial paralysis, and the similarity is output as the dynamic symmetry deviation value.

[0141] For example, the threshold set for similarity ΔD can be, for instance, 0.1 or 0.05. If this threshold is exceeded, it is considered that there is a dynamic symmetry deviation in facial paralysis.

[0142] It should be noted that the S200 step, by using a CNN model with multiple convolutions and pooling, can effectively extract rich local and global features from preprocessed facial images, such as the location of facial key points. By inputting facial key points and muscle texture features as temporal data into the LSTM architecture, the model can understand the temporal dimension of expression changes and capture continuous facial movements (such as the development of a smile). Through the DTW algorithm, even when there are differences in speed or intensity, it can accurately compare two expression sequences, improving the flexibility and accuracy of matching. The above methods can help detect potential facial nerve damage at an early stage, which is of great significance for clinical diagnosis.

[0143] S300: Extracting the second electromyographic feature of the preprocessed facial electromyographic signal;

[0144] In this embodiment of the application, step S300 extracts the second electromyographic feature of the preprocessed facial electromyographic signal, including:

[0145] The preprocessed facial electromyography signals were processed by short-time Fourier transform to obtain the time-frequency characteristic energy spectrum of facial paralysis.

[0146] Specifically, it can be expressed as:

[0147] Let the original signal be x(t), the mathematical expression for STFT is:

[0148]

[0149] Where X(t, f) represents the local time spectrum of the original signal x(t) at time t and frequency f, ω(·) represents the window function, and e -j2πfτ Fourier basis functions representing frequency components.

[0150] Further, the time-frequency characteristic energy spectrum of facial paralysis was obtained:

[0151]

[0152] Where |·| represents absolute value.

[0153] Because facial paralysis patients may experience imbalances or slowed responses in the activity of certain muscles when smiling or performing other actions, it is necessary to calculate the dynamic spectral difference.

[0154] The dynamic spectral difference is calculated based on the time-frequency characteristic energy spectrum of facial paralysis and used as the second electromyographic feature;

[0155] Specifically, it can be expressed as:

[0156]

[0157] Among them, E n(f) represents the time-frequency characteristic energy spectrum of a normal healthy person, and f1 and f2 represent the lower limit and upper limit of the spectrum, respectively.

[0158] S400: Input the output features of the first model and the electromyographic signal features into the second model, perform feature fusion, obtain the total loss function, and output the classification category;

[0159] In this embodiment of the application, step S400 involves inputting the output features of the first model and the electromyographic signal features into the second model for feature fusion to obtain a classification loss function and output the classification category. Specifically, this includes:

[0160] Define the output features of the first model as visual features, represented as:

[0161] Z = {Z1, Z, ..., Z} i} i = 1, 2, ..., n (26)

[0162] Among them, Z i This represents the i-th vector output by the first model, which includes facial key points, muscle texture features, and dynamic symmetry deviation.

[0163] Electromyographic characteristics are defined as follows:

[0164] E = {E1, E2, ..., E} i} i = 1, 2, ..., n (27)

[0165] Among them, E i This represents the dynamic spectral difference features extracted using short-time Fourier transform.

[0166] Visual features and electromyographic features are input into an attention-based Transformer model. The model is processed through two layers of self-attention layers with different features and a feedforward network. The features are then weighted and fused, and a classification label is output.

[0167] Specifically, the self-attention mechanism (Multi-Head Self-Attention) of the two layers (visual features and EMG features) of the Transformer model can be set as follows:

[0168]

[0169] Where Q, K, and V are the query, key, and value matrices, respectively.

[0170] Configure the feed-forward neural network of the Transformer model as follows:

[0171] FFN(X)=max(0,XW1+b1)W2+b2 (29)

[0172] In each Transformer encoder layer, visual and electromyographic features are processed through a self-attention layer and a feedforward network to learn higher-level feature representations.

[0173] Assuming the feature representation output by the Transformer is H, the final classification result is:

[0174]

[0175] Among them, W c 、b c This represents the weights and biases of the classification layer. This indicates the predicted category label.

[0176] For example, the classification label can be represented as y∈{0,1,2,3}, where 0 represents normal, 1 represents mild facial paralysis, 2 represents moderate facial paralysis, and 3 represents severe facial paralysis.

[0177] The total loss function is obtained based on the similarity measurement function and classification loss function in the second model.

[0178] Specifically, through comparative learning models, the subtle differences between different facial paralysis expressions can be learned;

[0179] Define a similarity metric function to determine whether two feature representations belong to the same category, expressed as:

[0180]

[0181] Where y represents the unpredicted category label, D(x) i x j ) represents the eigenvector x i x j Euclidean distance, σ is a hyperparameter representing the minimum distance boundary, and max(·) represents taking the maximum value.

[0182] The classification loss function is:

[0183]

[0184] The total loss function is:

[0185] L = L classi +λL contra (33)

[0186] S500: Input the acquired features and total loss function into the second model to obtain the classification probability, which is used to update the output of the second model.

[0187] In this embodiment of the application, step S500, which inputs the acquired features and the total loss function into the second model to obtain the classification probability for updating the output of the second model, includes: inputting the acquired features and the total loss function into the second model for fusion, as shown below:

[0188] F(t)=[S, RMS, ΔD, ΔE(f), L]

[0189] Where S represents the first facial feature, RMS represents the first electromyographic feature, ΔD represents the dynamic symmetry deviation, ΔE(f) represents the dynamic spectral difference, and L represents the total loss function.

[0190] It should be noted that integrating information from different sources (such as facial image features, electromyographic signal features, dynamic symmetry bias, etc.) into a unified feature vector enables the model to understand the input data from a more comprehensive perspective, capture more subtle facial expression changes, and thus improve classification accuracy.

[0191] The classification probability is obtained through iteration of the second model and used to update the output classification of the second model, as follows:

[0192]

[0193] It should be noted that comprehensive information is used to optimize the total loss function, which is an indicator that measures the difference between the predicted result and the actual category. Ultimately, the model outputs the classification category based on the principle of minimizing the loss function, that is, determining the most likely expression type or other relevant labels.

[0194] Example 2

[0195] The above is an illustrative scheme of a bimodal facial paralysis recognition optimization method according to this embodiment. It should be noted that the technical solution of this bimodal facial paralysis recognition optimization system and the technical solution of the above-described bimodal facial paralysis recognition optimization method belong to the same concept. For details not described in detail in the technical solution of the bimodal facial paralysis recognition optimization system in this embodiment, please refer to the description of the technical solution of the above-described bimodal facial paralysis recognition optimization method.

[0196] This embodiment also provides a system for optimizing a dual-modal facial paralysis recognition method, including:

[0197] The acquisition module is used to acquire facial images and facial electromyography signals, perform preprocessing, and acquire first facial features and first electromyography features;

[0198] The first feature extraction module is used to input the preprocessed facial image into the first model, extract features, output facial key points and muscle texture features, obtain facial expression movement trajectory based on time series, compare the movement trajectory with the standard template, and output dynamic symmetry deviation.

[0199] The second feature extraction module is used to extract the second electromyographic features of the preprocessed facial electromyographic signals.

[0200] The feature fusion module is used to input the output features of the first model and the electromyographic signal features into the second model, perform feature fusion, obtain the total loss function, and output the classification category.

[0201] The update module is used to input the acquired features and the total loss function into the second model to obtain the classification probability, which is then used to update the output of the second model.

[0202] This embodiment also provides an electronic device suitable for dual-modal facial paralysis recognition optimization, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the dual-modal facial paralysis recognition optimization method proposed in the above embodiment.

[0203] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements the dual-modal facial paralysis recognition optimization method proposed in the above embodiments.

[0204] The storage medium proposed in this embodiment belongs to the same inventive concept as the dual-modal facial paralysis recognition optimization method proposed in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0205] Example 3

[0206] Reference Figures 3-4 Based on the previous embodiment, this embodiment provides an application case of the dual-modal facial paralysis recognition optimization method to illustrate the feasibility and beneficial effects of our solution.

[0207] like Figure 3 The image shown is a dynamic wavelet transform spectrum energy diagram of an electromyographic signal. According to... Figure 2 The dynamic spectral energy results were calculated to have the following characteristics: ① Root mean square (RMS) value: 0.0404; ② Average value of dynamic spectral difference: 0.1028; ③ Root mean square (RMS) value of healthy individuals: 0.0367; ④ Root mean square (RMS) value of facial paralysis patients: 0.0613; ⑤ Average value of dynamic spectral difference value of healthy individuals: 0.0527; ⑥ Average value of dynamic spectral difference value of facial paralysis patients: 0.2604.

[0208] Based on the above features, the second model is input to obtain... Figure 3 The training effect of the prediction loss function;

[0209] from Figure 4The training results of the loss function show that the optimization method designed in this invention achieves a lower final loss value, indicating that the model can accurately predict four types of people: normal, mild facial paralysis, moderate facial paralysis, and severe facial paralysis.

[0210] Based on the above description of the implementation methods, those skilled in the art will clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0211] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A dual-modal facial paralysis recognition optimization method, characterized in that, include: Acquire facial images and facial electromyography (EMG) signals, perform preprocessing, and obtain first facial features and first EMG features; The first electromyographic feature is obtained by processing facial electromyographic signals through a bandpass filter and combining wavelet denoising to obtain a denoised signal x. d (t), the root mean square (RMS) is calculated based on the denoised signal and used as the first electromyographic feature (RMS); The preprocessed facial image is input into the first model for feature extraction, outputting facial key points and muscle texture features, and obtaining the facial movement trajectory based on the time series. The movement trajectory is compared with the standard template to output the dynamic symmetry deviation. Among them, the facial expression trajectory is obtained based on the time series, and the trajectory is compared with the standard template to output the dynamic symmetry deviation. This includes: inputting the extracted facial key points and muscle texture features as time series data into the LSTM architecture in the first model to obtain the facial expression trajectory. The acquired facial expression trajectory is compared with a standard facial expression template, and time and amplitude alignment is performed using the DTW algorithm, specifically including: Let the set of m standard templates be U = {U1, U2, ..., U...} m }; Let X = {X1, X2, ..., Xn} be the set of facial expression trajectories of n patients. n }; Let the goal of DTW be the optimal matching path P = {(i1,j1),(i2,j2),…,(i k ,j k )}; Aligning U and X in time and magnitude yields the DTW distance metric, represented as: Where i∈m, j∈n; Calculate the dynamic symmetry deviation between the output patient and the standard template, including: Combining the distance metric of DTW, the similarity between the synchronicity of time series and the changes in feature amplitude is calculated, and expressed as: Where T represents the total number of facial paralysis patient X and standard template U at time step t; If the similarity exceeds the set threshold, it is considered that there is a dynamic symmetry deviation of facial paralysis, and the similarity is output as the dynamic symmetry deviation value. The second electromyographic features of the preprocessed facial electromyographic signals were extracted, including: The preprocessed facial electromyography signals were processed by short-time Fourier transform to obtain the time-frequency characteristic energy spectrum of facial paralysis. The dynamic spectral difference is calculated based on the time-frequency characteristic energy spectrum of facial paralysis and used as the second electromyographic feature; The output features of the first model and the electromyographic signal features are input into the second model for feature fusion to obtain the total loss function and output the classification category, specifically including: Define the output features of the first model as visual features, represented as: Z={Z1,Z,…,Z i }i=1,2,…,n Among them, Z i This represents the i-th vector output by the first model, which includes facial key points, muscle texture features, and dynamic symmetry deviation. Electromyographic characteristics are defined as follows: E={E1,E2,…,E i }i=1,2,…,n Among them, E i This represents the dynamic spectral difference features extracted using short-time Fourier transform. Visual features and electromyographic features are input into an attention-based Transformer model. The model is processed through two layers of self-attention layers with different features and a feedforward network. The features are then weighted and fused, and a classification label is output. The total loss function is obtained based on the similarity measurement function and classification loss function in the second model; The acquired features and total loss function are input into the second model to obtain classification probabilities, which are then used to update the output classification category of the second model and determine the most likely expression type.

2. The dual-modal facial paralysis recognition optimization method as described in claim 1, characterized in that, Obtain the first facial features, including: Define the left and right coordinates of a pair of mirror images of the corners of the mouth on the face, and calculate the Euclidean distance between the left and right coordinates; Define the vector angles of the left and right coordinates about the axis of symmetry, respectively, to calculate the symmetry angle deviation; Define an arc formed by fitting discrete coordinates within a preset range of the corner of the mouth or the brow bone, and calculate the curvature of the arc; Define the initial key points of the face when smiling or closing the eyes, as well as the key points of facial expression movements, and calculate the amplitude of expression changes to obtain the left-right change rate of symmetry; The facial symmetry features are calculated by using the curvature of the arc and the rate of change of symmetry to the left and right as constraints, and based on the Euclidean distance between the left and right coordinates and the symmetry angle deviation.

3. The dual-modal facial paralysis recognition optimization method as described in claim 2, characterized in that, The preprocessed facial image is input into the first model for feature extraction, and the facial key points and muscle texture features are output. This includes: inputting the facial image into a CNN model that includes an LSTM architecture and a DTW algorithm. The model outputs the predicted coordinates of n facial key points through multiple convolutional, pooling, and fully connected layers. Contrast and homogeneity features in muscle texture are extracted based on the gray-level co-occurrence matrix.

4. The dual-modal facial paralysis recognition optimization method as described in claim 3, characterized in that, The acquired features and the total loss function are input into the second model to obtain classification probabilities, which are then used to update the output of the second model. This includes fusing the acquired features and the total loss function into the second model, as shown below: F(t) = [S, RMS, ΔD, ΔE(f), L] Where S represents the first facial feature, RMS represents the first electromyographic feature, ΔD represents the dynamic symmetry deviation, ΔE(f) represents the dynamic spectral difference, and L represents the total loss function. The classification probability is obtained through iteration of the second model and used to update the output classification of the second model, as follows:

5. A system applied to the dual-modal facial paralysis recognition optimization method according to any one of claims 1-4, characterized in that, include: The acquisition module is used to acquire facial images and facial electromyography signals, perform preprocessing, and acquire first facial features and first electromyography features; The first feature extraction module is used to input the preprocessed facial image into the first model, extract features, output facial key points and muscle texture features, obtain facial expression movement trajectory based on time series, compare the movement trajectory with the standard template, and output dynamic symmetry deviation. The second feature extraction module is used to extract the second electromyographic features of the preprocessed facial electromyographic signals. The feature fusion module is used to input the output features of the first model and the electromyographic signal features into the second model, perform feature fusion, obtain the total loss function, and output the classification category. The update module is used to input the acquired features and the total loss function into the second model to obtain the classification probability, which is then used to update the output of the second model.

6. An electronic device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the dual-modal facial paralysis recognition optimization method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the bimodal facial paralysis recognition optimization method according to any one of claims 1 to 4.

Images