Multi-modal emotion recognition method based on zero-knowledge machine learning

Abstract

The application relates to a multi-modal emotion recognition method based on zero-knowledge machine learning, belonging to the cross field of artificial intelligence and information security. By extracting key features of local face images, voice signals and posture data multi-modal data through an attention module first, then transmitting the features to a server after homomorphic encryption, training a convolutional neural network (CNN) model in the encryption domain, and finally transmitting the results back to the local for decryption and emotion recognition, the method has the advantages that by using the homomorphic encryption technology, the privacy data of the user is always kept in an encrypted state during the entire data processing and transmission process, the fusion of multi-modal features not only improves the accuracy of emotion recognition, but also enhances the robustness of the system in complex environments, so that the emotion recognition result is more stable and reliable, and the method can be widely applied to intelligent monitoring systems, user emotion analysis, human-computer interaction, psychological health monitoring and other fields.

Description

TECHNICAL FIELD

[0001] The present application belongs to the cross field of artificial intelligence and information security, and specifically relates to a multi-modal emotion recognition method based on zero-knowledge machine learning (zkML). BACKGROUND

[0002] With the rapid development of artificial intelligence and big data technology, emotion recognition has been widely applied in intelligent security, smart home, health monitoring and other fields. However, traditional emotion recognition methods mostly rely on a large amount of plaintext data for training and reasoning, which poses a great risk in data privacy protection.

[0003] Currently, emotion recognition is mainly achieved through the following steps:

[0004] 1. Data collection: Obtain face images, voice signals and posture data through cameras, microphones or other sensors.

[0005] 2. Data preprocessing: Normalize, crop and other processes are performed on the collected images to extract the face region; noise reduction, feature extraction and other processes are performed on the voice signals; standardization is performed on the posture data.

[0006] 3. Feature extraction: Use deep learning models such as convolutional neural networks (CNN) to extract key features from face images, extract MFCC features from voice signals, and extract motion features from posture data.

[0007] 4. Emotion classification: Input the extracted multi-modal features into the classifier to output the emotion recognition result.

[0008] Although these methods have made certain progress in accuracy, they have the following shortcomings in data privacy protection:

[0009] 1. Data privacy leakage: Unencrypted data is easily stolen or tampered with during data collection and transmission, leading to user privacy leakage.

[0010] 2. Data storage security: A large amount of user data is stored on servers, posing a risk of illegal access and misuse of data.

[0011] 3. Data processing transparency: In existing emotion recognition systems, the data processing process is not transparent, and users cannot control and supervise how their data is used.

[0012] To solve the above problems, homomorphic encryption technology emerges as the times require. Unlike traditional encryption techniques, homomorphic encryption allows computations to be performed on encrypted data, enabling complex operations without revealing the plaintext data. The advent of homomorphic encryption provides an effective solution for data privacy protection, particularly suitable for scenarios requiring the processing of large amounts of sensitive data.

[0013] Despite the significant advantages of homomorphic encryption, it still faces the following challenges in practical applications:

[0014] 1. High computational overhead: Homomorphic encryption has high computational complexity, requiring higher system performance.

[0015] 2. Difficulty in model training: Machine learning model training in the encrypted domain requires overcoming various technical challenges, such as gradient calculation and model updating.

[0016] 3. Data transmission efficiency: Encrypted data transmission needs to ensure efficiency and security to prevent attacks during transmission.

[0017] With the rapid development of artificial intelligence and information security technology, protecting user privacy has become increasingly important in data processing and transmission. SUMMARY

[0018] The present invention proposes a multi-modal emotion recognition method based on zero-knowledge machine learning, which combines homomorphic encryption and attention mechanisms to achieve data privacy protection and efficient emotion recognition.

[0019] The technical solution adopted by the present invention includes the following steps:

[0020] Step one, data collection and preprocessing: the local device collects user's face images, voice signals and posture data, and improves data quality through preprocessing steps;

[0021] Step two, feature extraction: use attention mechanism to extract features from images, extract MFCC features from voice signals, and extract motion features from posture data, all these multi-modal features are fused into a comprehensive feature vector;

[0022] Step three, zero-knowledge proof: the local device generates a zero-knowledge proof to verify the authenticity of the feature data, and uses a homomorphic encryption algorithm to encrypt the comprehensive feature vector. This encryption method allows computations to be performed on encrypted data, ensuring data privacy during transmission and processing. The encrypted feature vector and zero-knowledge proof are transmitted to the remote server through a secure channel. During transmission, data remains encrypted to prevent any third party from stealing or tampering with the data.

[0023] Step four, server-side encryption calculation: after receiving the encrypted feature vector and zero-knowledge proof, the server first verifies the authenticity of the data, and then uses a convolutional neural network (CNN) for model training in the encrypted domain. Due to the characteristics of homomorphic encryption, the server can perform calculations on encrypted data without decryption, thereby protecting data privacy.

[0024] Step five, emotion recognition inference and result processing: after training is complete, the server transmits the encrypted inference result back to the local device through a secure channel. The local device decrypts the received encrypted result to obtain the final result of emotion recognition, and decodes the decrypted prediction value into the corresponding emotion label to display to the user.

[0025] The step one of the application comprises:

[0026] (1) Multi-modal emotion data collection:

[0027] The portrait photo of the user is taken by the portrait camera of the local device to obtain the facial feature data of the user. The posture data of the user is captured by a depth camera. The voice information of the user is collected by a microphone device. These data will be used to analyze the emotional state of the user, wherein the emotion can be labeled as seven basic emotions of happiness, anger, sadness, fear, surprise, disgust and calmness.

[0028] (2) Data preprocessing:

[0029] 1) Use the face detection algorithm MTCNN to detect the face region in the portrait image I, and obtain the boundary box of the face, including the left upper corner coordinates (x, y) and the width and height (w, h). According to the detected boundary box, the face region image I face is cropped.

[0030] I face =I[y:y+h,x:x+w]

[0031] Convert the RGB image to a grayscale image to reduce the amount of calculation, and the grayscale image directly reflects the brightness change, which is more helpful for emotion recognition. The conversion formula is:

[0032] I gray =0.2989·I R +0.5870·I G +0.1140·I B

[0033] Wherein, I R , I G and I B represent the pixel value matrix of the red, green and blue channels of the image respectively.

[0034] The pixel value is normalized to the range [0, 1] to eliminate the influence of brightness difference, and meanwhile, the normalized data can accelerate the convergence speed of the gradient descent algorithm and improve the training efficiency, and the normalization formula is:

[0035]

[0036] 2) using a pose estimation algorithm OpenPose to detect human key points, and obtaining 3D coordinates (x i ,y i ,z i ) of each key point;

[0037] In order to eliminate the difference between different user individuals, the key point coordinates are normalized:

[0038]

[0039] The max and min in the normalization process are the maximum and minimum values of all key point coordinates, respectively representing the maximum and minimum ranges of the user posture on each coordinate axis (x, y, z), and through these values, the coordinates of all key points are scaled to the range [0, 1], thereby eliminating the influence of different user body types, so that the posture data is compared and processed on the same scale;

[0040] The normalized key point coordinates (x i ′,y i ′,z i ′) constitute a data matrix P;

[0041] 3) the collected speech signal is pre-emphasized to improve the energy of the high frequency part and compensate for the loss of high frequency in the sound production process, then the emphasized speech signal is divided into multiple short time frames for windowing processing, and then the windowed frame signal is subjected to fast Fourier transform FFT to convert the time domain signal into a frequency domain signal;

[0042]

[0043] Wherein, Y(f) is the frequency spectrum of the frequency domain signal, y frame (t) is the windowed frame signal, t is the time index, f is the frequency index, and N is the length of the frame signal. Through Fourier transform, the energy distribution of the signal at different frequencies is obtained, which is helpful for extracting the frequency characteristics in the signal for subsequent speech processing tasks.

[0044] The step two comprises:

[0045] (1) using an attention mechanism combined with a simplified convolutional neural network CNN to extract key features of a face image;

[0046] The input image is a standardized gray image I norm Through a convolutional layer using a 3x3 convolution kernel, 32 filters, and a ReLU activation function, followed by a 2x2 max pooling layer, a pooled feature map P1 is obtained.

[0047] Then, the spatial attention and channel attention mechanisms are used to weight the feature map, making the model more focused on the most important parts and features of the image.

[0048] The spatial attention map is calculated for P1 through a 1x1 convolution and a softmax function:

[0049] Attention spatial = softmax(Conv1x1(P1))

[0050] After global average pooling is performed on each channel of P1, the channel attention weights are calculated through a fully connected layer:

[0051] Attention channel = σ(W1(GAP(P1)))

[0052] Where σ is the Sigmoid activation function, W1 is the weight of the fully connected layer, and GAP is the global average pooling.

[0053] The attention-weighted feature map F is obtained:

[0054] F attention = Attention spatial ·P1+Attention channel ·P1

[0055] The weighted feature map F attention is flattened to form a one-dimensional feature vector f face .

[0056] (2) Extract the MFCC features reflecting the emotional state from the speech signal;

[0057] The spectrum Y(f) obtained in step one is passed through a set of Mel filters to obtain the Mel frequency energy spectrum Mel(Y(f)). The Mel frequency energy spectrum is then taken logarithm, and then discrete cosine transform (DCT) is performed to obtain the MFCC features:

[0058] MFCC = DCT(log(Mel(Y(f))))

[0059] Where log represents the logarithm operation, and DCT represents the discrete cosine transform.

[0060] The obtained MFCC features are flattened to obtain a one-dimensional feature vector f audio .

[0061] (3) Use the self-attention mechanism to extract features from the posture data, so that the model can focus on important features and key points in the posture data;

[0062] Input the normalized posture data matrix P in step 1,

[0063] Calculate the query matrix Q, key matrix K and value matrix V of the pose data:

[0064] Q=W Q P, K = W K P, V = W V P

[0065] Among them, W Q 、W K and W V is the weight matrix, and then the attention weight is calculated:

[0066]

[0067] Among them, d k is the dimension of the key vector, and the softmax function is defined as follows:

[0068]

[0069] The attention output is then calculated:

[0070] Output=Attention·V

[0071] Flatten the attention output to get a one-dimensional feature vector f pose ;

[0072] (4) The extracted facial feature vector, posture feature vector and voice feature vector are spliced ​​together to form a comprehensive feature vector f combined :

[0073] f combined =[f face ;f pose ;f audio ].

[0074] The generation of the zero-knowledge proof in step 3 of the present invention includes:

[0075] (1) First, the zero-knowledge proof system is initialized on the local device. The public key PK and private key SK of the system are generated using the zero-knowledge proof system zk-SNARKs. These keys will be used to generate and verify the proof of the feature vector without leaking any original data.

[0076] (2) Using the public key PK and the comprehensive feature vector f combinedA zero-knowledge proof pi is generated, which ensures the authenticity of the generation process of the feature vector without leaking the original data;

[0077] (3) In order to protect data privacy during transmission, the homomorphic encryption technology is used to encrypt the proof pi and the comprehensive feature vector f combined , to generate the encrypted proof c π and the encrypted feature data c f ;

[0078] The encrypted proof c π and the encrypted feature data c f are transmitted to the server for verification;

[0079] (4) After the server receives the encrypted proof c π and the encrypted feature data c f , the proof c π is first decrypted to obtain the proof pi, and then the server uses the proof pi to verify the authenticity and integrity of the encrypted feature data c f without decrypting c f , according to the verification result, the server decides whether to accept the transmitted data, and carries out subsequent processing in the data encryption state.

[0080] The server-side encryption calculation in step four of the application comprises:

[0081] (1) Design a convolutional neural network CNN architecture suitable for homomorphic encryption:

[0082] Convolution layer 1: 1x3 convolution kernel, 64 filters, ReLU activation function approximated as piecewise linear function;

[0083] Average pooling layer 1: 1x2 pooling kernel;

[0084] Convolution layer 2: 1x3 convolution kernel, 128 filters, ReLU activation function approximated as piecewise linear function;

[0085] Average pooling layer 2: 1x2 pooling kernel;

[0086] Convolution layer 3: 1x3 convolution kernel, 256 filters, ReLU activation function approximated as piecewise linear function;

[0087] Average pooling layer 3: 1x2 pooling kernel;

[0088] Fully connected layer: input dimension is the length of the convolution output flattened, output dimension is the number of emotion categories;

[0089] (2) Select multimodal data collected in a real environment as the emotion dataset, split the dataset into training set, test set and validation set in a ratio of 8:1:1, train the multimodal network, select and save the weight file with the best training effect, which contains the model parameters learned during the training process and will be used to generate prediction results in the inference phase. Finally, save the CNN architecture and weight file to the server;

[0090] (3) Forward propagation algorithm

[0091] 1) Input encrypted feature vector c f , c f is the input of the forward propagation;

[0092] 2) Convolutional layer 1:

[0093] Encrypted convolution calculation: Encrypt the feature vector c f Convolution operation is performed through convolution layer 1:

[0094] Enc(Z1)=Enc(W1)*c f

[0095] Among them, Enc(W1) is the encrypted convolution kernel of convolution layer 1, * represents the convolution operation, and Enc(Z1) is the encrypted feature map after convolution;

[0096] Encrypted activation function calculation: Apply a piecewise linear function to the encrypted feature map Enc(Z1) to approximate the ReLU activation operation;

[0097] Enc(A1)=f piecewise (Enc(Z1))

[0098] Among them, f piecewise (·) is the piecewise linear approximation function of ReLU, and Enc(A1) is the encrypted output after activation;

[0099] 3) Average pooling layer 1:

[0100] Perform a one-dimensional average pooling operation on the activated encrypted feature map Enc(A1) to reduce the dimension of the feature map;

[0101]

[0102] Among them, Enc(G1) is the encrypted feature map after pooling, and the size of the pooling window is 2;

[0103] 4) Repeat convolution, activation and pooling operations:

[0104] For Enc(G1), after convolution layer 2 and average pooling layer 2:

[0105] Enc(Z2) = Enc(W2) * Enc(G1)

[0106] Enc(A2) = f piecewise (Enc(Z2))

[0107]

[0108] Subsequently, Enc(G2) is passed through convolution layer 3 and average pooling layer 3:

[0109] Enc(Z3) = Enc(W3) * Enc(G2)

[0110] Enc(A3) = f piecewise (Enc(Z3))

[0111]

[0112] to obtain the final encrypted feature map Enc(G3);

[0113] 5) Fully connected layer:

[0114] The encrypted feature map Enc(G3) after pooling is flattened to obtain an encrypted long vector, and then the encrypted output vector is calculated through the fully connected layer:

[0115] Enc(Y) = Enc(W fc )·Enc(G3)

[0116] Wherein, Enc(W fc ) is the encrypted weight matrix of the fully connected layer obtained by training the model, and Enc(Y) is the encrypted output vector of the fully connected layer.

[0117] In step five of the application:

[0118] The server transmits the encrypted emotion recognition result Enc(Y) back to the local device through a secure channel, and then decrypts the encrypted result using the private key SK of the local device to obtain the plaintext vector y;

[0119] Then apply the Softmax function on the decrypted plaintext vector Y to convert it to a probability distribution:

[0120]

[0121] Wherein, result i represents the probability value of the i-th emotion category, Y i is the i-th element of the plaintext vector Y, indicates the sum of the exponents of all 7 emotion categories;

[0122] The final result is a probability array containing 7 elements, representing the probability of each emotion category;

[0123] The local device saves a list of emotion labels:

[0124] labels = ["Happy", "Angry", "Sad", "Fear", "Surprise", "Disgust", "Calm"]

[0125] Find the emotion category with the highest probability in result, find the corresponding emotion label according to labels, and then show the predicted emotion label to the user.

[0126] In the multi-modal emotion recognition method based on zero-knowledge machine learning (zkML) proposed in the application, the local multi-modal data (including face image, speech signal and posture data) is first extracted through the attention module to extract key features, and then transmitted to the server after homomorphic encryption. The training of the convolutional neural network (CNN) model is carried out in the encryption domain, and finally the result is transmitted back to the local for decryption and emotion recognition, so that data privacy protection and efficient emotion recognition are organically combined. Compared with traditional methods, the following advantages are obtained:

[0127] (1) Comprehensive protection of data privacy: By using homomorphic encryption technology, the user's privacy data remains encrypted throughout the data processing and transmission process. This means that the user's sensitive information is protected on both the local device and the remote server, avoiding the risk of data leakage. At the same time, the introduction of zero-knowledge proof ensures that the remote server can verify the authenticity of the data without exposing the original data, fundamentally improving data security.

[0128] (2) High efficiency of multi-modal feature fusion: The invention integrates multi-modal information such as face image, speech signal and posture data, and uses attention mechanism and feature fusion technology to generate a comprehensive feature vector. This multi-modal feature fusion not only improves the accuracy of emotion recognition, but also enhances the robustness of the system in complex environments, making the emotion recognition result more stable and reliable.

[0129] (3) Seamless integration of encrypted computing: Thanks to the characteristics of homomorphic encryption, the server can directly calculate the encrypted data without decryption, so that it can still perform complex deep learning model training and reasoning while protecting data privacy. This method overcomes the problem of performance degradation in traditional encryption methods, achieving a balance between privacy protection and efficient computation.

[0130] (4) Wide application potential: Based on the multi-modal emotion recognition method of the present application, due to its significant improvement in data privacy protection and emotion recognition accuracy, it can be widely applied to intelligent monitoring systems, user sentiment analysis, human-computer interaction, mental health monitoring and other fields, providing safer and more efficient solutions. BRIEF DESCRIPTION OF DRAWINGS

[0131] Figure 1 is a flowchart of the emotion recognition system of the present application;

[0132] Figure 2 is a simplified convolutional neural network diagram combined with the attention mechanism of the present application;

[0133] Figure 3 is a convolutional neural network CNN diagram of the homomorphic encryption of the server side of the present application. DETAILED DESCRIPTION

[0134] Referring to Figure 1 , step one, data acquisition and preprocessing:

[0135] (1) Multi-modal emotion data acquisition:

[0136] Take a portrait photo of the user through the portrait camera of the local device to obtain the facial feature data of the user; use a depth camera to capture the posture data of the user; collect the voice information of the user through a microphone device, and these data will be used to analyze the emotional state of the user, wherein the emotion can be marked as seven basic emotions of happiness, anger, sadness, fear, surprise, disgust and calmness;

[0137] (2) Data preprocessing:

[0138] 1) Use the face detection algorithm MTCNN to detect the face region in the portrait image I, and obtain the boundary box of the face, including the left upper corner coordinates (x, y) and the width and height (w, h), and crop the face region image I face from the detected boundary box;

[0139] I face = I[y:y+h, x:x+w]

[0140] Convert the RGB image to a grayscale image to reduce the amount of calculation, and the grayscale image directly reflects the brightness change, which is more helpful for emotion recognition, and the conversion formula is:

[0141] I gray = 0.2989·I R + 0.5870·I G + 0.1140·I B

[0142] IR , I G and I B respectively represent the pixel value matrix of the red, green and blue channels of the image;

[0143] The pixel value is normalized to the range of [0, 1] to eliminate the influence of brightness difference, and at the same time, the normalized data can accelerate the convergence speed of the gradient descent algorithm and improve the training efficiency. The normalization formula is:

[0144]

[0145] 2) Use the pose estimation algorithm OpenPose to detect human key points and obtain the 3D coordinates (x i , y i , z i ) of each key point;

[0146] In order to eliminate the differences between different user individuals, the key point coordinates are normalized:

[0147]

[0148] The max and min in the normalization process are the maximum and minimum values of all key point coordinates, respectively representing the maximum and minimum ranges of the user's posture in each coordinate axis (x, y, z). By these values, the coordinates of all key points are scaled to the range of [0, 1], thereby eliminating the influence of different user body types and enabling the posture data to be compared and processed on the same scale;

[0149] The normalized key point coordinates (x′ i , y′ i , z′ i ) constitute a data matrix P;

[0150] 3) Pre-emphasis processing is performed on the collected speech signal to improve the energy of the high frequency part and compensate for the loss of high frequency during the sound production. Subsequently, the emphasized speech signal is divided into multiple short time frames for windowing processing. Then, the windowed frame signal is subjected to fast Fourier transform (FFT) to convert the time domain signal to the frequency domain signal;

[0151]

[0152] where Y(f) is the frequency spectrum of the frequency domain signal, y frame (t) is the windowed frame signal, t is the time index, f is the frequency index, and N is the length of the frame signal. Through Fourier transform, the energy distribution of the signal at different frequencies is obtained, which is helpful for extracting the frequency features in the signal for subsequent speech processing tasks;

[0153] Step two: feature extraction:

[0154] Since the subsequent zero-knowledge proof and homomorphic encryption cannot handle large data volume, it is necessary to extract features simply on the local device to reduce the data volume to meet the requirement of encrypted transmission. In this step, an attention module is used to extract features of the face and posture data, and MFCC features reflecting the emotional state are extracted from the speech signal.

[0155] (1) Use the attention mechanism combined with the simplified convolutional neural network (CNN) to extract the key features of the face image.

[0156] Referring to Figure 2 , the input image is a standardized gray image I norm , which is passed through a convolution layer using a 3x3 convolution kernel, 32 filters, and a ReLU activation function, and then a 2x2 max pooling layer is used to obtain the pooled feature map P1.

[0157] Then, spatial attention and channel attention mechanisms are used to weight the feature map, so that the model focuses more on the most important parts and features of the image.

[0158] The spatial attention map is calculated for P1 by 1x1 convolution and softmax function:

[0159] Attention spatial = softmax(Conv1x1(P1))

[0160] After global average pooling is performed on each channel of P1, the channel attention weight is calculated by a fully connected layer:

[0161] Attention channel = σ(W1(GAP(P1)))

[0162] Where σ is the Sigmoid activation function, W1 is the weight of the fully connected layer, and GAP is the global average pooling.

[0163] The attention-weighted feature map is obtained:

[0164] F attention = Attention face ·P1+Attention audio ·P1

[0165] The weighted feature map F Q is flattened to form a one-dimensional feature vector f K .

[0166] (2) Extract the MFCC features reflecting the emotional state from the speech signal.

[0167] The spectrum Y(f) obtained in step one is passed through a set of mel filters to obtain a mel frequency energy spectrum Mel(Y(f)), the mel frequency energy spectrum is taken logarithm, and then discrete cosine transform (DCT) is performed to obtain the MFCC feature:

[0168] MFCC=DCT(log(Mel(Y(f))))

[0169] Wherein, log represents the logarithm operation, and DCT represents the discrete cosine transform;

[0170] The obtained MFCC feature is flattened to obtain a one-dimensional feature vector f audio ;

[0171] (3) using self-attention mechanism to extract features of the posture data, so that the model can pay attention to important features and key points in the posture data;

[0172] The normalized posture data matrix P in step one is input,

[0173] The query matrix Q, the key matrix K and the value matrix V of the posture data are calculated:

[0174] Q=W Q P,K=W K P,V=W V P

[0175] Wherein, W Q , W K and W V are weight matrices, and then the attention weight is calculated:

[0176]

[0177] Wherein, d k is the dimension of the key vector, and the definition of the softmax function is as follows:

[0178]

[0179] Then the attention output is calculated:

[0180] Output=Attention·V

[0181] The attention output result is flattened to obtain a one-dimensional feature vector f pose ;

[0182] (4) the face feature vector, the posture feature vector and the speech feature vector extracted are spliced to form a comprehensive feature vector f combined :

[0183] f combined =[fface pose audio

[0184] Step three: zero-knowledge proof

[0185] In this step, a zero-knowledge proof will be generated to ensure the authenticity and integrity of the feature data, and homomorphic encryption will be used to protect data privacy;

[0186] (1) First, initialize the zero-knowledge proof system on the local device, use the zero-knowledge proof system zk-SNARKs to generate the public key PK and private key SK of the system, these keys will be used to generate and verify the proof of the feature vector without revealing any original data;

[0187] (2) Use the public key PK and the integrated feature vector f combined to generate a zero-knowledge proof π, which ensures the authenticity of the generation process of the feature vector without revealing the original data;

[0188] (3) In order to protect data privacy during transmission, use homomorphic encryption technology to encrypt the proof π and the integrated feature vector f combined , generate the encrypted proof c π and the encrypted feature data c f ;

[0189] The encrypted proof c π and the encrypted feature data c f are transmitted to the server for verification;

[0190] (4) After the server receives the encrypted proof c π and the encrypted feature data c f , first decrypt the proof c π to get the proof π, then the server uses π to verify the authenticity and integrity of the encrypted feature data c f without decrypting c f , according to the verification result, the server decides whether to accept the transmitted data, and carries out subsequent processing in the data encryption state;

[0191] Step four: server-side encrypted computation

[0192] (1) The CNN architecture running in the homomorphic encryption domain needs to consider the computational efficiency and the processing capacity of encrypted data, the following is a convolutional neural network CNN architecture design suitable for homomorphic encryption, as shown in Figure 3 :

[0193] Convolutional layer 1: 1x3 convolution kernel, 64 filters, ReLU activation function approximated as piecewise linear function; ​​​

[0194] Average pooling layer 1: 1x2 pooling kernel;

[0195] Convolution layer 2: 1x3 convolution kernel, 128 filters, ReLU activation function approximated by piecewise linear function;

[0196] Average pooling layer 2: 1x2 pooling kernel;

[0197] Convolution layer 3: 1x3 convolution kernel, 256 filters, ReLU activation function approximated by piecewise linear function;

[0198] Average pooling layer 3: 1x2 pooling kernel;

[0199] Fully connected layer: input dimension is the length of the flattened convolution output, output dimension is the number of emotion categories;

[0200] (2) Select multi-modal data collected in real environment as emotion dataset, divide the dataset into training set, test set and validation set according to the ratio of 8:1:1, train the multi-modal network, select and save the weight file with the best training effect, which contains the model parameters learned in the training process, and will be used to generate prediction results in the inference stage, finally save the CNN architecture and weight file to the server side;

[0201] (3) Forward propagation algorithm

[0202] 1) Input encrypted feature vector c f , c f is the input of forward propagation;

[0203] 2) Convolution layer 1:

[0204] Encrypted convolution calculation: the encrypted feature vector c f is convolved by convolution layer 1:

[0205] Enc(Z1) = Enc(W1) * c f

[0206] Where, Enc(W1) is the encrypted convolution kernel of convolution layer 1, * represents convolution operation, and Enc(Z1) is the encrypted feature map after convolution;

[0207] Encrypted activation function calculation: apply piecewise linear function to the encrypted feature map Enc(Z1) to approximate ReLU activation operation; piecewise linear function is relatively easy to implement in homomorphic encryption domain, because it can be calculated by simple linear operation;

[0208] Enc(A1) = f piecewise (Enc(Z1))

[0209] Where, fpiecewise (·) is a piecewise linear approximation function of ReLU, and Enc(A1) is the encrypted output after activation;

[0210] 3) Average pooling layer 1:

[0211] One-dimensional average pooling operation is performed on the activated encrypted feature map Enc(A1) to reduce the dimension of the feature map;

[0212]

[0213] Wherein, Enc(G1) is the encrypted feature map after pooling, and the size of the pooling window is 2;

[0214] 4) Repeat the convolution, activation and pooling operations:

[0215] Enc(G1) is processed through convolution layer 2 and average pooling layer 2:

[0216] Enc(Z2) = Enc(W2) * Enc(G1)

[0217] Enc(A2) = f piecewise (Enc(Z2))

[0218]

[0219] Then Enc(G2) is processed through convolution layer 3 and average pooling layer 3:

[0220] Enc(Z3) = Enc(W3) * Enc(G2)

[0221] Enc(A3) = f piecewise (Enc(Z3))

[0222]

[0223] Get the final encrypted feature map Enc(G3);

[0224] 5) Fully connected layer:

[0225] The encrypted feature map Enc(G3) after pooling is flattened to obtain an encrypted long vector, and then the encrypted output vector is calculated through the fully connected layer:

[0226] Enc(Y) = Enc(W fc )·Enc(G3)

[0227] Wherein, Enc(W fc ) is the encrypted weight matrix of the fully connected layer obtained by training the model, and Enc(Y) is the encrypted output vector of the fully connected layer;

[0228] Step five: emotion recognition inference and result processing

[0229] The server transmits the encrypted emotion recognition result Enc(Y) back to the local device through a secure channel, and then decrypts the encrypted result using the private key SK of the local device to obtain the plaintext vector y;

[0230] Then apply the Softmax function on the decrypted plaintext vector Y to convert it into a probability distribution:

[0231]

[0232] Where, result i represents the probability value of the ith emotion category, Y i is the ith element of the plaintext vector Y, indicates the sum of the exponents of all 7 emotion categories;

[0233] The final result is a probability array containing 7 elements, representing the probability of each emotion category;

[0234] The local device saves the emotion label list:

[0235] labels = ["Happy", "Angry", "Sad", "Fear", "Surprise", "Disgust", "Calm"]

[0236] Find the emotion category with the maximum probability in result, find the corresponding emotion label according to labels, and then show the predicted emotion label to the user.

Claims

1. A multimodal emotion recognition method based on zero-knowledge machine learning, characterized in that: The following steps are involved: Step 1: Data collection and preprocessing: The local device collects the user's facial image, voice signal, and posture data; Step 2: Feature extraction: Use the attention mechanism to extract features from the image, extract MFCC features from the speech signal, and extract motion features from the posture data. All these multimodal features are fused into a comprehensive feature vector. Step 3: Zero-knowledge proof: The local device generates a zero-knowledge proof to verify the authenticity of the feature data and encrypts the comprehensive feature vector using a homomorphic encryption algorithm. The encrypted feature vector and zero-knowledge proof are transmitted to the remote server via a secure channel. The zero-knowledge proof generation includes: (1) First, initialize the zero-knowledge proof system on the local device and use the zero-knowledge proof system zk-SNARKs to generate the system's public key PK and private key SK; (2) Using public key PK and comprehensive feature vector Generate zero-knowledge proof 𝜋; (3) Use homomorphic encryption technology to prove 𝜋 and the comprehensive feature vector Encrypt and generate encrypted proof and encrypted feature data ; The encrypted proof And encrypted feature data Transmit to the server for verification; (4) The server receives the encrypted proof and encrypted feature data After that, first we prove Decrypt and get the proof 𝜋, then the server uses 𝜋 to verify the encrypted feature data authenticity and integrity, without the need for Decryption is performed. Based on the verification result, the server decides whether to accept the transmitted data and performs subsequent processing in the encrypted state; Step 4: Server-side encryption calculation: After receiving the encrypted feature vector and zero-knowledge proof, the server first verifies the authenticity of the data and then uses the convolutional neural network (CNN) to train the model in the encrypted domain. Step 5: Emotion Recognition Inference and Result Processing: After training is complete, the server transmits the encrypted inference results back to the local device via a secure channel. The local device decrypts the received encrypted results to obtain the final emotion recognition results, and decodes the decrypted prediction values ​​into corresponding emotion labels to display to the user.

2. The multimodal emotion recognition method based on zero-knowledge machine learning according to claim 1, characterized in that: The step one comprises: (1) Multimodal emotion data collection: The local device's portrait camera is used to take a portrait photo of the user and obtain the user's facial feature data; the depth camera is used to capture the user's posture data; and the microphone device is used to collect the user's voice information. This data will be used to analyze the user's emotional state, where emotions can be labeled as seven basic emotions: happiness, anger, sadness, fear, surprise, disgust, and calmness. (2) Data preprocessing: 1) Using the face detection algorithm MTCNN, in the portrait image The face area is detected and the bounding box of the face is obtained, including the upper left corner coordinates (x, y) and width and height (w, h). The face area image is cropped according to the detected bounding box ; ； Convert RGB images to grayscale images to reduce the amount of calculation. Grayscale images directly reflect brightness changes and are more helpful for emotion recognition. The conversion formula is: ； in, 、 and matrices representing the pixel values ​​of the red, green, and blue channels of the image, respectively; Normalize the pixel values ​​to the range of [0, 1] to eliminate the influence of brightness differences. At the same time, normalized data can speed up the convergence of the gradient descent algorithm and improve training efficiency. The normalization formula is: ； 2) Use the pose estimation algorithm OpenPose to detect the key points of the human body and obtain the 3D coordinates of each key point ( , , ); In order to eliminate the differences between different users, the key point coordinates are normalized: ； ； ； The max and min in the normalization process are the maximum and minimum values ​​of all key point coordinates, respectively representing the user posture on each coordinate axis. The maximum and minimum ranges on the , by which the coordinates of all key points are scaled to The range of , thus eliminating the impact of different user body shapes, allowing posture data to be compared and processed on the same scale; Normalized key point coordinates A data matrix is ​​formed ; 3) Pre-emphasize the collected speech signal to increase the energy of the high-frequency part and compensate for the high-frequency loss during the utterance process. Then, the pre-emphasized speech signal is divided into multiple short-time frames for windowing. Then, the windowed frame signal is subjected to Fast Fourier Transform (FFT) to convert the time domain signal into the frequency domain signal. ； in, is the spectrum of the frequency domain signal, is the frame signal after windowing, is the time index, is the frequency index, It is the length of the frame signal. Through Fourier transform, the energy distribution of the signal at different frequencies is obtained, which helps to extract the frequency features in the signal for subsequent speech processing tasks.

3. The multimodal emotion recognition method based on zero-knowledge machine learning according to claim 1, characterized in that: The second step includes: (1) Use the attention mechanism combined with a simplified convolutional neural network (CNN) to extract key features of face images; The input image is a normalized grayscale image , pass it through a convolution layer using a 3x3 convolution kernel, 32 filters, and a ReLU activation function, and then use a 2x2 maximum pooling layer to obtain the pooled feature map ; The feature maps are then weighted using spatial attention and channel attention mechanisms, allowing the model to focus more on the most important parts and features of the image; right The spatial attention map is calculated by 1x1 convolution and softmax function: ； right After global average pooling is performed on each channel of , the channel attention weight is calculated through the fully connected layer: ； in, is the Sigmoid activation function, is the weight of the fully connected layer, GAP is the global average pooling; Get the attention weighted feature map: ； The weighted feature map Flatten to form a one-dimensional feature vector ; (2) Extracting MFCC features reflecting emotional state from speech signals; The spectrum obtained in step 1 Through a set of Mel filters, the Mel frequency energy spectrum is obtained , take the logarithm of the Mel frequency energy spectrum, and then perform discrete cosine transform (DCT) to obtain MFCC features: ； Among them, log means taking logarithm operation, represents discrete cosine transform; Flatten the obtained MFCC features to obtain a one-dimensional feature vector ; (3) Use the self-attention mechanism to extract features from the posture data, so that the model can focus on important features and key points in the posture data; Input the normalized posture data matrix in step 1 , Calculate the query matrix Q, key matrix K and value matrix V of the pose data: ； in, 、 and is the weight matrix, and then the attention weight is calculated: ； in, is the dimension of the key vector, and the softmax function is defined as follows: ； The attention output is then calculated: ； Flatten the attention output to get a one-dimensional feature vector ; (4) Concatenate the extracted facial feature vector, posture feature vector and voice feature vector to form a comprehensive feature vector : 。 4. The multimodal emotion recognition method based on zero-knowledge machine learning according to claim 1, characterized in that: The server-side encryption calculation in step 4 includes: (1) Design a convolutional neural network (CNN) architecture suitable for homomorphic encryption: Convolutional layer 1: 1x3 convolution kernel, 64 filters, ReLU activation function is approximately a piecewise linear function; Average pooling layer 1: 1x2 pooling kernel; Convolutional layer 2: 1x3 convolution kernel, 128 filters, ReLU activation function is approximately a piecewise linear function; Average pooling layer 2: 1x2 pooling kernel; Convolutional layer 3: 1x3 convolution kernel, 256 filters, ReLU activation function is approximately a piecewise linear function; Average pooling layer 3: 1x2 pooling kernel; Fully connected layer: The input dimension is the length of the flattened convolution output, and the output dimension is the number of emotion categories; (2) Select multimodal data collected in a real environment as the emotion dataset, split the dataset into training set, test set and validation set in a ratio of 8:1:1, train the multimodal network, select and save the weight file with the best training effect, which contains the model parameters learned during the training process and will be used to generate prediction results in the inference stage. Finally, save the CNN architecture and weight file to the server; (3) Forward propagation algorithm 1) Input encrypted feature vector , is the input of the forward propagation; 2) Convolutional layer 1: Encrypted convolution calculation: Encrypt the feature vector Convolution operation is performed through convolution layer 1: ； in, is the encrypted convolution kernel of convolution layer 1, represents the convolution operation, It is the encrypted feature map after convolution; Encrypted activation function calculation: Encrypted feature map Apply a piecewise linear function to approximate the ReLU activation operation; ； in, is the piecewise linear approximation function of ReLU, is the encrypted output after activation; 3) Average pooling layer 1: Encrypted feature map after activation Perform a one-dimensional average pooling operation to reduce the dimension of the feature map; ； in, It is the encrypted feature map after pooling, and the size of the pooling window is 2; 4) Repeat convolution, activation and pooling operations: right After convolution layer 2 and average pooling layer 2: ； ； ； Then it will After convolution layer 3 and average pooling layer 3: ； ； ； Get the final encrypted feature map ; 5) Fully connected layer: Encrypted feature map after pooling Flatten it to get an encrypted long vector, and then calculate the encrypted output vector through the fully connected layer: ； in, is the encrypted weight matrix of the fully connected layer obtained by training the model, is the encrypted output vector of the fully connected layer.

5. The multimodal emotion recognition method based on zero-knowledge machine learning according to claim 1, characterized in that: In the step 5: The server will encrypt the emotion recognition results The encrypted result is transmitted back to the local device through a secure channel, and then the local device’s private key SK is used to decrypt the encrypted result to obtain the plaintext vector ; Then after decryption, the plaintext vector Apply the Softmax function to convert it into a probability distribution: ； in, Representative The probability value of the emotion category, is the plaintext vector No. elements, represents the sum of the indices of all seven emotion categories; Final Is a probability array containing 7 elements, representing the probability of each emotion category; The local device saves the emotion label list: ； Find out The emotion category with the highest probability, according to Find the corresponding emotion label and then display the predicted emotion label to the user.

Images