Multi-modal emotion recognition method based on zero-knowledge machine learning

Abstract

The invention relates to a multi-modal emotion recognition method based on zero-knowledge machine learning, and belongs to the crossing field of artificial intelligence and information security. Key features of local face images, voice signals and posture data multi-modal data are extracted through an attention module, then the key features are transmitted to a server after homomorphic encryption, training of a convolutional neural network (CNN) model is carried out in an encryption domain, and finally a result is sent back to the local for decryption and emotion recognition. The method has the advantages that by using the homomorphic encryption technology, in the whole data processing and transmission process, privacy data of a user is always kept in an encrypted state, and due to fusion of multi-modal features, the emotion recognition accuracy is improved, the robustness of the system in a complex environment is enhanced, the emotion recognition result is more stable and reliable, and the user experience is improved. The method can be widely applied to multiple fields of intelligent monitoring systems, user sentiment analysis, human-computer interaction, mental health monitoring and the like.

Description

Technical Field

[0001] The present invention belongs to the intersection of artificial intelligence and information security, and specifically relates to a multimodal emotion recognition method based on zero-knowledge machine learning (zkML). Background Art

[0002] With the rapid development of artificial intelligence and big data technology, emotion recognition has been widely used in many fields such as smart security, smart home, and health monitoring. However, most traditional emotion recognition methods rely on a large amount of plaintext data for training and reasoning, which poses a great risk in terms of data privacy protection.

[0003] Currently, emotion recognition is mainly achieved through the following steps:

[0004] 1. Data collection: Obtain facial images, voice signals and posture data through cameras, microphones or other sensors.

[0005] 2. Data preprocessing: normalize and crop the collected images to extract the face area; perform noise reduction and feature extraction on the voice signals; and standardize the posture data.

[0006] 3. Feature extraction: Use deep learning models such as convolutional neural networks (CNN) to extract key features from face images, MFCC features from speech signals, and motion features from posture data.

[0007] 4. Emotion classification: Input the extracted multimodal features into the classifier and output the emotion recognition results.

[0008] Although these methods have made some progress in accuracy, they have the following shortcomings in data privacy protection:

[0009] 1. Data privacy leakage: During the process of data collection and transmission, unencrypted data can be easily stolen or tampered with, leading to user privacy leakage.

[0010] 2. Data storage security: A large amount of user data is stored on the server, which poses a risk of illegal access and abuse of the data.

[0011] 3. Data processing transparency: In existing emotion recognition systems, the data processing process is not transparent and users cannot control and supervise how their data is used.

[0012] In order to solve the above problems, homomorphic encryption technology came into being. Unlike traditional encryption technology, homomorphic encryption allows calculations to be performed while keeping the data encrypted, thereby completing complex calculations without leaking plaintext data. The emergence of homomorphic encryption provides an effective solution for data privacy protection, especially for scenarios that need to process large amounts of sensitive data.

[0013] Although homomorphic encryption has significant advantages, it still faces the following challenges in practical applications:

[0014] 1. High computational overhead: Homomorphic encryption has high computational complexity and requires high system performance.

[0015] 2. Model training is difficult: Machine learning model training in an encrypted domain requires overcoming multiple technical difficulties, such as gradient calculation and model update.

[0016] 3. Data transmission efficiency: The transmission of encrypted data needs to ensure efficiency and security to prevent data from being attacked during transmission.

[0017] With the rapid development of artificial intelligence and information security technology, the importance of protecting user privacy in data processing and transmission has become increasingly prominent. Summary of the invention

[0018] This paper proposes a multimodal emotion recognition method based on zero-knowledge machine learning, which achieves data privacy protection and efficient emotion recognition by combining homomorphic encryption and attention mechanism.

[0019] The technical solution adopted by the present invention comprises the following steps:

[0020] Step 1: Data collection and preprocessing: The local device collects the user's face image, voice signal and posture data, and improves the data quality through preprocessing steps;

[0021] Step 2: Feature extraction: Use the attention mechanism to extract features from the image, extract MFCC features from the speech signal, and extract motion features from the posture data. All these multimodal features are fused into a comprehensive feature vector.

[0022] Step 3: Zero-knowledge proof: The local device generates a zero-knowledge proof to verify the authenticity of the feature data, and uses a homomorphic encryption algorithm to encrypt the comprehensive feature vector. This encryption method allows calculations to be performed while keeping the data encrypted, ensuring the privacy of the data during transmission and processing. The encrypted feature vector and zero-knowledge proof are transmitted to the remote server through a secure channel. During the transmission process, the data is always encrypted to prevent any third party from stealing or tampering with the data.

[0023] Step 4: Server-side encryption calculation: After receiving the encrypted feature vector and zero-knowledge proof, the server first verifies the authenticity of the data, and then uses the convolutional neural network (CNN) to train the model in the encrypted domain. Due to the characteristics of homomorphic encryption, the server can calculate the encrypted data without decryption, thereby protecting data privacy.

[0024] Step 5: Emotion recognition reasoning and result processing: After training is completed, the server transmits the encrypted reasoning results back to the local device through a secure channel. The local device decrypts the received encrypted results to obtain the final results of emotion recognition, and decodes the decrypted prediction values ​​into corresponding emotion labels to display to the user.

[0025] The step one of the present invention comprises:

[0026] (1) Multimodal emotion data collection:

[0027] The user's portrait photo is taken through the local device's portrait camera to obtain the user's facial feature data; the user's posture data is captured using the depth camera; the user's voice information is collected through the microphone device, and this data will be used to analyze the user's emotional state, where emotions can be marked as seven basic emotions: happiness, anger, sadness, fear, surprise, disgust, and calmness;

[0028] (2) Data preprocessing:

[0029] 1) Use the face detection algorithm MTCNN to detect the face area in the portrait image I, and obtain the face bounding box, including the upper left corner coordinates (x, y) and width and height (w, h). Crop the face area image I according to the detected bounding box face ;

[0030] I face =I[y:y+h,x:x+w]

[0031] Convert RGB images to grayscale images to reduce the amount of calculation. Grayscale images directly reflect brightness changes and are more helpful for emotion recognition. The conversion formula is:

[0032] I gray =0.2989·I R +0.5870·I G +0.1140·I B

[0033] Among them, I R ,I G and I B Represents the pixel value matrices of the red, green, and blue channels of the image respectively;

[0034] Normalize the pixel values ​​to the range of [0,1] to eliminate the influence of brightness differences. At the same time, normalized data can speed up the convergence of the gradient descent algorithm and improve training efficiency. The normalization formula is:

[0035]

[0036] 2) Use the posture estimation algorithm OpenPose to detect the key points of the human body and obtain the 3D coordinates (x i ,y i ,z i );

[0037] In order to eliminate the differences between different users, the key point coordinates are normalized:

[0038]

[0039] The max and min in the normalization process are the maximum and minimum values ​​of all key point coordinates, which respectively represent the maximum and minimum range of the user's posture on each coordinate axis (x, y, z). Through these values, the coordinates of all key points are scaled to the range of [0, 1], thereby eliminating the influence of different user body shapes and making the posture data comparable and processed on the same scale;

[0040] The normalized key point coordinates (x i ′,y i ′,z i ′) constitutes a data matrix P;

[0041] 3) Perform pre-emphasis processing on the collected speech signal to increase the energy of the high-frequency part and compensate for the high-frequency loss during the utterance process. Then, the emphasized speech signal is divided into multiple short-time frames for windowing processing. Then, the windowed frame signal is subjected to fast Fourier transform FFT to convert the time domain signal into a frequency domain signal.

[0042]

[0043] Among them, Y(f) is the spectrum of the frequency domain signal, y frame (t) is the windowed frame signal, t is the time index, f is the frequency index, and N is the length of the frame signal. Through Fourier transform, the energy distribution of the signal at different frequencies is obtained, which helps to extract the frequency features in the signal for subsequent speech processing tasks.

[0044] The second step of the present invention comprises:

[0045] (1) Use the attention mechanism combined with a simplified convolutional neural network (CNN) to extract the key features of face images;

[0046] The input image is the normalized grayscale image I norm , pass it through a convolution layer using a 3x3 convolution kernel, 32 filters, and a ReLU activation function, and then use a 2x2 maximum pooling layer to obtain the pooled feature map P1;

[0047] The feature maps are then weighted using spatial attention and channel attention mechanisms, allowing the model to focus more on the most important parts and features of the image;

[0048] Calculate the spatial attention map of P1 through 1x1 convolution and softmax function:

[0049] Attention spatial =softmax(Conv1x1(P1))

[0050] After global average pooling of each channel of P1, the channel attention weight is calculated through the fully connected layer:

[0051] Attention channel =σ(W1(GAP(P1)))

[0052] Among them, σ is the Sigmoid activation function, W1 is the weight of the fully connected layer, and GAP is the global average pooling;

[0053] Get the attention weighted feature map:

[0054] F attention =Attention spatial P1+Attention channel P1

[0055] The weighted feature map F attention Flatten to form a one-dimensional feature vector f face ;

[0056] (2) Extract MFCC features reflecting emotional state from speech signals;

[0057] The spectrum Y(f) obtained in step 1 is passed through a set of Mel filters to obtain the Mel frequency energy spectrum Mel(Y(f)), the logarithm of the Mel frequency energy spectrum is taken, and then discrete cosine transform (DCT) is performed to obtain the MFCC features:

[0058] MFCC = DCT(log(Mel(Y(f))))

[0059] Among them, log means taking logarithm operation, DCT means discrete cosine transform;

[0060] Flatten the obtained MFCC features to obtain a one-dimensional feature vector f audio ;

[0061] (3) Use the self-attention mechanism to extract features from posture data, so that the model can focus on important features and key points in the posture data;

[0062] Input the normalized posture data matrix P in step 1,

[0063] Calculate the query matrix Q, key matrix K and value matrix V of the pose data:

[0064] Q=W Q P, K = W K P,V=W V P

[0065] Among them, W Q , W K and W V is the weight matrix, and then the attention weight is calculated:

[0066]

[0067] Among them, d k is the dimension of the key vector, and the softmax function is defined as follows:

[0068]

[0069] The attention output is then calculated:

[0070] Output=Attention·V

[0071] Flatten the attention output to get a one-dimensional feature vector f pose ;

[0072] (4) Concatenate the extracted facial feature vector, posture feature vector, and voice feature vector to form a comprehensive feature vector f combined :

[0073] f combined =[f face ;f pose ;f audio ].

[0074] The generation of zero-knowledge proof in step 3 of the present invention includes:

[0075] (1) First, initialize the zero-knowledge proof system on the local device and use the zero-knowledge proof system zk-SNARKs to generate the system's public key PK and private key SK. These keys will be used to generate and verify proofs of feature vectors without leaking any original data.

[0076] (2) Using the public key PK and the comprehensive feature vector f combinedGenerate a zero-knowledge proof π, which ensures the authenticity of the feature vector generation process without leaking the original data;

[0077] (3) In order to protect data privacy during transmission, homomorphic encryption technology is used to encrypt the proof π and the comprehensive feature vector f combined Encrypt and generate the encrypted proof c π And the encrypted feature data c f ;

[0078] The encrypted proof c π And the encrypted feature data c f Transmit to the server for verification;

[0079] (4) The server receives the encrypted certificate c π and encrypted feature data c f After that, first prove c π Decrypt and obtain the proof π, then the server uses π to verify the encrypted feature data c f authenticity and integrity without the need for f Decryption is performed, and based on the verification results, the server decides whether to accept the transmitted data and performs subsequent processing in the data encrypted state.

[0080] The server-side encryption calculation in step 4 of the present invention includes:

[0081] (1) Design a convolutional neural network (CNN) architecture suitable for homomorphic encryption:

[0082] Convolutional layer 1: 1x3 convolution kernel, 64 filters, ReLU activation function is approximately a piecewise linear function;

[0083] Average pooling layer 1: 1x2 pooling kernel;

[0084] Convolutional layer 2: 1x3 convolution kernel, 128 filters, ReLU activation function is approximately a piecewise linear function;

[0085] Average pooling layer 2: 1x2 pooling kernel;

[0086] Convolutional layer 3: 1x3 convolution kernel, 256 filters, ReLU activation function is approximately a piecewise linear function;

[0087] Average pooling layer 3: 1x2 pooling kernel;

[0088] Fully connected layer: the input dimension is the length of the flattened convolution output, and the output dimension is the number of emotion categories;

[0089] (2) Select multimodal data collected in a real environment as the emotion dataset, divide the dataset into training set, test set and validation set in a ratio of 8:1:1, train the multimodal network, select and save the weight file with the best training effect, which contains the model parameters learned during the training process and will be used to generate prediction results in the inference stage. Finally, save the CNN architecture and weight file to the server.

[0090] (3) Forward Propagation Algorithm

[0091] 1) Input encrypted feature vector c f , c f is the input of the forward propagation;

[0092] 2) Convolutional layer 1:

[0093] Encrypted convolution calculation: Encrypt the feature vector c f Convolution operation is performed through convolution layer 1:

[0094] Enc(Z1)=Enc(W1)*c f

[0095] Among them, Enc(W1) is the encrypted convolution kernel of convolution layer 1, * represents the convolution operation, and Enc(Z1) is the encrypted feature map after convolution;

[0096] Encrypted activation function calculation: Apply a piecewise linear function to the encrypted feature map Enc(Z1) to approximate the ReLU activation operation;

[0097] Enc(A1)=f piecewise (Enc(Z1))

[0098] Among them, f piecewise (·) is the piecewise linear approximation function of ReLU, Enc(A1) is the encrypted output after activation;

[0099] 3) Average pooling layer 1:

[0100] Perform a one-dimensional average pooling operation on the activated encrypted feature map Enc(A1) to reduce the dimension of the feature map;

[0101]

[0102] Among them, Enc(G1) is the encrypted feature map after pooling, and the size of the pooling window is 2;

[0103] 4) Repeat convolution, activation and pooling operations:

[0104] For Enc(G1), after convolution layer 2 and average pooling layer 2:

[0105] Enc(Z2)=Enc(W2)*Enc(G1)

[0106] Enc(A2)=f piecewise (Enc(Z2))

[0107]

[0108] Then Enc(G2) ​​passes through convolution layer 3 and average pooling layer 3:

[0109] Enc(Z3)=Enc(W3)*Enc(G2)

[0110] Enc(A3)=f piecewise (Enc(Z3))

[0111]

[0112] Get the final encrypted feature map Enc(G3);

[0113] 5) Fully connected layer:

[0114] The pooled encrypted feature map Enc(G3) is flattened to obtain an encrypted long vector, and then the encrypted output vector is calculated through the fully connected layer:

[0115] Enc(Y)=Enc(W fc )·Enc(G3)

[0116] Among them, Enc(W fc ) is the encrypted weight matrix of the fully connected layer obtained by training the model, and Enc(Y) is the encrypted output vector of the fully connected layer.

[0117] In step 5 of the present invention:

[0118] The server transmits the encrypted emotion recognition result Enc(Y) back to the local device through a secure channel, and then uses the local device's private key SK to decrypt the encrypted result to obtain the plaintext vector y;

[0119] The Softmax function is then applied to the decrypted plaintext vector Y to convert it into a probability distribution:

[0120]

[0121] Among them, result i represents the probability value of the i-th emotion category, Y i is the i-th element of the plaintext vector Y, It means summing the indices of all 7 emotion categories;

[0122] The final result is a probability array containing 7 elements, indicating the probability of each emotion category;

[0123] The local device saves the list of emotion labels:

[0124] labels=["Happy", "Angry", "Sad", "Fear", "Surprise", "Disgust", "Calm"]

[0125] Find the emotion category with the highest probability in the result, find the corresponding emotion label according to labels, and then display the predicted emotion label to the user.

[0126] In the multimodal emotion recognition method based on zero-knowledge machine learning (zkML) proposed in the present invention, the local multimodal data (including face images, voice signals and posture data) is first extracted with key features by the attention module, and then transmitted to the server after homomorphic encryption, and the convolutional neural network CNN model is trained in the encrypted domain. Finally, the result is transmitted back to the local for decryption and emotion recognition, so that data privacy protection and efficient emotion recognition are organically combined. Compared with the traditional method, it has the following advantages:

[0127] (1) Comprehensive protection of data privacy: By using homomorphic encryption technology, the user's private data is always encrypted during the entire data processing and transmission process. This means that the user's sensitive information is protected whether on the local device or the remote server, avoiding the risk of data leakage. At the same time, the introduction of zero-knowledge proof ensures that the remote server can verify the authenticity of the data without exposing the original data, fundamentally improving data security.

[0128] (2) Efficiency of multimodal feature fusion: The present invention integrates multimodal information such as face images, voice signals and posture data, and uses attention mechanism and feature fusion technology to generate a comprehensive feature vector. This fusion of multimodal features not only improves the accuracy of emotion recognition, but also enhances the robustness of the system in complex environments, making the emotion recognition results more stable and reliable.

[0129] (3) Seamless integration of encrypted computing: Thanks to the characteristics of homomorphic encryption, the server can directly perform calculations on encrypted data without decryption, thereby protecting data privacy while still being able to perform complex deep learning model training and reasoning. This approach overcomes the problem of reduced computing performance in traditional encryption methods and achieves a balance between privacy protection and efficient computing.

[0130] (4) Wide application potential: The multimodal emotion recognition method based on the present invention can be widely used in multiple fields such as intelligent monitoring systems, user emotion analysis, human-computer interaction, and mental health monitoring due to its significant improvement in data privacy protection and emotion recognition accuracy, providing a safer and more efficient solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0131] Figure 1 is a flow chart of the emotion recognition system of the present invention;

[0132] Figure 2 It is a simplified convolutional neural network diagram combined with the attention mechanism of the present invention;

[0133] Figure 3 It is a homomorphically encrypted convolutional neural network (CNN) graph on the server side of the present invention. DETAILED DESCRIPTION

[0134] See also Figure 1 , Step 1, data collection and preprocessing:

[0135] (1) Multimodal emotion data collection:

[0136] The user's portrait photo is taken through the local device's portrait camera to obtain the user's facial feature data; the user's posture data is captured using the depth camera; the user's voice information is collected through the microphone device, and this data will be used to analyze the user's emotional state, where emotions can be marked as seven basic emotions: happiness, anger, sadness, fear, surprise, disgust, and calmness;

[0137] (2) Data preprocessing:

[0138] 1) Use the face detection algorithm MTCNN to detect the face area in the portrait image I, and obtain the face bounding box, including the upper left corner coordinates (x, y) and width and height (w, h). Crop the face area image I according to the detected bounding box face ;

[0139] I face =I[y:y+h,x:x+w]

[0140] Convert RGB images to grayscale images to reduce the amount of calculation. Grayscale images directly reflect brightness changes and are more helpful for emotion recognition. The conversion formula is:

[0141] I gray =0.2989·I R +0.5870·I G +0.1140·I B

[0142] Among them, IR ,I G and I B Represents the pixel value matrices of the red, green, and blue channels of the image respectively;

[0143] Normalize the pixel values ​​to the range of [0, 1] to eliminate the influence of brightness differences. At the same time, normalized data can speed up the convergence of the gradient descent algorithm and improve training efficiency. The normalization formula is:

[0144]

[0145] 2) Use the posture estimation algorithm OpenPose to detect the key points of the human body and obtain the 3D coordinates (x i ,y i , z i );

[0146] In order to eliminate the differences between different users, the key point coordinates are normalized:

[0147]

[0148] The max and min in the normalization process are the maximum and minimum values ​​of all key point coordinates, which respectively represent the maximum and minimum range of the user's posture on each coordinate axis (x, y, z). Through these values, the coordinates of all key points are scaled to the range of [0, 1], thereby eliminating the influence of different user body shapes and making the posture data comparable and processed on the same scale;

[0149] The normalized key point coordinates (x′ i , y′ i , z′ i ) constitutes a data matrix P;

[0150] 3) Perform pre-emphasis processing on the collected speech signal to increase the energy of the high-frequency part and compensate for the high-frequency loss during the utterance process. Then, the emphasized speech signal is divided into multiple short-time frames for windowing processing. Then, the windowed frame signal is subjected to fast Fourier transform FFT to convert the time domain signal into a frequency domain signal.

[0151]

[0152] Among them, Y(f) is the spectrum of the frequency domain signal, y frame (t) is the frame signal after windowing, t is the time index, f is the frequency index, and N is the length of the frame signal. Through Fourier transform, the energy distribution of the signal at different frequencies is obtained, which helps to extract the frequency features in the signal for subsequent speech processing tasks;

[0153] Step 2: Feature extraction:

[0154] Since the subsequent zero-knowledge proof and homomorphic encryption cannot handle large amounts of data, it is necessary to perform simple feature extraction on the local device to reduce the amount of data to meet the requirements of transmission encryption. In this step, the attention module will be used to extract features from face and posture data, and MFCC features that can reflect emotional states will be extracted from the voice signal.

[0155] (1) Use the attention mechanism combined with a simplified convolutional neural network (CNN) to extract the key features of face images;

[0156] See also Figure 2 , the input image is the standardized grayscale image I norm , pass it through a convolution layer using a 3x3 convolution kernel, 32 filters, and a ReLU activation function, and then use a 2x2 maximum pooling layer to obtain the pooled feature map P1;

[0157] The feature maps are then weighted using spatial attention and channel attention mechanisms, allowing the model to focus more on the most important parts and features of the image;

[0158] Calculate the spatial attention map of P1 through 1x1 convolution and softmax function:

[0159] Attention spatial =softmax(Conv1x1(P1))

[0160] After global average pooling of each channel of P1, the channel attention weight is calculated through the fully connected layer:

[0161] Attention channel =σ(W1(GAP(P1)))

[0162] Among them, σ is the Sigmoid activation function, W1 is the weight of the fully connected layer, and GAP is the global average pooling;

[0163] Get the attention weighted feature map:

[0164] F attention =Attention spatial P1+Attention channel P1

[0165] The weighted feature map F attention Flatten to form a one-dimensional feature vector f face ;

[0166] (2) Extract MFCC features reflecting emotional state from speech signals;

[0167] The spectrum Y(f) obtained in step 1 is passed through a set of Mel filters to obtain the Mel frequency energy spectrum Mel(Y(f)), the logarithm of the Mel frequency energy spectrum is taken, and then discrete cosine transform (DCT) is performed to obtain the MFCC features:

[0168] MFCC = DCT(log(Mel(Y(f))))

[0169] Among them, log means taking logarithm operation, DCT means discrete cosine transform;

[0170] Flatten the obtained MFCC features to obtain a one-dimensional feature vector f audio ;

[0171] (3) Use the self-attention mechanism to extract features from posture data, so that the model can focus on important features and key points in the posture data;

[0172] Input the normalized posture data matrix P in step 1,

[0173] Calculate the query matrix Q, key matrix K and value matrix V of the pose data:

[0174] Q=W Q P, K = W K P,V=W V P

[0175] Among them, W Q , W K and W V is the weight matrix, and then the attention weight is calculated:

[0176]

[0177] Among them, d k is the dimension of the key vector, and the softmax function is defined as follows:

[0178]

[0179] The attention output is then calculated:

[0180] Output=Attention·V

[0181] Flatten the attention output to get a one-dimensional feature vector f pose ;

[0182] (4) Concatenate the extracted facial feature vector, posture feature vector, and voice feature vector to form a comprehensive feature vector f combined :

[0183] f combined =[fface ;f pose ;f audio ]

[0184] Step 3: Zero-knowledge proof

[0185] In this step, a zero-knowledge proof will be generated to ensure the authenticity and integrity of the feature data, and homomorphic encryption will be used to protect data privacy;

[0186] (1) First, initialize the zero-knowledge proof system on the local device and use the zero-knowledge proof system zk-SNARKs to generate the system's public key PK and private key SK. These keys will be used to generate and verify proofs of feature vectors without leaking any original data.

[0187] (2) Using the public key PK and the comprehensive feature vector f combined Generate a zero-knowledge proof π, which ensures the authenticity of the feature vector generation process without leaking the original data;

[0188] (3) In order to protect data privacy during transmission, homomorphic encryption technology is used to encrypt the proof π and the comprehensive feature vector f combined Encrypt and generate the encrypted proof c π And the encrypted feature data c f ;

[0189] The encrypted proof c π And the encrypted feature data c f Transmit to the server for verification;

[0190] (4) The server receives the encrypted certificate c π and encrypted feature data c f After that, first prove c π Decrypt and obtain the proof π, then the server uses π to verify the encrypted feature data c f authenticity and integrity without the need for f Decryption is performed. Based on the verification result, the server decides whether to accept the transmitted data and performs subsequent processing in the encrypted state.

[0191] Step 4: Server-side encryption calculation

[0192] (1) The CNN architecture running in the homomorphic encryption domain needs to consider computational efficiency and encrypted data processing capabilities. The following is a convolutional neural network (CNN) architecture design suitable for homomorphic encryption. Figure 3 As shown:

[0193] Convolutional layer 1: 1x3 convolution kernel, 64 filters, ReLU activation function is approximately a piecewise linear function;

[0194] Average pooling layer 1: 1x2 pooling kernel;

[0195] Convolutional layer 2: 1x3 convolution kernel, 128 filters, ReLU activation function is approximately a piecewise linear function;

[0196] Average pooling layer 2: 1x2 pooling kernel;

[0197] Convolutional layer 3: 1x3 convolution kernel, 256 filters, ReLU activation function is approximately a piecewise linear function;

[0198] Average pooling layer 3: lx2 pooling kernel;

[0199] Fully connected layer: the input dimension is the length of the flattened convolution output, and the output dimension is the number of emotion categories;

[0200] (2) Select multimodal data collected in a real environment as the emotion dataset, divide the dataset into training set, test set and validation set in a ratio of 8:1:1, train the multimodal network, select and save the weight file with the best training effect, which contains the model parameters learned during the training process and will be used to generate prediction results in the inference stage. Finally, save the CNN architecture and weight file to the server.

[0201] (3) Forward Propagation Algorithm

[0202] 1) Input encrypted feature vector c f , c f is the input of the forward propagation;

[0203] 2) Convolutional layer 1:

[0204] Encrypted convolution calculation: Encrypt the feature vector c f Convolution operation is performed through convolution layer 1:

[0205] Enc(Z1)=Enc(W1)*c f

[0206] Among them, Enc(W1) is the encrypted convolution kernel of convolution layer 1, * represents the convolution operation, and Enc(Z1) is the encrypted feature map after convolution;

[0207] Encrypted activation function calculation: Apply a piecewise linear function to the encrypted feature map Enc(Z1) to approximate the ReLU activation operation; the piecewise linear function is easier to implement in the homomorphic encryption domain because it can be calculated through simple linear operations;

[0208] Enc(A1)=f piecewise (Enc(Z1))

[0209] Among them, fpiecewise (·) is the piecewise linear approximation function of ReLU, Enc(A1) is the encrypted output after activation;

[0210] 3) Average pooling layer 1:

[0211] Perform a one-dimensional average pooling operation on the activated encrypted feature map Enc(A1) to reduce the dimension of the feature map;

[0212]

[0213] Among them, Enc(G1) is the encrypted feature map after pooling, and the size of the pooling window is 2;

[0214] 4) Repeat convolution, activation and pooling operations:

[0215] For Enc(G1), after convolution layer 2 and average pooling layer 2:

[0216] Enc(Z2)=Enc(W2)*Enc(G1)

[0217] Enc(A2)=f piecewise (Enc(Z2))

[0218]

[0219] Then Enc(G2) ​​passes through convolution layer 3 and average pooling layer 3:

[0220] Enc(Z3)=Enc(W3)*Enc(G2)

[0221] Enc(A3)=f piecewise (Enc(Z3))

[0222]

[0223] Get the final encrypted feature map Enc(G3);

[0224] 5) Fully connected layer:

[0225] The pooled encrypted feature map Enc(G3) is flattened to obtain an encrypted long vector, and then the encrypted output vector is calculated through the fully connected layer:

[0226] Enc(Y)=Enc(W fc )·Enc(G3)

[0227] Among them, Enc(W fc ) is the encrypted weight matrix of the fully connected layer obtained by the training model, and Enc(Y) is the encrypted output vector of the fully connected layer;

[0228] Step 5: Emotion recognition reasoning and result processing

[0229] The server transmits the encrypted emotion recognition result Enc(Y) back to the local device through a secure channel, and then uses the local device's private key SK to decrypt the encrypted result to obtain the plaintext vector y;

[0230] The Softmax function is then applied to the decrypted plaintext vector Y to convert it into a probability distribution:

[0231]

[0232] Among them, result i represents the probability value of the i-th emotion category, Y i is the i-th element of the plaintext vector Y, It means summing the indices of all 7 emotion categories;

[0233] The final result is a probability array containing 7 elements, indicating the probability of each emotion category;

[0234] The local device saves the list of emotion labels:

[0235] labels=["Happy", "Angry", "Sad", "Fear", "Surprise", "Disgust", "Calm"]

[0236] Find the emotion category with the highest probability in the result, find the corresponding emotion label according to labels, and then display the predicted emotion label to the user.

Claims

1. A multimodal emotion recognition method based on zero-knowledge machine learning, characterized in that: The following steps are involved: Step 1: Data collection and preprocessing: The local device collects the user's face image, voice signal and posture data, and improves the data quality through preprocessing steps; Step 2: Feature extraction: Use the attention mechanism to extract features from the image, extract MFCC features from the speech signal, and extract motion features from the posture data. All these multimodal features are fused into a comprehensive feature vector. Step 3: Zero-knowledge proof: The local device generates a zero-knowledge proof to verify the authenticity of the feature data, and uses a homomorphic encryption algorithm to encrypt the comprehensive feature vector. This encryption method allows calculations to be performed while keeping the data encrypted, ensuring the privacy of the data during transmission and processing. The encrypted feature vector and zero-knowledge proof are transmitted to the remote server through a secure channel. During the transmission process, the data is always encrypted to prevent any third party from stealing or tampering with the data. Step 4: Server-side encryption calculation: After receiving the encrypted feature vector and zero-knowledge proof, the server first verifies the authenticity of the data, and then uses the convolutional neural network (CNN) to train the model in the encrypted domain. Due to the characteristics of homomorphic encryption, the server can calculate the encrypted data without decryption, thereby protecting data privacy. Step 5: Emotion recognition reasoning and result processing: After training is completed, the server transmits the encrypted reasoning results back to the local device through a secure channel. The local device decrypts the received encrypted results to obtain the final results of emotion recognition, and decodes the decrypted prediction values ​​into corresponding emotion labels to display to the user.

2. A multimodal emotion recognition method based on zero-knowledge machine learning according to claim 1, characterized in that: The step one comprises: (1) Multimodal emotion data collection: The user's portrait photo is taken through the local device's portrait camera to obtain the user's facial feature data; the user's posture data is captured using the depth camera; the user's voice information is collected through the microphone device, and this data will be used to analyze the user's emotional state, where emotions can be marked as seven basic emotions: happiness, anger, sadness, fear, surprise, disgust, and calmness; (2) Data preprocessing: 1) Use the face detection algorithm MTCNN to detect the face area in the portrait image I, and obtain the face bounding box, including the upper left corner coordinates (x, y) and width and height (w, h). Crop the face area image I according to the detected bounding box face ; I face =I[y:y+h,x:x+w] Convert RGB images to grayscale images to reduce the amount of calculation. Grayscale images directly reflect brightness changes and are more helpful for emotion recognition. The conversion formula is: I gray =0.2989·I R +0.5870·I G +0.1140·I B Among them, I R ,I G and I B Represents the pixel value matrices of the red, green, and blue channels of the image respectively; Normalize the pixel values ​​to the range of [0, 1] to eliminate the influence of brightness differences. At the same time, normalized data can speed up the convergence of the gradient descent algorithm and improve training efficiency. The normalization formula is: 2) Use the posture estimation algorithm OpenPose to detect the key points of the human body and obtain the 3D coordinates (x i ,y i , z i ); In order to eliminate the differences between different users, the key point coordinates are normalized: The max and min in the normalization process are the maximum and minimum values ​​of all key point coordinates, which respectively represent the maximum and minimum range of the user's posture on each coordinate axis (x, y, z). Through these values, the coordinates of all key points are scaled to the range of [0, 1], thereby eliminating the influence of different user body shapes and making the posture data comparable and processed on the same scale; The normalized key point coordinates (x′ i , y′ i , z′ i ) constitutes a data matrix P; 3) Perform pre-emphasis processing on the collected speech signal to increase the energy of the high-frequency part and compensate for the high-frequency loss during the utterance process. Then, the emphasized speech signal is divided into multiple short-time frames for windowing processing. Then, the windowed frame signal is subjected to fast Fourier transform FFT to convert the time domain signal into a frequency domain signal. Among them, Y(f) is the spectrum of the frequency domain signal, y frame (t) is the windowed frame signal, t is the time index, f is the frequency index, and N is the length of the frame signal. Through Fourier transform, the energy distribution of the signal at different frequencies is obtained, which helps to extract the frequency features in the signal for subsequent speech processing tasks.

3. The multimodal emotion recognition method based on zero-knowledge machine learning according to claim 1, characterized in that: The second step comprises: (1) Use the attention mechanism combined with a simplified convolutional neural network (CNN) to extract the key features of face images; The input image is the normalized grayscale image I norm , pass it through a convolution layer using a 3x3 convolution kernel, 32 filters, and a ReLU activation function, and then use a 2x2 maximum pooling layer to obtain the pooled feature map P1; The feature maps are then weighted using spatial attention and channel attention mechanisms, allowing the model to focus more on the most important parts and features of the image; Calculate the spatial attention map of P1 through 1x1 convolution and softmax function: Attention spatial =softmax(Conv1x1(P1)) After global average pooling of each channel of P1, the channel attention weight is calculated through the fully connected layer: Attention channel =σ(W1(GAP(P1))) Among them, σ is the Sigmoid activation function, W1 is the weight of the fully connected layer, and GAP is the global average pooling; Get the attention weighted feature map: F attention =Attention spatial ·P1+Attention channel ·P1 The weighted feature map F attention Flatten to form a one-dimensional feature vector f face ; (2) Extract MFCC features reflecting emotional state from speech signals; The spectrum Y(f) obtained in step 1 is passed through a set of Mel filters to obtain the Mel frequency energy spectrum Mel(Y(f)), the logarithm of the Mel frequency energy spectrum is taken, and then discrete cosine transform (DCT) is performed to obtain the MFCC features: MFCC = DCT(log(Mel(Y(f)))) Among them, log means taking logarithm operation, DCT means discrete cosine transform; Flatten the obtained MFCC features to obtain a one-dimensional feature vector f audio ; (3) Use the self-attention mechanism to extract features from posture data, so that the model can focus on important features and key points in the posture data; Input the normalized posture data matrix P in step 1, Calculate the query matrix Q, key matrix K and value matrix V of the pose data: Q=W Q P,K=W K P,V=W V P Among them, W Q , W K and W V is the weight matrix, and then the attention weight is calculated: Among them, d k is the dimension of the key vector, and the softmax function is defined as follows: The attention output is then calculated: Output=Attention·V Flatten the attention output to get a one-dimensional feature vector f pose ; (4) Concatenate the extracted facial feature vector, posture feature vector, and voice feature vector to form a comprehensive feature vector f combined : f combined =[f face ;f pose ;f audio ]。 4. The multimodal emotion recognition method based on zero-knowledge machine learning according to claim 1, characterized in that: The zero-knowledge proof generation in step 3 includes: (1) First, initialize the zero-knowledge proof system on the local device and use the zero-knowledge proof system zk-SNARKs to generate the system's public key PK and private key SK. These keys will be used to generate and verify proofs of feature vectors without leaking any original data. (2) Using the public key PK and the comprehensive feature vector f combined Generate a zero-knowledge proof π, which ensures the authenticity of the feature vector generation process without leaking the original data; (3) In order to protect data privacy during transmission, homomorphic encryption technology is used to encrypt the proof π and the comprehensive feature vector f combined Encrypt and generate the encrypted proof c π and the encrypted feature data c f ; The encrypted proof c π And the encrypted feature data c f Transmit to the server for verification; (4) The server receives the encrypted certificate c π and encrypted feature data c f After that, first prove c π Decrypt and obtain the proof π, then the server uses π to verify the encrypted feature data c f authenticity and integrity without the need for f Decryption is performed, and based on the verification results, the server decides whether to accept the transmitted data and performs subsequent processing in the data encrypted state.

5. The multimodal emotion recognition method based on zero-knowledge machine learning according to claim 1, characterized in that: The server-side encryption calculation in step 4 includes: (1) Design a convolutional neural network (CNN) architecture suitable for homomorphic encryption: Convolutional layer 1: 1x3 convolution kernel, 64 filters, ReLU activation function is approximately a piecewise linear function; Average pooling layer 1: 1x2 pooling kernel; Convolutional layer 2: 1x3 convolution kernel, 128 filters, ReLU activation function is approximately a piecewise linear function; Average pooling layer 2: lx2 pooling kernel; Convolutional layer 3: 1x3 convolution kernel, 256 filters, ReLU activation function is approximately a piecewise linear function; Average pooling layer 3: 1x2 pooling kernel; Fully connected layer: the input dimension is the length of the flattened convolution output, and the output dimension is the number of emotion categories; (2) Select multimodal data collected in a real environment as the emotion dataset, divide the dataset into training set, test set and validation set in a ratio of 8:1:1, train the multimodal network, select and save the weight file with the best training effect, which contains the model parameters learned during the training process and will be used to generate prediction results in the inference stage. Finally, save the CNN architecture and weight file to the server. (3) Forward Propagation Algorithm 1) Input encrypted feature vector c f , c f is the input of the forward propagation; 2) Convolutional layer 1: Encrypted convolution calculation: Encrypt the feature vector c f Convolution operation is performed through convolution layer 1: Enc(Z1)=Enc(W1)*c f Among them, Enc(W1) is the encrypted convolution kernel of convolution layer 1, * represents the convolution operation, and Enc(Z1) is the encrypted feature map after convolution; Encrypted activation function calculation: Apply a piecewise linear function to the encrypted feature map Enc(Z1) to approximate the ReLU activation operation; Enc(A1)=f piecewise (Enc(Z1)) Among them, f piecewise (·) is the piecewise linear approximation function of ReLU, Enc(A1) is the encrypted output after activation; 3) Average pooling layer 1: Perform a one-dimensional average pooling operation on the activated encrypted feature map Enc(A1) to reduce the dimension of the feature map; Among them, Enc(G1) is the encrypted feature map after pooling, and the size of the pooling window is 2; 4) Repeat convolution, activation and pooling operations: For Enc(G1), after convolution layer 2 and average pooling layer 2: Enc(Z2)=Enc(W2)*Enc(G1) <h2 style=";text-align:left;direction:ltr">Enc(A2)=f<h2 style=";text-align:left;direction:ltr"> piecewise <h2 style=";text-align:left;direction:ltr"> (Enc(Z2)) Then Enc(G2) ​​passes through convolution layer 3 and average pooling layer 3: Enc(Z3)=Enc(W3)*Enc(G2) <h2 style=";text-align:left;direction:ltr">Enc(A3)=f<h2 style=";text-align:left;direction:ltr"> piecewise <h2 style=";text-align:left;direction:ltr"> (Enc(Z3)) Get the final encrypted feature map Enc(G3); 5) Fully connected layer: The pooled encrypted feature map Enc(G3) is flattened to obtain an encrypted long vector, and then the encrypted output vector is calculated through the fully connected layer: Enc(Y)=Enc(W fc )·Enc(G3) Among them, Enc(W f c) is the encrypted weight matrix of the fully connected layer obtained by the training model, and Enc(Y) is the encrypted output vector of the fully connected layer.

6. The multimodal emotion recognition method based on zero-knowledge machine learning according to claim 1, characterized in that: In the step 5: The server transmits the encrypted emotion recognition result Enc(Y) back to the local device through a secure channel, and then uses the local device's private key SK to decrypt the encrypted result to obtain the plaintext vector Y; The Softmax function is then applied to the decrypted plaintext vector Y to convert it into a probability distribution: Among them, result i represents the probability value of the i-th emotion category, Y i is the i-th element of the plaintext vector Y, It means summing the indices of all 7 emotion categories; The final result is a probability array containing 7 elements, indicating the probability of each emotion category; The local device saves the list of emotion labels: labels=["Happy", "Angry", "Sad", "Fear", "Surprise", "Disgust", "Calm"] Find the emotion category with the highest probability in the result, find the corresponding emotion label according to labels, and then display the predicted emotion label to the user.

Images