Edge-cloud collaborative based vehicle-mounted emotion computing task allocation method and system

Abstract

The application provides an edge-cloud collaborative vehicle-mounted emotion computing task allocation method and system, comprising the following steps: 1, collecting facial images, voice data and physiological indexes of drivers and passengers through a vehicle-mounted terminal, extracting key features and preprocessing; 2, transmitting the data to an edge node, if there is a historical personality profile, allocating a computing task and jumping to step 3, if there is no historical personality profile, uploading the data to a cloud server after homomorphic encryption and jumping to step 4; 3, the edge node analyzes the emotional state according to the key features of the facial images, voice data and physiological indexes, and generates dynamic instructions of an air conditioner according to different emotional states of the personality state of a user; 4, the cloud server completes data training in a ciphertext state, obtains a personality feature model, and transmits the personality features analyzed by the model back to the edge node to optimize air conditioner parameters under different personality-emotion combinations.

Description

Technical Field

[0001] This invention relates to the field of computing task allocation technology, and more specifically, to a method and system for allocating in-vehicle emotion computing tasks based on edge-cloud collaboration. Background Technology

[0002] Traditional car air conditioning systems adjust based on physical parameters such as temperature and humidity, failing to sense the driver's emotional state and personalized needs, resulting in insufficient comfort.

[0003] Current emotion computing technologies have the following shortcomings:

[0004] Centralized cloud processing: Emotion recognition solutions that rely on cloud servers are unable to meet real-time requirements due to network latency, and the uploading of sensitive data (such as facial images) poses privacy risks.

[0005] Vehicle terminal processing: Due to the limited computing power of vehicle terminals, they cannot handle complex multimodal fusion analysis (such as facial + voice + personality features).

[0006] Patent application CN118906749A discloses a dynamic vehicle air conditioning control method and device. The method includes: acquiring external environmental parameters and internal environmental parameters; the external environmental parameters include at least current time information and weather information of the vehicle's location, and the internal environmental parameters include at least internal temperature and the status of the occupants; determining the operating status of the vehicle air conditioning based on the external and internal environmental parameters; collecting user daily behavior data; acquiring a real-time dynamic air conditioning control strategy based on the external environmental parameters, the internal environmental parameters, and the user daily behavior data; and correcting the operating status of the vehicle air conditioning based on the real-time dynamic air conditioning control strategy. However, this patent cannot completely solve the existing technical problems, nor can it meet the needs of this invention. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for allocating in-vehicle emotion computing tasks based on edge-cloud collaboration.

[0008] The in-vehicle emotion computing task allocation method based on edge-cloud collaboration provided by the present invention includes:

[0009] Step 1: Collect facial images, voice data, and physiological indicators of drivers and passengers through the vehicle terminal, extract key features, and preprocess them;

[0010] Step 2: Transmit the data to the edge node. If a historical personality profile exists, assign a computing task and proceed to Step 3. If no historical personality profile exists, use the Paillier homomorphic encryption algorithm to homomorphically encrypt the data and then upload it to the cloud server. Proceed to Step 4.

[0011] Step 3: The edge nodes analyze the emotional state based on the key features of facial images, voice data and physiological indicators, and generate dynamic commands for the air conditioner according to the different emotional states of the user's personality.

[0012] Step 4: The cloud server completes data training in encrypted state to obtain a personality characteristic model. The personality characteristics analyzed by the model are transmitted back to the edge node to optimize the air conditioning parameters under different personality-emotion combinations.

[0013] Preferably, step 1 includes:

[0014] Extract key points 61 and 291 from the MediaPipe in the facial image to calculate the curvature of the mouth, and key points 33, 133, 362, and 263 to analyze the eye features;

[0015] Heart rate, respiratory rate, and blood pressure were extracted using a remote photoelectric pulse wave recording method.

[0016] Extract the MFCC features of the speech and delete the original recording data locally.

[0017] Preferably, step 2 includes:

[0018] Based on bandwidth, latency, and packet loss rate, the network status is divided into three levels: excellent, medium, and poor. When the network status is excellent, all tasks are allowed to be processed in the cloud. When the network status is medium, only non-sensitive tasks are processed in the cloud. When the network status is poor, cloud communication is prohibited, and all tasks are forced to be processed at the edge.

[0019] For multimodal fusion and personality modeling tasks, assignment to the cloud is only performed when the network condition is optimal;

[0020] For tasks such as voice emotion analysis, facial landmark detection, and physiological information analysis, processing is performed locally when the edge load is below a preset threshold; otherwise, the tasks are assigned to the cloud.

[0021] Facial data is encrypted using AES-256, voice data is encrypted using the SM2 national cryptographic algorithm, and physiological data is encrypted using CKKS homomorphic encryption.

[0022] Preferably, step 3 includes:

[0023] Obtain historical personality profiles, deploy a multimodal fusion model at edge nodes, input key features of facial images, voice data and physiological indicators, and output the emotional state with the highest probability.

[0024] First, calculate the self-attention weights for facial, physiological, and speech modal features;

[0025] The facial feature expression is:

[0026]

[0027] The physiological characteristic expression is:

[0028]

[0029] The speech feature expression is:

[0030]

[0031] Among them, W Q W K W V These represent the query, key, and value, respectively; F is the facial feature vector; P is the physiological feature vector; S is the speech feature vector; and d is the hidden dimension parameter.

[0032] Then, the intermodal correlation is calculated using the cross-attention matrix to generate a joint feature table;

[0033] Generate cross-modal queries and keys using the following expression:

[0034]

[0035] The cross-attention score matrix is ​​calculated using the following expression:

[0036]

[0037] The normalized correlation degree is expressed as:

[0038]

[0039] in, is the query projection parameter matrix from face to speech; corr(F,S) is the scalar value of the correlation between face and speech modalities; h is the hidden dimension of the cross-modal interaction space;

[0040] Next, a two-channel neural network is constructed, with the first channel used for emotion classification, specifically:

[0041] Input joint feature representation Generated by fusing facial, physiological, and voice features;

[0042] The linear transformation of a fully connected layer is expressed as:

[0043] Z = W fc ·F joint +bfc

[0044] in, This is the weight matrix of the fully connected layer; For bias terms of fully connected layers;

[0045] The sentiment probability distribution is output using the Softmax function, expressed as follows:

[0046] P emotion =softmax(Z)

[0047] in, This represents a probability vector for seven types of emotions: natural, happy, sad, angry, disgusted, surprised, and terrified.

[0048] The second channel is used for personality correction, specifically:

[0049] Input personality weight vector

[0050] Through a pre-set personality-emotion association matrix The weights of different personality types' preferences for emotions are represented, and the correction coefficients are calculated using the following expression:

[0051]

[0052] Adjusting emotional confidence:

[0053] P final =P emotion ⊙α

[0054] Where ⊙ represents element-wise multiplication; The final revised sentiment confidence score is used to generate air conditioning adjustment instructions.

[0055] Preferably, step 4 includes:

[0056] A personality trait model is constructed using a neural network decision tree. Data training is performed based on contrastive learning using encrypted similarity. Encrypted multimodal features are input, and the probability distributions of five personality dimensions are output, specifically:

[0057] The model input is encrypted multimodal features. Wherein: F multi represents the unencrypted joint feature vector of face, physiological and speech features; E(·) denotes the encryption operation based on Paillier homomorphic encryption;

[0058] The model structure uses a hybrid neural network decision tree, as shown in the following expression:

[0059] Decision tree path selection:

[0060] G = ReLU(W gate ·E(F muiti )+b gate )

[0061] in, This is the gate weight matrix; Let m be the path selection probability, and m be the number of branches in the decision tree; b gate The gated bias term; RELU() is the activation function;

[0062] Leaf node prediction:

[0063]

[0064] in, Let be the weight matrix of the i-th leaf node; The probability distribution of the five personality dimensions is as follows: Extraversion, Conscientiousness, Openness, Agreeableness, and Neuroticism; G i The probability of path selection for leaf node i; For the bias term of leaf node i;

[0065] After a universal personality model is generated in the cloud, it is transmitted to the edge node via a secure TEE channel.

[0066] Edge nodes periodically upload encrypted local optimization logs, and the cloud performs federated averaging to update the global personality model. When a user's personality-emotion combination deviates from the cluster center, model retraining is triggered.

[0067] The specific process of data training is as follows:

[0068] Initialize global personality model parameters Θ in the cloud golbal ;

[0069] Edge nodes download the initial model and locally store the encrypted user data E(F). multi ) and the personality tag Y personality ;

[0070] Optimize the local model based on the contrastive loss function:

[0071]

[0072] Where SimCLR is the contrastive learning loss, which enhances the ciphertext similarity of similar personality traits; λ is the balancing hyperparameter; N is the number of edge nodes participating in federated learning; Y i For tags; Let F' be the probability distribution of the five personality dimensions of leaf node i; multi This consists of unencrypted facial, physiological, and voice combined features;

[0073] Edge nodes upload encrypted model gradients E(ΔΘ) at preset intervals. local );

[0074] Gradients are aggregated in the cloud and the global model is updated.

[0075]

[0076] Where D(·) represents the Paillier decryption operation; K represents the number of edge nodes participating in the aggregation; η represents the learning rate; and k represents the index variable. Let be the global personality model parameters for the t-th iteration;

[0077] Calculate the Mahalanobis distance between the user's personality-emotion combination and the cluster center:

[0078]

[0079] Where x is the user feature vector; μ,Σ are the cluster center mean and covariance matrix;

[0080] If D Mahalanobis If the value is greater than 2σ, then the cloud will be triggered to retrain the user's model, where σ is the standard deviation.

[0081] The in-vehicle emotion computing task allocation system based on edge-cloud collaboration provided by the present invention includes:

[0082] Module M1: Collects facial images, voice data, and physiological indicators of drivers and passengers through the vehicle terminal, extracts key features, and preprocesses them;

[0083] Module M2: Transmits data to edge nodes. If historical personality profiles exist, it assigns computing tasks and calls module M3. If no historical personality profiles exist, it uses the Paillier homomorphic encryption algorithm to homomorphically encrypt the data before uploading it to the cloud server and calls module M4.

[0084] Module M3: Edge nodes analyze the emotional state based on key features of facial images, voice data, and physiological indicators, and generate dynamic commands for the air conditioner based on the different emotional states of the user's personality.

[0085] Module M4: The cloud server completes data training in encrypted state to obtain a personality characteristic model, and transmits the personality characteristics analyzed by the model back to the edge node to optimize the air conditioning parameters under different personality-emotion combinations.

[0086] Preferably, the module M1 includes:

[0087] Extract key points 61 and 291 from the MediaPipe in the facial image to calculate the curvature of the mouth, and key points 33, 133, 362, and 263 to analyze the eye features;

[0088] Heart rate, respiratory rate, and blood pressure were extracted using a remote photoelectric pulse wave recording method.

[0089] Extract the MFCC features of the speech and delete the original recording data locally.

[0090] Preferably, the module M2 includes:

[0091] Based on bandwidth, latency, and packet loss rate, the network status is divided into three levels: excellent, medium, and poor. When the network status is excellent, all tasks are allowed to be processed in the cloud. When the network status is medium, only non-sensitive tasks are processed in the cloud. When the network status is poor, cloud communication is prohibited, and all tasks are forced to be processed at the edge.

[0092] For multimodal fusion and personality modeling tasks, assignment to the cloud is only performed when the network condition is optimal;

[0093] For tasks such as voice emotion analysis, facial landmark detection, and physiological information analysis, processing is performed locally when the edge load is below a preset threshold; otherwise, the tasks are assigned to the cloud.

[0094] Facial data is encrypted using AES-256, voice data is encrypted using the SM2 national cryptographic algorithm, and physiological data is encrypted using CKKS homomorphic encryption.

[0095] Preferably, the module M3 includes:

[0096] Obtain historical personality profiles, deploy a multimodal fusion model at edge nodes, input key features of facial images, voice data and physiological indicators, and output the emotional state with the highest probability.

[0097] First, calculate the self-attention weights for facial, physiological, and speech modal features;

[0098] The facial feature expression is:

[0099]

[0100] The physiological characteristic expression is:

[0101]

[0102] The speech feature expression is:

[0103]

[0104] Among them, W Q W K W V These represent the query, key, and value, respectively; F is the facial feature vector; P is the physiological feature vector; S is the speech feature vector; and d is the hidden dimension parameter.

[0105] Then, the intermodal correlation is calculated using the cross-attention matrix to generate a joint feature table;

[0106] Generate cross-modal queries and keys using the following expression:

[0107]

[0108] The cross-attention score matrix is ​​calculated using the following expression:

[0109]

[0110] The normalized correlation degree is expressed as:

[0111]

[0112] in, is the query projection parameter matrix from face to speech; corr(F,S) is the scalar value of the correlation between face and speech modalities; h is the hidden dimension of the cross-modal interaction space;

[0113] Next, a two-channel neural network is constructed, with the first channel used for emotion classification, specifically:

[0114] Input joint feature representation Generated by fusing facial, physiological, and voice features;

[0115] The linear transformation of a fully connected layer is expressed as:

[0116] Z = W fc ·F joint +b fc

[0117] in, This is the weight matrix of the fully connected layer; For bias terms of fully connected layers;

[0118] The sentiment probability distribution is output using the Softmax function, expressed as follows:

[0119] P emotion =softmax(Z)

[0120] in, This represents a probability vector for seven types of emotions: natural, happy, sad, angry, disgusted, surprised, and terrified.

[0121] The second channel is used for personality correction, specifically:

[0122] Input personality weight vector

[0123] Through a pre-set personality-emotion association matrix The weights of different personality types' preferences for emotions are represented, and the correction coefficients are calculated using the following expression:

[0124]

[0125] Adjusting emotional confidence:

[0126] P final =P emotion ⊙α

[0127] Where ⊙ represents element-wise multiplication; The final revised sentiment confidence score is used to generate air conditioning adjustment instructions.

[0128] Preferably, the module M4 includes:

[0129] A personality trait model is constructed using a neural network decision tree. Data training is performed based on contrastive learning using encrypted similarity. Encrypted multimodal features are input, and the probability distributions of five personality dimensions are output, specifically:

[0130] The model input is encrypted multimodal features. Wherein: F multi represents the unencrypted joint feature vector of face, physiological and speech features; E(·) denotes the encryption operation based on Paillier homomorphic encryption;

[0131] The model structure uses a hybrid neural network decision tree, as shown in the following expression:

[0132] Decision tree path selection:

[0133] G = ReLU(W gate ·E(F muiti )+b gate )

[0134] in, This is the gate weight matrix; Let m be the path selection probability, and m be the number of branches in the decision tree; b gate The gated bias term; RELU() is the activation function;

[0135] Leaf node prediction:

[0136]

[0137] in, Let be the weight matrix of the i-th leaf node; The probability distribution of the five personality dimensions is as follows: Extraversion, Conscientiousness, Openness, Agreeableness, and Neuroticism; G i The probability of path selection for leaf node i; For the bias term of leaf node i;

[0138] After a universal personality model is generated in the cloud, it is transmitted to the edge node via a secure TEE channel.

[0139] Edge nodes periodically upload encrypted local optimization logs, and the cloud performs federated averaging to update the global personality model. When a user's personality-emotion combination deviates from the cluster center, model retraining is triggered.

[0140] The specific process of data training is as follows:

[0141] Initialize global personality model parameters Θ in the cloud golbal ;

[0142] Edge nodes download the initial model and locally store the encrypted user data E(F). multi ) and the personality tag Y personality ;

[0143] Optimize the local model based on the contrastive loss function:

[0144]

[0145] Where SimCLR is the contrastive learning loss, which enhances the ciphertext similarity of similar personality traits; λ is the balancing hyperparameter; N is the number of edge nodes participating in federated learning; Y i For tags; Let F' be the probability distribution of the five personality dimensions of leaf node i; multi This consists of unencrypted facial, physiological, and voice combined features;

[0146] Edge nodes upload encrypted model gradients E(ΔΘ) at preset intervals. local );

[0147] Gradients are aggregated in the cloud and the global model is updated.

[0148]

[0149] Where D(·) represents the Paillier decryption operation; K represents the number of edge nodes participating in the aggregation; η represents the learning rate; and k represents the index variable. Let be the global personality model parameters for the t-th iteration;

[0150] Calculate the Mahalanobis distance between the user's personality-emotion combination and the cluster center:

[0151]

[0152] Where x is the user feature vector; μ,Σ are the cluster center mean and covariance matrix;

[0153] If D Mahalanobis If the value is greater than 2σ, then the cloud will be triggered to retrain the user's model, where σ is the standard deviation.

[0154] Compared with the prior art, the present invention has the following beneficial effects:

[0155] This invention utilizes edge-cloud collaborative computing to achieve efficient, real-time, and secure allocation of in-vehicle emotion computing tasks. By analyzing data such as facial expressions, voice tone, and physiological indicators of drivers and passengers, it identifies their personality traits and emotional states, personalizing the user's needs for comfort and health, and achieving adaptive adjustment of the cabin environment to "a thousand faces for a thousand people". Attached Figure Description

[0156] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0157] Figure 1 A flowchart illustrating the data processing workflow for edge-cloud collaboration;

[0158] Figure 2 This diagram illustrates the rules for how the edge and cloud collaborate. Detailed Implementation

[0159] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0160] Example

[0161] like Figure 1 This invention provides a method for allocating in-vehicle emotion computing tasks based on edge-cloud collaboration, including:

[0162] Step 1: The vehicle terminal extracts key feature data through the camera and microphone; collects facial expressions and physiological indicators, and collects voice tone through the microphone;

[0163] The key feature data of the face image includes MediaPipe keypoints 61 and 291, used to calculate the curvature of the mouth, and MediaPipe keypoints 33, 133, 362, and 263, used to calculate the inner and outer corners of the left and right eyes.

[0164] Key features of voice data include emotional features such as voice and intonation;

[0165] Key physiological data include indicators such as heart rate and blood pressure.

[0166] Step 2: Collect and anonymize the data and transmit it to the edge node. If a historical personality profile exists, assign a calculation task and proceed to Step 3; if no historical personality profile exists, use the Paillier homomorphic encryption algorithm to homomorphically encrypt the data and upload it to the cloud, then proceed to Step 4.

[0167] Data other than non-critical features is anonymized. Specifically, facial data is encrypted using AES-256, voice data is encrypted using the SM2 national cryptographic algorithm, and physiological information is encrypted using a homomorphic encryption algorithm based on the CKKS scheme.

[0168] Step 3: The edge node analyzes the emotional state based on the feature values ​​of facial key points, physiological information and voice information, and generates dynamic commands for the air conditioner according to the different emotional states of the user's personality.

[0169] Step 4: The cloud server trains the personality trait model in encrypted form and transmits the analyzed personality traits back to the edge nodes to optimize the air conditioning parameters under different personality-emotion combinations.

[0170] Step 1 includes the following steps:

[0171] Step 1.1: Perform preprocessing on the collected data, such as noise reduction and compression;

[0172] Step 1.2: Extract key points from the facial images captured by the camera and quantify facial features;

[0173] Step 1.3: Extract facial images captured by the camera and extract physiological information (heart rate, respiratory rate, blood pressure, etc.) through deep analysis of remote photoelectric pulse wave recording method;

[0174] Step 1.4: Extract MFCC features from the speech;

[0175] Step 1.5: Delete immediately after local feature extraction and discard the original video recording to protect user privacy and security.

[0176] like Figure 2 Step 2 includes the following steps:

[0177] Step 2.1: Assign tasks based on network status and task complexity; the specific rules are as follows:

[0178] Step 2.1.1: Network status is "Excellent" (bandwidth ≥ 20Mbps, latency ≤ 50ms, packet loss rate < 1%): Allow cloud processing of all tasks;

[0179] Step 2.2.2: Network status is "Medium" (bandwidth 5-20Mbps, latency 50-100ms, packet loss rate 1%-5%): Only run non-sensitive tasks in the cloud;

[0180] Step 2.2.3: When the network status is "poor" (bandwidth < 5Mbps, latency > 100ms, packet loss rate ≥ 5%): Cloud communication is prohibited, and all tasks are forced to be processed at the edge.

[0181] Step 2.2.4: High-complexity tasks (multimodal fusion analysis - personality modeling): Only assigned to cloud processing when the network status is "excellent";

[0182] Step 2.2.5: Low-complexity tasks (voice emotion analysis, facial landmark detection, physiological information analysis): Process locally when the edge device load is ≤70%, otherwise distribute to the cloud.

[0183] Step 3 includes the following steps:

[0184] Step 3.1: Obtain the facial feature vector from Step 1.2;

[0185] Step 3.2: Obtain the physiological feature vector from Step 1.3;

[0186] Step 3.3: Obtain the speech feature vector from Step 1.4;

[0187] Step 3.4: Obtain historical personality profiles (Extroverted, Conscientious, Open, Agreeable, Neurotic);

[0188] Step 3.5: Input the feature vectors from Steps 3.1, 3.2, and 3.3 into the multimodal fusion model deployed at the edge nodes, and output the emotional state with the highest probability among the seven emotional states (natural, happy, sad, angry, disgusted, surprised, and terrified).

[0189] Deploying a multimodal fusion model specifically includes:

[0190] Step 3.5.1: Calculate the self-attention weights for facial, physiological, and speech modal features;

[0191] For each modality's features, self-attention weights are calculated in three steps:

[0192] 1. Generate queries, keys, values, and vectors.

[0193] 2. Obtain the weights through dot product and normalization.

[0194] 3. Feature fusion: Weighted summation of value vectors using weights.

[0195] The facial feature expression is:

[0196]

[0197] The physiological characteristic expression is:

[0198]

[0199] The speech feature expression is:

[0200]

[0201] Among them, W Q W K W V These represent the query, key, and value, respectively; F is the facial feature vector; P is the physiological feature vector; S is the speech feature vector; and d is the hidden dimension parameter.

[0202] Step 3.5.2: Calculate the intermodal correlation degree using the cross-attention matrix to generate a joint feature table;

[0203] Generate cross-modal queries and keys using the following expression:

[0204]

[0205] The cross-attention score matrix is ​​calculated using the following expression:

[0206]

[0207] The normalized correlation degree is expressed as:

[0208]

[0209] in, is the query projection parameter matrix from face to speech; corr(F,S) is the scalar value of the correlation between face and speech modalities; h is the hidden dimension of the cross-modal interaction space.

[0210] Step 3.5.3: Construct a dual-channel neural network. Channel 1: Joint feature representation → fully connected layer → emotion classification (softmax outputs 7 class probabilities); Channel 2: Personality weight vector × emotion probability matrix → personality correction coefficient (adjusts the final emotion confidence).

[0211] Channel 1 (Emotion Classification Channel):

[0212] 1. Input joint feature representation (d is the feature dimension, which is generated by fusing facial, physiological and voice features).

[0213] 2. The linear transformation of the fully connected layer is expressed as:

[0214] Z = W fc ·F joint +b fc

[0215] in, The weight matrix for the fully connected layer (mapped to 7 emotion categories); This is the bias term for the fully connected layer.

[0216] 3. Output the sentiment probability distribution using the Softmax function, the expression of which is:

[0217] P emotion =softmax(Z)

[0218] in, This represents a probability vector for seven types of emotions (natural, happy, sad, etc.).

[0219] Channel 2 (Personality Correction Channel):

[0220] 1. Input personality weight vector (Corresponding to the five major personality traits: extraversion, conscientiousness, openness, agreeableness, and neuroticism).

[0221] 2. Through a pre-set personality-emotion association matrix (This represents the weight of different personality traits' preferences for emotions), and the correction coefficient is calculated using the following expression:

[0222]

[0223] 3. Adjust emotional confidence:

[0224] P final =P emotion ⊙α

[0225] Where ⊙ represents element-wise multiplication; The final revised sentiment confidence score is used to generate air conditioning adjustment instructions.

[0226]

[0227] Step 3.6: Issue air conditioner modification instructions based on different personality-emotion combinations. The specific rules are shown in the table below:

[0228]

[0229]

[0230]

[0231]

[0232]

[0233] Step 4 includes the following steps:

[0234] Step 4.1: Construct a personality feature model using a neural network decision tree, train the data based on comparative learning of encrypted similarity, input encrypted multimodal features, and output the probability distribution of the five personality dimensions;

[0235] 1. Model input: Encrypted multimodal features Wherein: F multi =[E face ,F physio ,F voice ] represents the unencrypted joint feature vector of face, physiology, and speech; E(·) represents the encryption operation based on Paillier homomorphic encryption.

[0236] 2. Model Structure: A Hybrid Neural Decision Tree (HNDT) is used, with the following expression:

[0237] Decision tree path selection:

[0238] G = ReLU(W gate ·E(F muiti )+b gate )

[0239] in, This is the gate weight matrix; The path selection probability is denoted by m (where m is the number of branches in the decision tree).

[0240] Leaf node prediction:

[0241]

[0242] in, Let be the weight matrix of the i-th leaf node; The probability distribution of the five personality dimensions (extroversion, conscientiousness, openness, agreeableness, and neuroticism).

[0243] 3. Ciphertext Calculation:

[0244] Using the additive homomorphism of Paillier homomorphic encryption, the calculation of W·E(F)+b is completed in the ciphertext state;

[0245] Activation functions (such as ReLU and softmax) are executed after decryption to ensure that the model output is parseable.

[0246] Step 4.2: After generating a universal personality model in the cloud, it is transmitted to the edge node through a secure TEE channel;

[0247] Step 4.3: Continuous learning mechanism. Edge nodes periodically (every 24 hours) upload encrypted local optimization logs. The cloud performs federated averaging to update the global personality model. When a user's personality-emotion combination deviates from the cluster center by more than 2σ, model retraining is triggered.

[0248] The specific process of data training:

[0249] 1. Initialize training:

[0250] Initialize global personality model parameters Θ in the cloud golbal ;

[0251] Edge nodes download the initial model and locally store the encrypted user data E(F). multi ) and the personality tag Y personality (Implicitly labeled through user history behavior).

[0252] 2. Local training (edge ​​nodes):

[0253] Optimize the local model based on the contrastive loss function:

[0254]

[0255] SimCLR is a contrastive learning loss that enhances the ciphertext similarity of personality traits of the same type; λ is a balancing hyperparameter.

[0256] 3. Federal Average (Cloud):

[0257] Edge nodes upload encrypted model gradients E(ΔΘ) every 24 hours. local );

[0258] Gradients are aggregated in the cloud and the global model is updated.

[0259]

[0260] Where D(·) is the Paillier decryption operation; K is the number of edge nodes participating in the aggregation; and η is the learning rate.

[0261] 4. Retraining trigger mechanism:

[0262] Calculate the Mahalanobis distance between the user's personality-emotion combination and the cluster center:

[0263]

[0264] Where x is the user feature vector; μ,Σ are the cluster center mean and covariance matrix.

[0265] If D Mahalanobis >2σ triggers cloud-based model retraining for a specific user.

[0266]

[0267] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0268] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for allocating in-vehicle emotion computing tasks based on edge-cloud collaboration, characterized in that, include: Step 1: Collect facial images, voice data, and physiological indicators of drivers and passengers through the vehicle terminal, extract key features, and preprocess them; Step 2: Transmit the data to the edge node. If a historical personality profile exists, assign a calculation task and proceed to Step 3. If no historical personality profile exists, the data is homomorphically encrypted using the Paillier homomorphic encryption algorithm and then uploaded to the cloud server. Proceed to step 4. Step 3: The edge nodes analyze the emotional state based on the key features of facial images, voice data and physiological indicators, and generate dynamic commands for the air conditioner according to the different emotional states of the user's personality. Step 4: The cloud server completes data training in encrypted state to obtain a personality characteristic model, and transmits the personality characteristics analyzed by the model back to the edge node to optimize the air conditioning parameters under different personality-emotion combinations; Step 2 includes: Based on bandwidth, latency, and packet loss rate, the network status is divided into three levels: excellent, medium, and poor. When the network status is excellent, all tasks are allowed to be processed in the cloud. When the network status is medium, only non-sensitive tasks are processed in the cloud. When the network status is poor, cloud communication is prohibited, and all tasks are forced to be processed at the edge. For multimodal fusion and personality modeling tasks, assignment to the cloud is only performed when the network condition is optimal; For tasks such as voice emotion analysis, facial landmark detection, and physiological information analysis, processing is performed locally when the edge load is below a preset threshold; otherwise, the tasks are assigned to the cloud. Facial data is encrypted using AES-256, voice data is encrypted using the SM2 national cryptographic algorithm, and physiological data is encrypted using CKKS homomorphic encryption. Step 3 includes: Obtain historical personality profiles, deploy a multimodal fusion model at edge nodes, input key features of facial images, voice data and physiological indicators, and output the emotional state with the highest probability. First, calculate the self-attention weights for facial, physiological, and speech modal features; The facial feature expression is: The physiological characteristic expression is: The speech feature expression is: in, , , These are respectively query, key, and value; For facial feature vectors; For physiological feature vectors; For speech feature vectors; To hide dimension parameters; Then, the intermodal correlation is calculated using the cross-attention matrix to generate a joint feature table; Generate cross-modal queries and keys using the following expression: The cross-attention score matrix is ​​calculated using the following expression: The normalized correlation degree is expressed as: in, The query projection parameter matrix from face to speech; This represents a scalar value indicating the correlation between facial and speech modalities. The hidden dimension of the cross-modal interaction space; Next, a two-channel neural network is constructed, with the first channel used for emotion classification, specifically: Input joint feature representation It is generated by fusing facial, physiological, and voice features; The linear transformation of a fully connected layer is expressed as: in, This is the weight matrix of the fully connected layer; For bias terms of fully connected layers; The sentiment probability distribution is output using the Softmax function, expressed as follows: in, This represents a probability vector for seven types of emotions: natural, happy, sad, angry, disgusted, surprised, and terrified. The second channel is used for personality correction, specifically: Input personality weight vector ; Through a pre-set personality-emotion association matrix The weights of different personality types' preferences for emotions are represented, and the correction coefficients are calculated using the following expression: Adjusting emotional confidence: Where ⊙ represents element-wise multiplication; The final revised sentiment confidence score is used to generate air conditioning adjustment instructions.

2. The method for allocating in-vehicle emotion computing tasks based on edge-cloud collaboration according to claim 1, characterized in that, Step 1 includes: Extract key points 61 and 291 from the MediaPipe in the facial image to calculate the curvature of the mouth, and key points 33, 133, 362, and 263 to analyze the eye features; Heart rate, respiratory rate, and blood pressure were extracted using a remote photoelectric pulse wave recording method. Extract the MFCC features of the speech and delete the original recording data locally.

3. The in-vehicle emotion computing task allocation method based on edge-cloud collaboration according to claim 1, characterized in that, Step 4 includes: A personality trait model is constructed using a neural network decision tree. Data training is performed based on contrastive learning using encrypted similarity. Encrypted multimodal features are input, and the probability distributions of five personality dimensions are output, specifically: The model input is encrypted multimodal features. ,in: This is an unencrypted joint feature vector of facial, physiological, and speech features; This indicates an encryption operation based on Paillier homomorphic encryption; The model structure uses a hybrid neural network decision tree, as shown in the following expression: Decision tree path selection: in, This is the gate weight matrix; Let m be the path selection probability, and m be the number of branches in the decision tree; This is a gated bias term; For activation functions; Leaf node prediction: in, For the first The weight matrix of the leaf nodes; The probability distribution of the five personality dimensions is as follows: Extraversion, Conscientiousness, Openness, Agreeableness, and Neuroticism. The probability of path selection for leaf node i; For the bias term of leaf node i; After a universal personality model is generated in the cloud, it is transmitted to the edge node via a secure TEE channel. Edge nodes periodically upload encrypted local optimization logs, and the cloud performs federated averaging to update the global personality model. When a user's personality-emotion combination deviates from the cluster center, model retraining is triggered. The specific process of data training is as follows: Initialize global personality model parameters in the cloud ; Edge nodes download the initial model and store encrypted user data locally. and personality tags ; Optimize the local model based on the contrastive loss function: in, To compare learning loss and enhance the ciphertext similarity of similar personality traits; To balance the hyperparameters; N is the number of edge nodes participating in federated learning; For tags; Let i be the probability distribution of the five personality dimensions of leaf node i. This consists of unencrypted facial, physiological, and voice combined features; Edge nodes upload encrypted model gradients at preset intervals. ; Gradients are aggregated in the cloud and the global model is updated. in, Decryption operation for Paillier; The number of edge nodes participating in the aggregation; k is the learning rate; k is the index variable; Let be the global personality model parameters for the t-th iteration; Calculate the Mahalanobis distance between the user's personality-emotion combination and the cluster center: in, User feature vector; The cluster center mean and covariance matrix; like This will trigger the cloud to retrain the user's model. The standard deviation is denoted as .

4. A vehicle-mounted emotion computing task allocation system based on edge-cloud collaboration, characterized in that, include: Module M1: Collects facial images, voice data, and physiological indicators of drivers and passengers through the vehicle terminal, extracts key features, and preprocesses them; Module M2: Transmits data to edge nodes. If historical personality profiles exist, it assigns computing tasks and calls module M3. If no historical personality profiles exist, it uses the Paillier homomorphic encryption algorithm to homomorphically encrypt the data before uploading it to the cloud server and calls module M4. Module M3: Edge nodes analyze the emotional state based on key features of facial images, voice data, and physiological indicators, and generate dynamic commands for the air conditioner based on the different emotional states of the user's personality. Module M4: The cloud server completes data training in encrypted state to obtain a personality characteristic model, and transmits the personality characteristics analyzed by the model back to the edge node to optimize the air conditioning parameters under different personality-emotion combinations; The module M2 includes: Based on bandwidth, latency, and packet loss rate, the network status is divided into three levels: excellent, medium, and poor. When the network status is excellent, all tasks are allowed to be processed in the cloud. When the network status is medium, only non-sensitive tasks are processed in the cloud. When the network status is poor, cloud communication is prohibited, and all tasks are forced to be processed at the edge. For multimodal fusion and personality modeling tasks, assignment to the cloud is only performed when the network condition is optimal; For tasks such as voice emotion analysis, facial landmark detection, and physiological information analysis, processing is performed locally when the edge load is below a preset threshold; otherwise, the tasks are assigned to the cloud. Facial data is encrypted using AES-256, voice data is encrypted using the SM2 national cryptographic algorithm, and physiological data is encrypted using CKKS homomorphic encryption. The module M3 includes: Obtain historical personality profiles, deploy a multimodal fusion model at edge nodes, input key features of facial images, voice data and physiological indicators, and output the emotional state with the highest probability. First, calculate the self-attention weights for facial, physiological, and speech modal features; The facial feature expression is: The physiological characteristic expression is: The speech feature expression is: in, , , These are respectively query, key, and value; For facial feature vectors; For physiological feature vectors; For speech feature vectors; To hide dimension parameters; Then, the intermodal correlation is calculated using the cross-attention matrix to generate a joint feature table; Generate cross-modal queries and keys using the following expression: The cross-attention score matrix is ​​calculated using the following expression: The normalized correlation degree is expressed as: in, The query projection parameter matrix from face to speech; This represents a scalar value indicating the correlation between facial and speech modalities. The hidden dimension of the cross-modal interaction space; Next, a two-channel neural network is constructed, with the first channel used for emotion classification, specifically: Input joint feature representation It is generated by fusing facial, physiological, and voice features; The linear transformation of a fully connected layer is expressed as: in, This is the weight matrix of the fully connected layer; For bias terms of fully connected layers; The sentiment probability distribution is output using the Softmax function, expressed as follows: in, This represents a probability vector for seven types of emotions: natural, happy, sad, angry, disgusted, surprised, and terrified. The second channel is used for personality correction, specifically: Input personality weight vector ; Through a pre-set personality-emotion association matrix The weights of different personality types' preferences for emotions are represented, and the correction coefficients are calculated using the following expression: Adjusting emotional confidence: Where ⊙ represents element-wise multiplication; The final revised sentiment confidence score is used to generate air conditioning adjustment instructions.

5. The in-vehicle emotion computing task allocation system based on edge-cloud collaboration according to claim 4, characterized in that, The module M1 includes: Extract key points 61 and 291 from the MediaPipe in the facial image to calculate the curvature of the mouth, and key points 33, 133, 362, and 263 to analyze the eye features; Heart rate, respiratory rate, and blood pressure were extracted using a remote photoelectric pulse wave recording method. Extract the MFCC features of the speech and delete the original recording data locally.

6. The in-vehicle emotion computing task allocation system based on edge-cloud collaboration according to claim 4, characterized in that, The module M4 includes: A personality trait model is constructed using a neural network decision tree. Data training is performed based on contrastive learning using encrypted similarity. Encrypted multimodal features are input, and the probability distributions of five personality dimensions are output, specifically: The model input is encrypted multimodal features. ,in: This is an unencrypted joint feature vector of facial, physiological, and speech features; This indicates an encryption operation based on Paillier homomorphic encryption; The model structure uses a hybrid neural network decision tree, as shown in the following expression: Decision tree path selection: in, This is the gate weight matrix; Let m be the path selection probability, and m be the number of branches in the decision tree; This is a gated bias term; For activation functions; Leaf node prediction: in, For the first The weight matrix of the leaf nodes; The probability distribution of the five personality dimensions is as follows: Extraversion, Conscientiousness, Openness, Agreeableness, and Neuroticism. The probability of path selection for leaf node i; For the bias term of leaf node i; After a universal personality model is generated in the cloud, it is transmitted to the edge node via a secure TEE channel. Edge nodes periodically upload encrypted local optimization logs, and the cloud performs federated averaging to update the global personality model. When a user's personality-emotion combination deviates from the cluster center, model retraining is triggered. The specific process of data training is as follows: Initialize global personality model parameters in the cloud ; Edge nodes download the initial model and store encrypted user data locally. and personality tags ; Optimize the local model based on the contrastive loss function: in, To compare learning loss and enhance the ciphertext similarity of similar personality traits; To balance the hyperparameters; N is the number of edge nodes participating in federated learning; For tags; Let i be the probability distribution of the five personality dimensions of leaf node i. This consists of unencrypted facial, physiological, and voice combined features; Edge nodes upload encrypted model gradients at preset intervals. ; Gradients are aggregated in the cloud and the global model is updated. in, Decryption operation for Paillier; The number of edge nodes participating in the aggregation; k is the learning rate; k is the index variable; Let be the global personality model parameters for the t-th iteration; Calculate the Mahalanobis distance between the user's personality-emotion combination and the cluster center: in, User feature vector; The cluster center mean and covariance matrix; like This will trigger the cloud to retrain the user's model. The standard deviation is denoted as .

Images