Vehicle-mounted emotion calculation task allocation method and system based on edge-cloud collaboration

Abstract

The invention provides a vehicle-mounted emotion calculation task allocation method and system based on edge-cloud collaboration, and the method comprises the steps: 1, collecting the face image, voice data and physiological indexes of a driver and a passenger through a vehicle-mounted terminal, extracting key features, and carrying out the preprocessing; 2, transmitting the data to an edge node, if a historical personality file exists, allocating a calculation task, and skipping to a step 3; if the historical personality file does not exist, performing homomorphic encryption on the data and then uploading the data to a cloud server, and skipping to step 4; 3, the edge nodes analyze emotional states according to the face images, the voice data and the key features of the physiological indexes, and dynamic instructions of the air conditioner are generated according to different emotional states of the personality states of the user; and 4, the cloud server completes data training in a ciphertext state to obtain a personality feature model, personality features analyzed by the model are sent back to the edge nodes, and air conditioner parameters under different personality-emotion combinations are optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of computing task allocation, and in particular to a method and system for allocating in-vehicle emotion computing tasks based on edge-cloud collaboration. Background Art

[0002] Traditional car air conditioners are adjusted based on physical parameters such as temperature and humidity, and are unable to perceive the driver's emotional state and personalized needs, resulting in insufficient comfort.

[0003] Existing emotion computing technology has the following shortcomings:

[0004] Cloud-based centralized processing: Emotion recognition solutions that rely on cloud servers cannot meet real-time requirements due to network latency, and uploading sensitive data (such as facial images) poses privacy risks.

[0005] On-board terminal processing: Due to the limitations of the computing power of the on-board terminal, it is unable to handle complex multimodal fusion analysis (such as face + voice + personality traits).

[0006] Patent application CN118906749A discloses a dynamic vehicle air conditioning control method and device, comprising: obtaining external and internal environmental parameters; the external environmental parameters including at least the current time and weather information at the vehicle's location, and the internal environmental parameters including at least the internal temperature and the status of the vehicle's occupants; determining the operating status of the vehicle air conditioner based on the external and internal environmental parameters; collecting user daily behavior data; obtaining a real-time dynamic air conditioning control strategy based on the external and internal environmental parameters, and the user daily behavior data; and correcting the operating status of the vehicle air conditioner based on the real-time dynamic air conditioning control strategy. However, this patent fails to fully resolve existing technical issues and fails to meet the requirements of the present invention. Summary of the Invention

[0007] In view of the defects in the prior art, the purpose of the present invention is to provide a method and system for allocating in-vehicle emotion computing tasks based on edge-cloud collaboration.

[0008] The method for allocating in-vehicle emotion computing tasks based on edge-cloud collaboration provided by the present invention includes:

[0009] Step 1: Collect facial images, voice data, and physiological indicators of the driver and passengers through the vehicle terminal, extract key features, and pre-process them;

[0010] Step 2: Transmit the data to the edge node. If a historical personality profile exists, assign the computing task and jump to step 3. If no historical personality profile exists, use the Paillier homomorphic encryption algorithm to homomorphically encrypt the data and upload it to the cloud server, then jump to step 4.

[0011] Step 3: The edge node analyzes the emotional state based on the key features of facial images, voice data, and physiological indicators, and generates dynamic instructions for the air conditioner based on the different emotional states of the user's personality;

[0012] Step 4: The cloud server completes data training in an encrypted state, obtains a personality trait model, transmits the personality traits analyzed by the model back to the edge node, and optimizes the conditioning parameters under different personality-emotion combinations.

[0013] Preferably, the step 1 comprises:

[0014] Extract MediaPipe key points 61 and 291 from the facial image to calculate the curvature of the mouth, and key points 33, 133, 362, and 263 to analyze eye features;

[0015] Extract heart rate, respiratory rate and blood pressure by remote photoelectric pulse wave recording method;

[0016] Extract the MFCC features of the speech and delete the original recorded data locally.

[0017] Preferably, the step 2 comprises:

[0018] Based on bandwidth, latency, and packet loss rate, the network status is divided into three levels: excellent, medium, and poor. When the network status is excellent, the cloud is allowed to process all tasks; when the network status is medium, only non-sensitive tasks are processed on the cloud; when the network status is poor, cloud communication is prohibited and all tasks are forced to be processed on the edge.

[0019] For multimodal fusion and personality modeling tasks, they are only assigned to the cloud when the network status is good;

[0020] For voice emotion analysis, facial key point detection, and physiological information analysis tasks, they are processed locally when the edge load is below a preset threshold, and otherwise distributed to the cloud;

[0021] Facial data is encrypted using AES-256, voice data is encrypted using the SM2 national secret algorithm, and physiological data is encrypted using CKKS homomorphic encryption.

[0022] Preferably, the step 3 includes:

[0023] Obtain historical personality profiles, deploy a multimodal fusion model on edge nodes, input key features of facial images, voice data, and physiological indicators, and output the most likely emotional state;

[0024] First, calculate the self-attention weights of facial, physiological, and speech modal features;

[0025] The facial feature expression is:

[0026]

[0027] The physiological characteristic expression is:

[0028]

[0029] The speech feature expression is:

[0030]

[0031] Among them, W Q 、W K 、W V are query, key, and value respectively; F is the facial feature vector; P is the physiological feature vector; S is the speech feature vector; d is the hidden dimension parameter;

[0032] Then, the correlation between modalities is calculated through the cross attention matrix to generate a joint feature table;

[0033] Generate cross-modal query and key, the expression is:

[0034]

[0035] Calculate the cross attention score matrix, the expression is:

[0036]

[0037] Normalized correlation, expression is:

[0038]

[0039] in, is the query projection parameter matrix from face to speech; corr(F,S) is the scalar value of the correlation between face and speech modalities; h is the hidden dimension of the cross-modal interaction space;

[0040] Then we build a two-channel neural network. The first channel is used for emotion classification. Specifically:

[0041] Input joint feature representation Generated by the fusion of facial, physiological and voice features;

[0042] The linear transformation of the fully connected layer is expressed as:

[0043] Z=W fc ·F joint +bfc

[0044] in, is the weight matrix of the fully connected layer; is the bias term of the fully connected layer;

[0045] The emotion probability distribution is output through the Softmax function, and the expression is:

[0046] P emotion =softmax(Z)

[0047] in, A probability vector representing seven categories of emotions: natural, happy, sad, angry, disgusted, surprised, and frightened;

[0048] The second channel is used for personality modification, specifically:

[0049] Input personality weight vector

[0050] Through the preset personality-emotion correlation matrix Indicates the preference weights of different personalities for emotions and calculates the correction coefficient. The expression is:

[0051]

[0052] Adjust emotional confidence:

[0053] P final =P emotion ⊙α

[0054] Among them, ⊙ represents element-by-element multiplication; is the final corrected emotion confidence, which is used to generate air conditioning adjustment instructions.

[0055] Preferably, step 4 includes:

[0056] A personality trait model is constructed using a neural network decision tree. Data training is performed based on contrastive learning of ciphertext similarity. Encrypted multimodal features are input and the probability distribution of the five personality dimensions is output. Specifically:

[0057] Model input is encrypted multimodal features Among them: F multi is the unencrypted joint feature vector of face, physiological and voice; E(·) represents the encryption operation based on Paillier homomorphic encryption;

[0058] The model structure adopts a hybrid neural network decision tree, and the expression is as follows:

[0059] Decision tree path selection:

[0060] G=ReLU(W gate ·E(F muiti )+b gate )

[0061] in, is the gating weight matrix; is the path selection probability, m is the number of decision tree branches; b gate is the gate bias term; RELU() is the activation function;

[0062] Leaf node prediction:

[0063]

[0064] in, is the weight matrix of the i-th leaf node; is the probability distribution of the five personality dimensions, which are extraversion, conscientiousness, openness, agreeableness, and neuroticism; G i is the path selection probability of leaf node i; is the bias term of leaf node i;

[0065] After the general personality model is generated in the cloud, it is transmitted to the edge node through the TEE secure channel;

[0066] Edge nodes regularly upload encrypted local optimization logs, and the cloud performs federated averaging to update the global personality model. When a user's personality-emotion combination deviates from the cluster center, model retraining is triggered.

[0067] The specific process of data training is as follows:

[0068] Initialize global personality model parameters Θ on the cloud golbal ;

[0069] The edge node downloads the initial model and stores the user encrypted data E(F multi ) and marked with personality label Y personality ;

[0070] Optimize the local model based on the contrastive loss function:

[0071]

[0072] Among them, SimCLR is the contrastive learning loss, which enhances the ciphertext similarity of similar personality traits; λ is the balance hyperparameter; N is the number of edge nodes participating in federated learning; Y i For labels; is the probability distribution of the five personality dimensions of leaf node i; F' multi Unencrypted facial, physiological, and voice joint features;

[0073] The edge node uploads the encrypted model gradient E(ΔΘ) at a preset time interval local );

[0074] The cloud aggregates gradients and updates the global model;

[0075]

[0076] Where D(·) is the Paillier decryption operation; K is the number of edge nodes participating in the aggregation; η is the learning rate; k is the index variable; is the global personality model parameter of the tth iteration;

[0077] Calculate the Mahalanobis distance between the user's personality-emotion combination and the cluster center:

[0078]

[0079] Where x is the user feature vector; μ, Σ are the cluster center mean and covariance matrix;

[0080] If D Mahalanobis >2σ, it triggers the cloud to retrain the user's model, where σ is the standard deviation.

[0081] The in-vehicle emotion computing task allocation system based on edge-cloud collaboration provided by the present invention includes:

[0082] Module M1: collects facial images, voice data, and physiological indicators of drivers and passengers through the vehicle terminal, extracts key features, and pre-processes them;

[0083] Module M2: Transmits the data to the edge node. If a historical personality profile exists, it assigns a computing task and calls module M3. If no historical personality profile exists, it homomorphically encrypts the data using the Paillier homomorphic encryption algorithm and uploads it to the cloud server, calling module M4.

[0084] Module M3: The edge node analyzes the emotional state based on the key features of facial images, voice data, and physiological indicators, and generates dynamic instructions for the air conditioner based on the different emotional states of the user's personality;

[0085] Module M4: The cloud server completes data training in an encrypted state, obtains a personality trait model, transmits the personality traits analyzed by the model back to the edge node, and optimizes the conditioning parameters under different personality-emotion combinations.

[0086] Preferably, the module M1 includes:

[0087] Extract MediaPipe key points 61 and 291 from the facial image to calculate the curvature of the mouth, and key points 33, 133, 362, and 263 to analyze eye features;

[0088] Extract heart rate, respiratory rate and blood pressure by remote photoelectric pulse wave recording method;

[0089] Extract the MFCC features of the speech and delete the original recorded data locally.

[0090] Preferably, the module M2 includes:

[0091] Based on bandwidth, latency, and packet loss rate, the network status is divided into three levels: excellent, medium, and poor. When the network status is excellent, the cloud is allowed to process all tasks; when the network status is medium, only non-sensitive tasks are processed on the cloud; when the network status is poor, cloud communication is prohibited and all tasks are forced to be processed on the edge.

[0092] For multimodal fusion and personality modeling tasks, they are only assigned to the cloud when the network status is good;

[0093] For voice emotion analysis, facial key point detection, and physiological information analysis tasks, they are processed locally when the edge load is below a preset threshold, and otherwise distributed to the cloud;

[0094] Facial data is encrypted using AES-256, voice data is encrypted using the SM2 national secret algorithm, and physiological data is encrypted using CKKS homomorphic encryption.

[0095] Preferably, the module M3 includes:

[0096] Obtain historical personality profiles, deploy a multimodal fusion model on edge nodes, input key features of facial images, voice data, and physiological indicators, and output the most likely emotional state;

[0097] First, calculate the self-attention weights of facial, physiological, and speech modal features;

[0098] The facial feature expression is:

[0099]

[0100] The physiological characteristic expression is:

[0101]

[0102] The speech feature expression is:

[0103]

[0104] Among them, W Q 、W K 、W V are query, key, and value respectively; F is the facial feature vector; P is the physiological feature vector; S is the speech feature vector; d is the hidden dimension parameter;

[0105] Then, the correlation between modalities is calculated through the cross attention matrix to generate a joint feature table;

[0106] Generate cross-modal query and key, the expression is:

[0107]

[0108] Calculate the cross attention score matrix, the expression is:

[0109]

[0110] Normalized correlation, expression is:

[0111]

[0112] in, is the query projection parameter matrix from face to speech; corr(F,S) is the scalar value of the correlation between face and speech modalities; h is the hidden dimension of the cross-modal interaction space;

[0113] Then we build a two-channel neural network. The first channel is used for emotion classification. Specifically:

[0114] Input joint feature representation Generated by the fusion of facial, physiological and voice features;

[0115] The linear transformation of the fully connected layer is expressed as:

[0116] Z=W fc ·F joint +b fc

[0117] in, is the weight matrix of the fully connected layer; is the bias term of the fully connected layer;

[0118] The emotion probability distribution is output through the Softmax function, and the expression is:

[0119] P emotion =softmax(Z)

[0120] in, A probability vector representing seven categories of emotions: natural, happy, sad, angry, disgusted, surprised, and frightened;

[0121] The second channel is used for personality modification, specifically:

[0122] Input personality weight vector

[0123] Through the preset personality-emotion correlation matrix Indicates the preference weights of different personalities for emotions and calculates the correction coefficient. The expression is:

[0124]

[0125] Adjust emotional confidence:

[0126] P final =P emotion ⊙α

[0127] Among them, ⊙ represents element-by-element multiplication; is the final corrected emotion confidence, which is used to generate air conditioning adjustment instructions.

[0128] Preferably, the module M4 includes:

[0129] A personality trait model is constructed using a neural network decision tree. Data training is performed based on contrastive learning of ciphertext similarity. Encrypted multimodal features are input and the probability distribution of the five personality dimensions is output. Specifically:

[0130] Model input is encrypted multimodal features Among them: F multi is the unencrypted joint feature vector of face, physiological and voice; E(·) represents the encryption operation based on Paillier homomorphic encryption;

[0131] The model structure adopts a hybrid neural network decision tree, and the expression is as follows:

[0132] Decision tree path selection:

[0133] G=ReLU(W gate ·E(F muiti )+b gate )

[0134] in, is the gating weight matrix; is the path selection probability, m is the number of decision tree branches; b gate is the gate bias term; RELU() is the activation function;

[0135] Leaf node prediction:

[0136]

[0137] in, is the weight matrix of the i-th leaf node; is the probability distribution of the five personality dimensions, which are extraversion, conscientiousness, openness, agreeableness, and neuroticism; G i is the path selection probability of leaf node i; is the bias term of leaf node i;

[0138] After the general personality model is generated in the cloud, it is transmitted to the edge node through the TEE secure channel;

[0139] Edge nodes regularly upload encrypted local optimization logs, and the cloud performs federated averaging to update the global personality model. When a user's personality-emotion combination deviates from the cluster center, model retraining is triggered.

[0140] The specific process of data training is as follows:

[0141] Initialize global personality model parameters Θ on the cloud golbal ;

[0142] The edge node downloads the initial model and stores the user encrypted data E(F multi ) and marked with personality label Y personality ;

[0143] Optimize the local model based on the contrastive loss function:

[0144]

[0145] Among them, SimCLR is the contrastive learning loss, which enhances the ciphertext similarity of similar personality traits; λ is the balance hyperparameter; N is the number of edge nodes participating in federated learning; Y i For labels; is the probability distribution of the five personality dimensions of leaf node i; F' multi Unencrypted facial, physiological, and voice joint features;

[0146] The edge node uploads the encrypted model gradient E(ΔΘ) at a preset time interval local );

[0147] The cloud aggregates gradients and updates the global model;

[0148]

[0149] Where D(·) is the Paillier decryption operation; K is the number of edge nodes participating in the aggregation; η is the learning rate; k is the index variable; is the global personality model parameter of the tth iteration;

[0150] Calculate the Mahalanobis distance between the user's personality-emotion combination and the cluster center:

[0151]

[0152] Where x is the user feature vector; μ, Σ are the cluster center mean and covariance matrix;

[0153] If D Mahalanobis >2σ, it triggers the cloud to retrain the user's model, where σ is the standard deviation.

[0154] Compared with the prior art, the present invention has the following beneficial effects:

[0155] This invention uses edge-cloud collaborative computing to achieve efficient, real-time and secure allocation of in-vehicle emotion computing tasks. By analyzing the driver and passengers' facial expressions, voice intonation, physiological indicators and other data, it identifies their personality traits and emotional state, personalizes the user's needs for comfort and health, and realizes adaptive adjustment of the cabin environment to suit each person. BRIEF DESCRIPTION OF THE DRAWINGS

[0156] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0157] Figure 1 This is a flowchart of data processing for edge-cloud collaboration;

[0158] Figure 2 A diagram illustrating the rules for edge-cloud collaboration. DETAILED DESCRIPTION

[0159] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0160] Example

[0161] like Figure 1 The present invention provides a method for allocating vehicle-mounted emotion computing tasks based on edge-cloud collaboration, comprising:

[0162] Step 1: The vehicle terminal uses a camera and microphone to extract key feature data; collects facial expressions and physiological indicators, and uses a microphone to collect voice intonation;

[0163] The key feature data of the facial image includes MediaPipe key points No. 61 and No. 291, which are used to calculate the curvature of the mouth, and the inner and outer corners of the left and right eyes: MediaPipe key points No. 33, 133, 362, and 263.

[0164] The key feature data of speech data include voice, intonation and other emotional feature data;

[0165] The key data of physiological information are indicators such as heart rate and blood pressure.

[0166] Step 2: Collect the desensitized data and transmit it to the edge node. If there is a historical personality profile, assign the computing task and jump to step 3. If there is no historical personality profile, use the Paillier homomorphic encryption algorithm to homomorphically encrypt the data and upload it to the cloud, then jump to step 4.

[0167] Data other than non-critical feature data is desensitized. Facial data is encrypted with AES-256, voice data is encrypted with the SM2 national secret algorithm, and physiological information uses a homomorphic encryption algorithm based on the CKKS scheme.

[0168] Step 3: The edge node analyzes the emotional state based on the facial key points, physiological information, and voice information, and generates dynamic air conditioning instructions based on the different emotional states of the user's personality;

[0169] Step 4: The cloud server completes data training in an encrypted state to generate a personality trait model, and transmits the analyzed personality traits back to the edge node to optimize the conditioning parameters under different personality-emotion combinations.

[0170] The step 1 comprises the following steps:

[0171] Step 1.1: Pre-process the collected data by noise reduction, compression, etc.

[0172] Step 1.2: Extract key points from the facial image captured by the camera and digitize facial features;

[0173] Step 1.3: Extract the facial image captured by the camera and extract physiological information (heart rate, respiratory rate, blood pressure, etc.) through deep analysis remote photoelectric pulse wave recording method;

[0174] Step 1.4: Extract MFCC features of speech;

[0175] Step 1.5: After local features are extracted, delete them immediately and discard the original recording to protect user privacy.

[0176] like Figure 2 , the step 2 comprises the following steps:

[0177] Step 2.1: Assign tasks based on network status and task complexity. The specific rules are as follows:

[0178] Step 2.1.1: If the network status is "Excellent" (bandwidth ≥ 20 Mbps, latency ≤ 50 ms, packet loss rate < 1%): allow the cloud to process all tasks.

[0179] Step 2.2.2: If the network status is "medium" (bandwidth 5-20 Mbps, latency 50-100 ms, packet loss rate 1%-5%): Only run non-sensitive tasks in the cloud.

[0180] Step 2.2.3: If the network status is "poor" (bandwidth < 5Mbps, latency > 100ms, packet loss rate ≥ 5%), cloud communication is prohibited and all tasks are forced to be processed at the edge.

[0181] Step 2.2.4: High-complexity tasks (multimodal fusion analysis-personality modeling): Assign them to the cloud for processing only when the network status is "excellent";

[0182] Step 2.2.5: Low-complexity tasks (speech emotion analysis, facial key point detection, physiological information analysis): Process locally when the edge device load is ≤70%, otherwise distribute to the cloud.

[0183] The step 3 comprises the following steps:

[0184] Step 3.1: Get the facial feature vector from step 1.2;

[0185] Step 3.2: Obtain the physiological feature vector from step 1.3;

[0186] Step 3.3: Get the speech feature vector from step 1.4;

[0187] Step 3.4: Obtain historical personality profile (extraversion, conscientiousness, openness, agreeableness, neuroticism);

[0188] Step 3.5: The multimodal fusion model deployed at the edge node inputs the feature vectors of steps 3.1, 3.2, and 3.3, and outputs the emotional state with the highest probability among the seven emotional states (natural, happy, sad, angry, disgusted, surprised, and frightened);

[0189] Deploying a multimodal fusion model specifically includes:

[0190] Step 3.5.1: Calculate the self-attention weights of facial, physiological, and speech features.

[0191] For each modality feature, self-attention weighting is completed in three steps:

[0192] 1. Generate query, key, value, vector.

[0193] 2. Obtain weights through dot product and normalization.

[0194] 3. Feature fusion: weighted sum of value vectors using weights.

[0195] The facial feature expression is:

[0196]

[0197] The physiological characteristic expression is:

[0198]

[0199] The speech feature expression is:

[0200]

[0201] Among them, W Q 、W K 、W V are query, key, and value respectively; F is the facial feature vector; P is the physiological feature vector; S is the speech feature vector; and d is the hidden dimension parameter.

[0202] Step 3.5.2: Calculate the inter-modal correlation through the cross-attention matrix and generate a joint feature table;

[0203] Generate cross-modal query and key, the expression is:

[0204]

[0205] Calculate the cross attention score matrix, the expression is:

[0206]

[0207] Normalized correlation, expression is:

[0208]

[0209] in, is the query projection parameter matrix from face to speech; corr(F,S) is the scalar value of the correlation between face and speech modalities; h is the hidden dimension of the cross-modal interaction space.

[0210] Step 3.5.3: Construct a two-channel neural network. Channel 1: Joint feature representation → Fully connected layer → Emotion classification (softmax outputs 7-category probabilities); Channel 2: Personality weight vector × emotion probability matrix → Personality correction coefficient (to adjust the final emotion confidence).

[0211] Channel 1 (emotion classification channel):

[0212] 1. Input joint feature representation (d is the feature dimension, generated by the fusion of facial, physiological and voice features).

[0213] 2. Linear transformation of the fully connected layer, the expression is:

[0214] Z=W fc ·F joint +b fc

[0215] in, is the weight matrix of the fully connected layer (mapped to 7 emotion categories); is the bias term of the fully connected layer.

[0216] 3. Output the emotion probability distribution through the Softmax function, the expression is:

[0217] P emotion =softmax(Z)

[0218] in, A probability vector representing 7 types of emotions (natural, happy, sad, etc.).

[0219] Channel 2 (Personality Correction Channel):

[0220] 1. Input personality weight vector (Corresponding to the five major personality traits: extraversion, conscientiousness, openness, agreeableness, and neuroticism).

[0221] 2. Through the preset personality-emotion correlation matrix Represents the preference weight of different personalities for emotions), calculate the correction coefficient, the expression is:

[0222]

[0223] 3. Adjust emotional confidence:

[0224] P final =P emotion ⊙α

[0225] Among them, ⊙ represents element-by-element multiplication; is the final corrected emotion confidence, which is used to generate air conditioning adjustment instructions.

[0226]

[0227] Step 3.6: Make conditioning modification instructions based on different personality-emotion combinations. The specific rules are shown in the following table:

[0228]

[0229]

[0230]

[0231]

[0232]

[0233] The step 4 comprises the following steps:

[0234] Step 4.1: Use a neural network decision tree to build a personality trait model, perform data training based on contrastive learning of ciphertext similarity, input encrypted multimodal features, and output the probability distribution of the five personality dimensions;

[0235] 1. Model input: encrypted multimodal features Among them: F multi =[E face ,F physio ,F voice ] is the unencrypted joint feature vector of face, physiological and voice; E(·) represents the encryption operation based on Paillier homomorphic encryption.

[0236] 2. Model structure: Hybrid Neural Decision Tree (HNDT) is used, and the expression is as follows:

[0237] Decision tree path selection:

[0238] G=ReLU(W gate ·E(F muiti )+b gate )

[0239] in, is the gating weight matrix; is the path selection probability (m is the number of decision tree branches).

[0240] Leaf node prediction:

[0241]

[0242] in, is the weight matrix of the i-th leaf node; is the probability distribution of the five personality dimensions (extraversion, conscientiousness, openness, agreeableness, and neuroticism).

[0243] 3. Ciphertext calculation:

[0244] Using the additive homomorphism of Paillier homomorphic encryption, the calculation of W·E(F)+b is completed in the ciphertext state;

[0245] Activation functions (such as ReLU and softmax) are performed after decryption to ensure that the model output is parseable.

[0246] Step 4.2: After the general personality model is generated in the cloud, it is transmitted to the edge node through the TEE secure channel;

[0247] Step 4.3: Continuous learning mechanism: edge nodes regularly upload encrypted local optimization logs (every 24 hours), and the cloud performs federated averaging to update the global personality model. When a user's personality-emotion combination deviates from the cluster center by >2σ, model retraining is triggered.

[0248] Specific process of data training:

[0249] 1. Initialize training:

[0250] Initialize global personality model parameters Θ on the cloud golbal ;

[0251] The edge node downloads the initial model and stores the user encrypted data E(F multi ) and marked with personality label Y personality (Implicitly marked through user historical behavior).

[0252] 2. Local training (edge nodes):

[0253] Optimize the local model based on the contrastive loss function:

[0254]

[0255] Among them, SimCLR is the contrastive learning loss, which enhances the ciphertext similarity of similar personality features; λ is the balancing hyperparameter.

[0256] 3. Federal average (cloud):

[0257] Edge nodes upload encrypted model gradients E(ΔΘ) every 24 hours local );

[0258] The cloud aggregates gradients and updates the global model;

[0259]

[0260] Where D(·) is the Paillier decryption operation; K is the number of edge nodes participating in the aggregation; and η is the learning rate.

[0261] 4. Retraining trigger mechanism:

[0262] Calculate the Mahalanobis distance between the user personality-emotion combination and the cluster center:

[0263]

[0264] Among them, x is the user feature vector; μ, Σ are the cluster center mean and covariance matrix.

[0265] If D Mahalanobis >2σ, triggering the cloud to retrain the model for a specific user.

[0266]

[0267] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0268] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A method for allocating in-vehicle emotion computing tasks based on edge-cloud collaboration, characterized in that: include: Step 1: Collect facial images, voice data, and physiological indicators of the driver and passengers through the vehicle terminal, extract key features, and pre-process them; Step 2: Transmit the data to the edge node. If a historical personality profile exists, assign the computing task and jump to step 3. If there is no historical personality profile, the data is homomorphically encrypted using the Paillier homomorphic encryption algorithm and uploaded to the cloud server, and then the process goes to step 4. Step 3: The edge node analyzes the emotional state based on the key features of facial images, voice data, and physiological indicators, and generates dynamic instructions for the air conditioner based on the different emotional states of the user's personality; Step 4: The cloud server completes data training in an encrypted state, obtains a personality trait model, transmits the personality traits analyzed by the model back to the edge node, and optimizes the conditioning parameters under different personality-emotion combinations.

2. The method for allocating vehicle-mounted emotion computing tasks based on edge-cloud collaboration according to claim 1 is characterized in that: The step 1 comprises: Extract MediaPipe key points 61 and 291 from the facial image to calculate the curvature of the mouth, and key points 33, 133, 362, and 263 to analyze eye features; Extract heart rate, respiratory rate and blood pressure by remote photoelectric pulse wave recording method; Extract the MFCC features of the speech and delete the original recorded data locally.

3. The method for allocating vehicle-mounted emotion computing tasks based on edge-cloud collaboration according to claim 1, characterized in that: The step 2 includes: Based on bandwidth, latency, and packet loss rate, the network status is divided into three levels: excellent, medium, and poor. When the network status is excellent, the cloud is allowed to process all tasks; when the network status is medium, only non-sensitive tasks are processed on the cloud; when the network status is poor, cloud communication is prohibited and all tasks are forced to be processed on the edge. For multimodal fusion and personality modeling tasks, they are only assigned to the cloud when the network status is good; For voice emotion analysis, facial key point detection, and physiological information analysis tasks, they are processed locally when the edge load is below a preset threshold, and otherwise distributed to the cloud; Facial data is encrypted using AES-256, voice data is encrypted using the SM2 national secret algorithm, and physiological data is encrypted using CKKS homomorphic encryption.

4. The method for allocating vehicle-mounted emotion computing tasks based on edge-cloud collaboration according to claim 1, characterized in that: The step 3 includes: Obtain historical personality profiles, deploy a multimodal fusion model on edge nodes, input key features of facial images, voice data, and physiological indicators, and output the most likely emotional state; First, calculate the self-attention weights of facial, physiological, and speech modal features; The facial feature expression is: The physiological characteristic expression is: The speech feature expression is: Among them, W Q 、W K 、W V are query, key, and value respectively; F is the facial feature vector; P is the physiological feature vector; S is the speech feature vector; d is the hidden dimension parameter; Then, the correlation between modalities is calculated through the cross attention matrix to generate a joint feature table; Generate cross-modal query and key, the expression is: Calculate the cross attention score matrix, the expression is: Normalized correlation, expression is: in, is the query projection parameter matrix from face to speech; corr(F,S) is the scalar value of the correlation between face and speech modalities; h is the hidden dimension of the cross-modal interaction space; Then we build a two-channel neural network. The first channel is used for emotion classification. Specifically: Input joint feature representation Generated by the fusion of facial, physiological and voice features; The linear transformation of the fully connected layer is expressed as: Z=W fc ·F joint +b fc in, is the weight matrix of the fully connected layer; is the bias term of the fully connected layer; The emotion probability distribution is output through the Softmax function, and the expression is: P emotion =softmax(Z) in, A probability vector representing seven categories of emotions: natural, happy, sad, angry, disgusted, surprised, and frightened; The second channel is used for personality modification, specifically: Input personality weight vector Through the preset personality-emotion correlation matrix Indicates the preference weights of different personalities for emotions and calculates the correction coefficient. The expression is: Adjust emotional confidence: P final =P emotion ⊙α Among them, ⊙ represents element-by-element multiplication; is the final corrected emotion confidence, which is used to generate air conditioning adjustment instructions.

5. The method for allocating vehicle-mounted emotion computing tasks based on edge-cloud collaboration according to claim 4 is characterized in that: The step 4 comprises: A personality trait model is constructed using a neural network decision tree. Data training is performed based on contrastive learning of ciphertext similarity. Encrypted multimodal features are input and the probability distribution of the five personality dimensions is output. Specifically: Model input is encrypted multimodal features Among them: F multi is the unencrypted joint feature vector of face, physiological and voice; E(·) represents the encryption operation based on Paillier homomorphic encryption; The model structure adopts a hybrid neural network decision tree, and the expression is as follows: Decision tree path selection: G=ReLU(W gate ·E(F muiti )+b gate ) in, is the gating weight matrix; is the path selection probability, m is the number of decision tree branches; b gate is the gate bias term; RELU() is the activation function; Leaf node prediction: in, is the weight matrix of the i-th leaf node; is the probability distribution of the five personality dimensions, which are extraversion, conscientiousness, openness, agreeableness, and neuroticism; G i is the path selection probability of leaf node i; is the bias term of leaf node i; After the general personality model is generated in the cloud, it is transmitted to the edge node through the TEE secure channel; Edge nodes regularly upload encrypted local optimization logs, and the cloud performs federated averaging to update the global personality model. When a user's personality-emotion combination deviates from the cluster center, model retraining is triggered. The specific process of data training is as follows: Initialize global personality model parameters Θ on the cloud golbal ; The edge node downloads the initial model and stores the user encrypted data E(F multi ) and marked with personality label Y personality ; Optimize the local model based on the contrastive loss function: Among them, SimCLR is the contrastive learning loss, which enhances the ciphertext similarity of similar personality traits; λ is the balance hyperparameter; N is the number of edge nodes participating in federated learning; Y i For labels; is the probability distribution of the five personality dimensions of leaf node i; F' multi Unencrypted facial, physiological, and voice joint features; The edge node uploads the encrypted model gradient E(ΔΘ) at a preset time interval local ); The cloud aggregates gradients and updates the global model; Where D(·) is the Paillier decryption operation; K is the number of edge nodes participating in the aggregation; η is the learning rate; k is the index variable; is the global personality model parameter of the tth iteration; Calculate the Mahalanobis distance between the user's personality-emotion combination and the cluster center: Where x is the user feature vector; μ, Σ are the cluster center mean and covariance matrix; If D Mahalanobis >2σ, it triggers the cloud to retrain the user's model, where σ is the standard deviation.

6. A vehicle-mounted emotion computing task allocation system based on edge-cloud collaboration, characterized in that: include: Module M1: collects facial images, voice data, and physiological indicators of drivers and passengers through the vehicle terminal, extracts key features, and pre-processes them; Module M2: Transmits the data to the edge node. If a historical personality profile exists, it assigns a computing task and calls module M3. If no historical personality profile exists, it homomorphically encrypts the data using the Paillier homomorphic encryption algorithm and uploads it to the cloud server, calling module M4. Module M3: The edge node analyzes the emotional state based on the key features of facial images, voice data, and physiological indicators, and generates dynamic instructions for the air conditioner based on the different emotional states of the user's personality; Module M4: The cloud server completes data training in an encrypted state, obtains a personality trait model, transmits the personality traits analyzed by the model back to the edge node, and optimizes the conditioning parameters under different personality-emotion combinations.

7. The vehicle-mounted emotion computing task allocation system based on edge-cloud collaboration according to claim 6 is characterized in that: The module M1 includes: Extract MediaPipe key points 61 and 291 from the facial image to calculate the curvature of the mouth, and key points 33, 133, 362, and 263 to analyze eye features; Extract heart rate, respiratory rate and blood pressure by remote photoelectric pulse wave recording method; Extract the MFCC features of the speech and delete the original recorded data locally.

8. The vehicle-mounted emotion computing task allocation system based on edge-cloud collaboration according to claim 6 is characterized in that: The module M2 includes: Based on bandwidth, latency, and packet loss rate, the network status is divided into three levels: excellent, medium, and poor. When the network status is excellent, the cloud is allowed to process all tasks; when the network status is medium, only non-sensitive tasks are processed on the cloud; when the network status is poor, cloud communication is prohibited and all tasks are forced to be processed on the edge. For multimodal fusion and personality modeling tasks, they are only assigned to the cloud when the network status is good; For voice emotion analysis, facial key point detection, and physiological information analysis tasks, they are processed locally when the edge load is below a preset threshold, and otherwise distributed to the cloud; Facial data is encrypted using AES-256, voice data is encrypted using the SM2 national secret algorithm, and physiological data is encrypted using CKKS homomorphic encryption.

9. The vehicle-mounted emotion computing task allocation system based on edge-cloud collaboration according to claim 6 is characterized in that: The module M3 includes: Obtain historical personality profiles, deploy a multimodal fusion model on edge nodes, input key features of facial images, voice data, and physiological indicators, and output the most likely emotional state; First, calculate the self-attention weights of facial, physiological, and speech modal features; The facial feature expression is: The physiological characteristic expression is: The speech feature expression is: Among them, W Q 、W K 、W V are query, key, and value respectively; F is the facial feature vector; P is the physiological feature vector; S is the speech feature vector; d is the hidden dimension parameter; Then, the correlation between modalities is calculated through the cross attention matrix to generate a joint feature table; Generate cross-modal query and key, the expression is: Calculate the cross attention score matrix, the expression is: Normalized correlation, expression is: in, is the query projection parameter matrix from face to speech; corr(F,S) is the scalar value of the correlation between face and speech modalities; h is the hidden dimension of the cross-modal interaction space; Then we build a two-channel neural network. The first channel is used for emotion classification. Specifically: Input joint feature representation Generated by the fusion of facial, physiological and voice features; The linear transformation of the fully connected layer is expressed as: Z=W fc ·F joint +b fc in, is the weight matrix of the fully connected layer; is the bias term of the fully connected layer; The emotion probability distribution is output through the Softmax function, and the expression is: P emotion =softmax(Z) in, A probability vector representing seven categories of emotions: natural, happy, sad, angry, disgusted, surprised, and frightened; The second channel is used for personality modification, specifically: Input personality weight vector Through the preset personality-emotion correlation matrix Indicates the preference weights of different personalities for emotions and calculates the correction coefficient. The expression is: Adjust emotional confidence: P final =P emotion ⊙α Among them, ⊙ represents element-by-element multiplication; is the final corrected emotion confidence, which is used to generate air conditioning adjustment instructions.

10. The vehicle-mounted emotion computing task allocation system based on edge-cloud collaboration according to claim 9 is characterized in that: The module M4 includes: A personality trait model is constructed using a neural network decision tree. Data training is performed based on contrastive learning of ciphertext similarity. Encrypted multimodal features are input and the probability distribution of the five personality dimensions is output. Specifically: Model input is encrypted multimodal features Among them: F multi is the unencrypted joint feature vector of face, physiological and voice; E(·) represents the encryption operation based on Paillier homomorphic encryption; The model structure adopts a hybrid neural network decision tree, and the expression is as follows: Decision tree path selection: G=ReLU(W gate ·E(F muiti )+b gate ) in, is the gating weight matrix; is the path selection probability, m is the number of decision tree branches; b gate is the gate bias term; RELU() is the activation function; Leaf node prediction: in, is the weight matrix of the i-th leaf node; is the probability distribution of the five personality dimensions, which are extraversion, conscientiousness, openness, agreeableness, and neuroticism; G i is the path selection probability of leaf node i; is the bias term of leaf node i; After the general personality model is generated in the cloud, it is transmitted to the edge node through the TEE secure channel; Edge nodes regularly upload encrypted local optimization logs, and the cloud performs federated averaging to update the global personality model. When a user's personality-emotion combination deviates from the cluster center, model retraining is triggered. The specific process of data training is as follows: Initialize global personality model parameters Θ on the cloud golbal ; The edge node downloads the initial model and stores the user encrypted data E(F multi ) and marked with personality label Y personality ; Optimize the local model based on the contrastive loss function: Among them, SimCLR is the contrastive learning loss, which enhances the ciphertext similarity of similar personality traits; λ is the balance hyperparameter; N is the number of edge nodes participating in federated learning; Y i For labels; is the probability distribution of the five personality dimensions of leaf node i; F' multi Unencrypted facial, physiological, and voice joint features; The edge node uploads the encrypted model gradient E(ΔΘ) at a preset time interval local ); The cloud aggregates gradients and updates the global model; Where D(·) is the Paillier decryption operation; K is the number of edge nodes participating in the aggregation; η is the learning rate; k is the index variable; is the global personality model parameter of the tth iteration; Calculate the Mahalanobis distance between the user's personality-emotion combination and the cluster center: Where x is the user feature vector; μ, Σ are the cluster center mean and covariance matrix; If D Mahalanobis >2σ, it triggers the cloud to retrain the user's model, where σ is the standard deviation.

Images