Driver emotion recognition model adaptive updating method and device, and electronic equipment

Abstract

The application provides a driver emotion recognition model adaptive updating method and device and electronic equipment, constructs texture feature vectors and geometric feature vectors based on face detection results of a driver face video frame sequence and carries out fusion, carries out emotion recognition through a personalized emotion recognition model of a vehicle end, carries out multi-dimensional credibility evaluation based on emotion recognition results, marks out difficult samples based on multi-dimensional credibility evaluation results and preset threshold information and iteratively updates parameters of the personalized emotion recognition model, desensitizes target data corresponding to the difficult samples, and sends the desensitized data and the updated parameters of the personalized emotion recognition model to the cloud end, so that the cloud end sends the global emotion recognition model after updating to the vehicle end. The application can improve the accuracy and stability of the vehicle end emotion recognition, and ensure the universality of the vehicle end for accurately recognizing the emotions of different drivers.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and in particular to an adaptive update method, apparatus, and electronic device for a driver emotion recognition model. Background Technology

[0002] With the development of intelligent cockpits, in-vehicle emotion recognition has gradually become a key aspect of driving safety monitoring. However, existing technologies still have the following shortcomings: 1) Most emotion recognition models perform well in recognizing positive emotions, but are weak in recognizing negative emotions and are prone to misjudgment, especially in real-world in-vehicle scenarios where they cannot reliably support driving safety warnings; 2) There are significant individual differences in the facial expression range, muscle movement habits, and emotional expression styles of different drivers, making it easy for general models to fail to accurately recognize specific drivers over the long term; 3) When existing emotion recognition methods identify incorrect or uncertain emotions, they lack an automatic evaluation mechanism for the quality of the model's output. The model cannot self-correct or learn, leading to a gradual decline in recognition accuracy after long-term operation. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method, apparatus and electronic device for adaptive updating of a driver emotion recognition model, so as to alleviate the above-mentioned problems existing in the related art.

[0004] In a first aspect, embodiments of the present invention provide an adaptive update method for a driver emotion recognition model, comprising: constructing texture feature vectors and geometric feature vectors based on face detection results of a driver's facial video frame sequence, and fusing the texture feature vectors and geometric feature vectors; performing emotion recognition on the fused vectors using a personalized emotion recognition model on the vehicle end, and performing multi-dimensional credibility assessment based on the emotion recognition results; marking difficult samples from the driver's facial video frame sequence based on the multi-dimensional credibility assessment results and preset threshold information, and iteratively updating the parameters of the personalized emotion recognition model based on the obtained difficult samples; desensitizing the target data corresponding to the difficult samples, and sending the desensitized data and the updated parameters of the personalized emotion recognition model to the cloud, so that the cloud updates the global emotion recognition model based on the desensitized data and the updated parameters and sends the updated model to the vehicle end.

[0005] Secondly, embodiments of the present invention also provide an adaptive update device for a driver emotion recognition model, comprising: a fusion module, configured to construct texture feature vectors and geometric feature vectors based on face detection results of a driver facial video frame sequence, and to fuse the texture feature vectors and geometric feature vectors; an evaluation module, configured to perform emotion recognition on the fused vectors using a personalized emotion recognition model on the vehicle end, and to perform multi-dimensional credibility evaluation based on the emotion recognition results; a first update module, configured to mark difficult samples from the driver facial video frame sequence based on the multi-dimensional credibility evaluation results and preset threshold information, and to iteratively update the parameters of the personalized emotion recognition model based on the obtained difficult samples; and a second update module, configured to desensitize the target data corresponding to the difficult samples, and send the desensitized data and the updated parameters of the personalized emotion recognition model to the cloud, so that the cloud updates the global emotion recognition model based on the desensitized data and the updated parameters and sends the updated model to the vehicle end.

[0006] Thirdly, embodiments of the present invention also provide an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the driver emotion recognition model adaptive update method described in the first aspect above.

[0007] This invention provides a method, apparatus, and electronic device for adaptive updating of a driver emotion recognition model. First, texture feature vectors and geometric feature vectors are constructed and fused based on face detection results from a driver's facial video frame sequence. Then, a personalized emotion recognition model on the vehicle side performs emotion recognition on the fused vectors and conducts a multi-dimensional credibility assessment based on the emotion recognition results. Next, based on the multi-dimensional credibility assessment results and preset threshold information, difficult samples are marked from the driver's facial video frame sequence. The parameters of the personalized emotion recognition model are iteratively updated based on the obtained difficult samples. Then, the target data corresponding to the difficult samples is desensitized, and the desensitized data and the updated parameters of the personalized emotion recognition model are sent to the cloud, so that the cloud updates the global emotion recognition model based on the desensitized data and updated parameters and sends it to the vehicle side. By employing the aforementioned technology, facial images of the driver are captured by an in-vehicle camera, and feature vector fusion is performed after face detection. A multi-dimensional credibility assessment mechanism is then introduced to mark difficult samples and iteratively update the parameters of the personalized emotion recognition model on the vehicle side, thereby improving the accuracy and stability of emotion recognition on the vehicle side. Furthermore, the data from difficult samples is sent to the cloud so that the cloud can update the global emotion recognition model before sending it back to the vehicle side, forming a collaborative model optimization architecture between the vehicle side and the cloud. This enables the emotion recognition model to have adaptive update capabilities, thereby ensuring the universality of accurate emotion recognition for different drivers on the vehicle side.

[0008] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0009] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0010] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating an adaptive update method for a driver emotion recognition model according to an embodiment of the present invention. Figure 2 This is an example diagram of the adaptive update process of the driver emotion recognition model in an embodiment of the present invention; Figure 3 This is an example diagram illustrating data interaction between the vehicle terminal system and the cloud subsystem in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an adaptive update device for a driver emotion recognition model according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] Currently, existing in-vehicle emotion recognition technologies have the following shortcomings: 1) Most emotion recognition models perform well in recognizing positive emotions, but are weak in recognizing negative emotions and are prone to misjudgment, especially in real-world in-vehicle scenarios where they cannot reliably support driving safety warnings; 2) There are significant individual differences in the facial expression range, muscle movement habits, and emotional expression styles of different drivers, making it easy for general models to fail to accurately recognize specific drivers over the long term; 3) When existing emotion recognition methods identify incorrect or uncertain emotions, they lack an automatic evaluation mechanism for the quality of the model's output, and the model cannot self-correct or learn, resulting in a gradual decrease in recognition accuracy after long-term operation.

[0014] Based on this, the present invention provides a driver emotion recognition model adaptive update method, device and electronic device, which can alleviate the above-mentioned problems existing in related technologies.

[0015] To facilitate understanding of this embodiment, a driver emotion recognition model adaptive update method disclosed in this embodiment of the invention will first be described in detail. (See [link to relevant documentation]). Figure 1 As shown, the method may include the following steps: Step S102: Construct texture feature vectors and geometric feature vectors based on the face detection results of the driver's face video frame sequence, and then fuse the texture feature vectors and geometric feature vectors.

[0016] Step S104: Perform emotion recognition on the fused vector using the personalized emotion recognition model on the vehicle side, and conduct multi-dimensional credibility assessment based on the emotion recognition results.

[0017] Step S106: Based on the multi-dimensional credibility assessment results and preset threshold information, difficult samples are marked from the driver's facial video frame sequence, and the parameters of the personalized emotion recognition model are iteratively updated based on the obtained difficult samples.

[0018] Step S108: De-identify the target data corresponding to the difficult samples, and send the de-identified data and the updated parameters of the personalized emotion recognition model to the cloud, so that the cloud can update the global emotion recognition model based on the de-identified data and the updated parameters and send the updated model to the vehicle.

[0019] This invention provides an adaptive update method for a driver emotion recognition model. First, texture feature vectors and geometric feature vectors are constructed and fused based on face detection results from a driver's facial video frame sequence. Then, a personalized emotion recognition model on the vehicle side performs emotion recognition on the fused vectors and conducts a multi-dimensional credibility assessment based on the emotion recognition results. Next, based on the multi-dimensional credibility assessment results and preset threshold information, difficult samples are marked from the driver's facial video frame sequence. The parameters of the personalized emotion recognition model are iteratively updated based on the obtained difficult samples. Then, the target data corresponding to the difficult samples is desensitized, and the desensitized data and the updated parameters of the personalized emotion recognition model are sent to the cloud. This allows the cloud to update the global emotion recognition model based on the desensitized data and updated parameters and send it to the vehicle side. This operation method captures driver facial images through an in-vehicle camera and performs feature vector fusion after face detection. It then introduces a multi-dimensional credibility assessment mechanism to mark difficult samples and iteratively updates the parameters of the personalized emotion recognition model on the vehicle, improving the accuracy and stability of emotion recognition on the vehicle. Furthermore, it uses difficult samples to send relevant data to the cloud so that the cloud can update the global emotion recognition model and then send it back to the vehicle, forming a collaborative model optimization architecture between the vehicle and the cloud. This enables the emotion recognition model to have adaptive update capabilities, thereby ensuring the universality of accurate emotion recognition for different drivers on the vehicle.

[0020] As one possible implementation, the face detection result may include a set of face detection boxes; based on this, the construction of texture feature vectors and geometric feature vectors based on the face detection result of the driver's face video frame sequence in step S102 above may include: A1) generating an initial image sequence based on the set of face detection boxes; A2) calculating the texture feature vector corresponding to each initial image in the initial image sequence through a convolutional neural network; A3) performing face key point detection on each initial image in the initial image sequence, and constructing the geometric feature vector corresponding to each initial image based on the face key point detection result of each initial image.

[0021] The system continuously captures video sequences of the driver's face using an in-vehicle camera, and obtains bounding boxes for the facial regions using a face detection model. The image is cropped and aligned to a uniform size to obtain a normalized image. This normalized image is then input into a convolutional neural network to output a texture feature vector. ,in, For normalized images, For texture feature space, This function calculates the texture feature vector. A normalized image is input into a facial landmark detection model to obtain a set of landmarks (including landmarks in areas such as the eyes, eyebrows, and mouth). Based on this set, the distances between landmarks are calculated to obtain geometric quantities such as eye opening and mouth opening. Geometric feature vectors are then constructed based on these geometric quantities. ,in, For geometric quantities, For geometric feature space, The function is used to compute the geometric feature vectors; two types of features (i.e., texture feature vectors and geometric feature vectors) can be fused into a unified sentiment representation space through a linear mapping. Specifically, the fused features (i.e., the fused vectors) can be calculated using the following formula:

[0022] in, The fused vector , This is the weight matrix. For bias, It is a non-linear activation function. This is the fusion function.

[0023] As one possible implementation, the emotion recognition result may include the category and classification probability of each emotion corresponding to the fused vector of each initial image. Based on this, the multi-dimensional credibility assessment based on the emotion recognition result in step S104 above may include: B1) calculating the maximum value of the classification probability corresponding to each initial image as the single-frame confidence; B2) calculating the probability distribution entropy corresponding to each initial image based on the classification probability corresponding to each initial image; B3) counting the number of initial images whose single-frame confidence is continuously less than a preset confidence threshold within each first sliding window as the number of consecutive low-confidence frames; B4) calculating the average classification probability of each emotion within each second sliding window as the emotion intensity value, and performing trend analysis on the emotion intensity value of each emotion to obtain the emotion intensity change trend of each emotion; B5) evaluating the credibility of the personalized emotion recognition model corresponding to each emotion based on the obtained single-frame confidence, probability distribution entropy, number of consecutive low-confidence frames, and emotion intensity change trend.

[0024] For example, the above-mentioned step of evaluating the credibility of the personalized emotion recognition model corresponding to each emotion based on the obtained single-frame confidence, probability distribution entropy, number of consecutive low-confidence frames and emotion intensity change trend may include: normalizing the obtained single-frame confidence, probability distribution entropy, number of consecutive low-confidence frames and emotion intensity change trend and inputting them into a preset credibility scoring model, and calculating and outputting the credibility score of each emotion through the preset credibility scoring model.

[0025] Continuing from the previous example, the fused features can be input into the vehicle-side emotion classifier (i.e., a personalized emotion recognition model) for emotion recognition. The emotion classifier outputs an emotion probability vector corresponding to the image. Maximum emotional probability and the corresponding emotion categories, among which, For the first The classification probability of each emotion category. For the number of emotion categories, That is The maximum value of the included classification probabilities.

[0026] To assess the reliability of personalized emotion recognition models for the current emotion recognition task, a model reliability index system consisting of four parts was constructed: (1) Single frame confidence : maximum probability ,like ,in If the probability threshold is set, the personalized emotion recognition model may be unreliable for the current emotion recognition task; (2) Probability distribution entropy This is used to measure whether a personalized emotion recognition model is ambiguous in its emotion classification and whether there is difficulty in distinguishing different emotion categories. ,like ,in If the probability distribution entropy threshold is too high, it indicates that the classification probability distribution is too uniform, and the personalized emotion recognition model cannot clearly determine the emotion category. (3) Length of consecutive low confidence windows That is, in length of The number of low-confidence frames is counted within the sliding window. ,in For indicator functions, if ,in If the number of frames with low confidence is less than the threshold, it indicates that the personalized emotion recognition model remains uncertain for a period of time, which is a typical unreliable state.

[0027] (4) Trends in the intensity of emotions Define a sliding window as a set of fixed-length frames sampled continuously over time. Let the current time be... The length of the sliding window is The current window is then denoted as ,in The number of frames contained in the current window. This can be set according to the video frame rate (e.g., set to 5 to 30 frames per second); Indicates the first The frame belongs to the emotion category. The predicted probability can then be... As an emotion category In the Emotional intensity of frames Emotional categories In the window The average emotional intensity within is defined as The trend of change in emotion intensity is defined as the difference between the average emotion intensity of two adjacent windows: ,in Indicates emotion category The trend of changes in the intensity of emotions; if Falling into the emotion category Preset legal trend range Besides, that is If the result is not consistent with the temporal evolution pattern of the emotion category output by the personalized emotion recognition model, it indicates that the emotion category judgment result may be incorrect. Different emotion categories have their own natural temporal patterns. For example, the intensity of surprise should have a rapidly decreasing trend, the intensity of fear should have a rapidly increasing and highly fluctuating trend, the intensity of annoyance should have a slowly increasing trend, and the intensity of anger should have a continuously increasing trend with a large intensity value. Continuing from the previous example, a model credibility score can be defined based on the aforementioned model credibility index system. ,like ,in If the threshold for credibility scoring is set, then the personalized emotion recognition model's judgment of the emotion category for the corresponding segment is considered unreliable or may contain misjudgments. If the credibility score is credible, then the personalized emotion recognition model's emotion recognition results for the corresponding segment are considered reliable. The specific definition of the credibility score is as follows: First, define the current trend of emotion intensity change. to the legal trend range The distance is:

[0028] The sentiment trend consistency score is redefined as:

[0029] in, For this emotion category Maximum allowable trend deviation; if Falling within the legal trend range Inside, then ; Deviation The greater the degree, The closer to 0; The model credibility score is obtained by normalizing and weighting the single-frame confidence score, probability distribution entropy, length of consecutive low-confidence windows, and sentiment trend consistency score. ; where the normalized confidence value of a single frame is denoted as The normalized value of the probability distribution entropy is denoted as The normalized value of the continuous low confidence window length is denoted as The model credibility score is then defined as:

[0030] in, , , , As weight, satisfying ;when When the current segment's emotion recognition result is deemed unreliable; when At that time, the emotion recognition result of the current segment is determined to be reliable.

[0031] As one possible implementation, the multi-dimensional credibility assessment results may include probability distribution entropy, the number of consecutive low-confidence frames, the trend of emotion intensity change, and the credibility score of each emotion. The preset threshold information may include a probability distribution entropy threshold, a consecutive low-confidence frame number threshold, and the range of emotion intensity change trend and credibility score threshold for each emotion. Based on this, the step S106 above, which involves marking difficult samples from the driver's facial video frame sequence based on the multi-dimensional credibility assessment results and preset threshold information, may include: marking segments in the driver's facial video frame sequence that correspond to the multi-dimensional credibility assessment results meeting preset conditions as difficult samples. Wherein, the multi-dimensional credibility assessment results meeting preset conditions include at least one of the following: the credibility score is less than its corresponding credibility score threshold, the probability distribution entropy is greater than the probability distribution entropy threshold, the number of consecutive low-confidence frames is greater than the consecutive low-confidence frame number threshold, and the emotion intensity change trend is not within its corresponding emotion intensity change trend range.

[0032] Continuing from the previous example, if the emotion recognition result of the current segment meets any of the following conditions: the model's reliability is too low ( ), excessively long continuous uncertainty window ( ), the probability distribution entropy is too high ( Inconsistent sentiment trends , If a valid trend range is pre-defined for the corresponding emotion category, the system will automatically label the segment as a "difficult sample" and save the corresponding original image sequence, facial key point sequence, and probability vector. Sentiment Trends Information such as uncertainty labels (i.e. labels that indicate uncertainty about the emotional category of the segment).

[0033] As a possible implementation, before iteratively updating the personalized emotion recognition model based on the obtained difficult samples in step S106, the following operation can also be performed: write the desensitized data and the updated parameters into a preset data pool, so that when the amount of data in the preset data pool reaches a preset data amount threshold, the parameters of the personalized emotion recognition model based on the obtained difficult samples in step S106 can be iteratively updated.

[0034] As one possible implementation, the step S106 above, which iteratively updates the parameters of the personalized emotion recognition model based on the obtained difficult samples, may include: iteratively updating the parameters of the personalized emotion recognition model through low-rank adaptation based on the obtained difficult samples.

[0035] Following the previous example, in the model deployment phase, the initial general emotion recognition model is first pre-trained on the cloud server to obtain the general emotion recognition model. Then, the pre-trained emotion recognition model is pruned and quantized to reduce the computational load and storage usage. Finally, the obtained emotion recognition model is deployed to the vehicle.

[0036] Continuing from the previous example, during the model fine-tuning phase (when model parameters are updated), a low-rank adaptation (LORA) fine-tuning method is adopted. During fine-tuning, all original parameters in the emotion recognition model (such as a personalized or general emotion recognition model) are kept frozen and not updated. For each core weight matrix in the emotion recognition model... Introduce a pair of low-rank matrices (Initialized to a random Gaussian distribution) and (Initialized to a zero matrix), making the forward propagation computation become ,in For the input of emotion recognition models The output of the emotion recognition model is only for... and Update while maintaining the original weight matrix The optimization is achieved by minimizing the emotion recognition model's emotion recognition error (such as cross-entropy loss) on its corresponding personalized driver dataset. and To enable the emotion recognition model to be quickly adapted to driver emotion recognition tasks under limited computing power.

[0037] Continuing from the previous example, whenever the system captures a difficult sample, it adds it to the corresponding driver's incremental data pool. When the amount of data in this incremental data pool reaches a certain threshold, the system automatically triggers a small-scale fine-tuning of the vehicle-side emotion recognition model. Specifically, this can be achieved by activating the incremental learning engine to start the fine-tuning process of the vehicle-side emotion recognition model when a certain number of new samples have accumulated in the driver's personalized dataset on the vehicle's local end. For the LORA method, the optimized low-rank matrix needs to be... and Compared with the original weight matrix Merge to obtain a new weight matrix This results in a complete new model that requires no additional inference overhead. This personalized emotion recognition model replaces the original general emotion recognition model or the previous version of the personalized emotion recognition model and serves as the model called by the emotion recognition engine the next time the vehicle starts.

[0038] It should be noted that the purpose of fine-tuning the vehicle-side emotion recognition model is to independently construct individualized parameters for the emotion recognition model for the current drivers of the vehicle, so that the vehicle-side emotion recognition model gradually evolves from general emotion recognition to individual emotion recognition.

[0039] As one possible implementation, the basic topology of the global emotion recognition model is the same as that of the personalized emotion recognition model. The target data may include at least one of the following data corresponding to the corresponding difficult samples: texture feature vector, geometric feature vector, fused vector, classification probability, single frame confidence, number of consecutive low confidence frames, and emotion intensity change trend. Based on this, the operation of updating the global emotion recognition model based on the desensitized data and updated parameters performed by the cloud in step S108 above may include: iteratively training the global emotion recognition model using the desensitized data, and fine-tuning the parameters of the global emotion recognition model according to the updated parameters during the iterative training process to achieve the update of the global emotion recognition model.

[0040] Continuing from the previous example, when the vehicle is in a good network condition, the system can perform desensitization processing on the features extracted from the difficult sample segments generated and saved in steps S102 to S106 (including but not limited to: texture feature vectors extracted by convolutional neural networks, geometric feature vectors formed by facial key points, fused vectors, and emotion probability sequences / trend statistics on consecutive frames). This desensitization process includes at least one of the following: uploading only feature vectors or statistics without uploading the original face image, normalizing the key point coordinates, reducing precision or randomly perturbing them, removing license plate, identity, timestamp, location and other related identity information, and retaining only the machine learning feature representations required for model training. The desensitized feature data is then obtained. The model increment parameters (i.e., the parameter update results obtained after fine-tuning the vehicle-side emotion recognition model based on difficult samples) are defined as the set of parameters added or updated by the personalized emotion recognition model relative to the initial general emotion recognition model during the fine-tuning process of the personalized emotion recognition model. When using the LORA fine-tuning method, the model increment parameters are preferably the low-rank matrices corresponding to each injection layer (i.e., ...). and The parameter values ​​or weight difference information derived from the low-rank matrix. The vehicle-side selectively uploads anonymized feature data and incremental model parameters obtained by fine-tuning the local emotion recognition model to the cloud server. This uploaded data has had personal identification information removed, retaining only machine learning features used for model optimization. The cloud server can aggregate anonymized data and incremental model parameters from a massive amount of vehicle-side data. The cloud server can then use this data to perform large-scale statistical analysis to discover common emotional patterns and negative expression features across regions and populations. Based on this, the cloud can use the aggregated massive data to retrain the global general emotion recognition model (which shares the same basic network structure as the vehicle-side personalized emotion recognition model), generating a new generation of global general emotion recognition model with stronger performance and better generalization ability, which is then sent to the vehicle-side. This updated global general emotion recognition model can then serve as a new starting point for fine-tuning subsequent personalized emotion recognition models on each vehicle-side.

[0041] For ease of understanding, the implementation process of the above-mentioned adaptive update method for driver emotion recognition model is described in the following example using a specific application.

[0042] like Figure 2 As shown, the adaptive update process of the driver emotion recognition model can mainly include the following steps: Step S1: Predict driver emotion based on multi-domain feature fusion model.

[0043] After performing face detection on the driver's facial video sequence continuously captured by the vehicle camera using a face detection model, the detection boxes of the obtained facial regions are cropped and aligned to a uniform size to obtain a normalized image. The normalized image is then processed by a convolutional neural network to obtain texture features. After the normalized image is processed by a facial keypoint detection model, keypoint sets for regions such as eyes, eyebrows, and mouth are obtained. Geometric features are constructed based on the keypoint sets. The texture features and geometric features are then fused into a unified emotion representation space through linear mapping to obtain fused features. The fused features are then input into the driver emotion recognition model on the vehicle for emotion recognition. The driver emotion recognition model outputs the corresponding emotion probability vector, the maximum emotion probability, and the corresponding emotion category.

[0044] Step S2: Credibility assessment of the driver emotion recognition model.

[0045] Calculate the single-frame confidence score based on the output of the driver emotion recognition model. Probability distribution entropy Length of consecutive low confidence windows Trends in Emotional Intensity And calculate the model credibility score. And then through comparison With credibility scoring threshold This is used to determine whether the emotion recognition result of the current segment is reliable.

[0046] Step S3: Unreliable data samples (i.e., difficult samples) are automatically captured.

[0047] If the emotion recognition result of the current segment meets any of the following conditions: , , , If the system automatically labels the segment with an uncertainty tag to characterize it as a "hard sample", it will save the corresponding original image sequence, facial key point sequence, and probability vector. Sentiment Trends Information such as uncertainty labels.

[0048] Step S4: Fine-tune the small sample on the vehicle side.

[0049] During the model deployment phase, an initial general emotion recognition model is pre-trained on a cloud server. After pruning and quantization, the pre-trained emotion recognition model is obtained and deployed to the vehicle. During the model fine-tuning phase, the LoRa fine-tuning method is used to minimize the emotion recognition error (such as cross-entropy loss) to fine-tune the emotion recognition model.

[0050] Step S5: Personalized emotion recognition model update and iteration.

[0051] Each time a difficult sample is captured, the system adds it to the corresponding incremental data pool. When the amount of data in the corresponding incremental data pool reaches a certain threshold, the incremental learning engine is activated to perform the fine-tuning process of the vehicle-side emotion recognition model, so as to realize the individualized parameters of the emotion recognition model independently constructed for the driver of the vehicle currently in use.

[0052] Step S6: Update the global general emotion recognition model collaboratively in the cloud.

[0053] When the vehicle is in a good network condition, the system desensitizes the relevant features of difficult sample segments. The vehicle only selectively uploads the desensitized feature data and the incremental model parameters obtained by fine-tuning the local emotion recognition model to the cloud server. The cloud server aggregates the desensitized data and incremental model parameters from a large number of vehicles to perform large-scale statistical analysis, and retrains the global general emotion recognition model to generate a new generation of global general emotion recognition model.

[0054] Step S7: Global universal model distribution.

[0055] The cloud distributes a new generation of global universal emotion recognition model to eligible vehicles via the network. Under conditions that ensure driving safety (such as when the vehicle is turned off or undergoing an upgrade), the vehicle silently completes the update of the emotion recognition model and the synchronization of related emotion recognition strategies.

[0056] Example 1: In the highly similar emotions of "annoyance-anger", difficult samples are automatically labeled and the individualized emotion recognition model for drivers is learned through mechanisms such as continuous frame emotion probability sequence analysis, emotion trend analysis, continuous uncertain window length calculation, probability distribution entropy calculation, and model credibility score calculation.

[0057] Example 1 Scenario Description: A driver's actual expression is "anger," but the emotion recognition model often identifies it as "annoyance." Annoyance facial action units are mostly characterized by a slight frown, fatigue, and a slight downward pull of the corners of the mouth, while anger facial action units are mostly characterized by a deeper frown, tense eye muscles, and a clenched jaw. However, due to significant differences in facial expression habits compared to standard datasets, especially on Asian faces, the differences are more blurred.

[0058] The adaptive update steps of the driver emotion recognition model in Example 1 are described as follows: During driving, the system continuously collects facial images and performs continuous frame emotion recognition (model's original prediction) through the vehicle-side emotion recognition model; as shown in Table 1, it is the time series output by the emotion recognition model within a certain time period.

[0059] Table 1. Examples of consecutive frame emotion probability sequences

[0060] As shown in Table 1, the emotion recognition model predicted that the driver was in an angry state during this period, but the driver's actual emotional state was annoyance.

[0061] Conduct model credibility assessment: a) The maximum probability confidence score is satisfied for 12 consecutive frames. ; b) Calculate the probability distribution entropy for each frame. Because "anger" and "boredom" are close in magnitude over a long period, the average probability distribution entropy is... The model's judgment can be considered fuzzy; c) Calculate the length of the consecutive low-confidence window, which is actually satisfied for 12 consecutive frames. This can be considered a highly uncertain event; d) Analyzing the consistency of emotional trends, the trend of "annoyance" should be characterized by slow fluctuations or a slight increase in emotional intensity, while the trend of "anger" should be characterized by a rapid increase in emotional intensity. The trend of the "anger" intensity sequence is calculated as follows: Therefore, this trend does not conform to the pattern that an "anger" trend should exhibit; e) Calculate the model credibility score It can be seen that the model has low credibility. Combining a) to d), it can be seen that the continuous uncertainty window is too long, the probability distribution entropy is too high, and the sentiment trend is inconsistent. Therefore, the system judges that there is a risk of misjudgment and triggers the automatic capture of difficult samples.

[0062] Automatic capture of difficult samples: The system automatically captures the sequence data of these 12 frames, including the original image sequence, feature fusion vector sequence, facial key points, probability distribution sequence, and trend, and automatically labels the difficult samples as: "Difficult sample - annoyance / anger confusion".

[0063] After a certain number of difficult samples are reached, the system triggers LoRA fine-tuning. LoRA fine-tuning uses sequence samples for model training and fine-tuning. Updates are only made within the "boredom-anger" subspace, improving the model's ability to differentiate between "boredom" and "anger" in a personalized way.

[0064] The fine-tuned model outputs the sentiment probabilities for consecutive frames, as shown in Table 2.

[0065] Table 2. Examples of continuous frame sentiment probability sequences output by the fine-tuned model.

[0066] As shown in Table 2, the trend of the continuous frame emotion probability sequence output by the fine-tuned model more clearly points to the emotion of "annoyance", and the MRS is significantly improved.

[0067] When a vehicle is connected to the network, the vehicle uploads anonymized data related to difficult samples and incremental model parameters obtained during LORA fine-tuning to the cloud. The cloud aggregates more difficult samples from vehicles to update the global general emotion recognition model and distributes the new generation of global general emotion recognition model to the vehicle, thereby improving the overall emotion recognition capability of the vehicle.

[0068] In Example 1, the data uploaded from the vehicle to the cloud mainly includes: (1) the fusion feature vector sequence, key point geometric feature sequence, category probability distribution sequence, trend statistics, and MRS related indicators extracted and anonymized from difficult sample fragments; (2) the low-rank matrix parameter update amount or its equivalent weight difference representation obtained after LORA fine-tuning based on difficult sample fragments. The vehicle does not upload the original image frames to the cloud, but only uploads the anonymized intermediate features and parameter increments (i.e., model increment parameters) to the cloud.

[0069] Example 2: Establishing an adaptive update system for the driver emotion recognition model, such as... Figure 3 As shown, the system mainly includes a vehicle terminal subsystem and a cloud subsystem.

[0070] like Figure 3 As shown, the vehicle terminal system includes: The emotion recognition module is used to identify the driver's emotions and output the emotion probability distribution; The model credibility assessment module is used to calculate the model credibility score; The difficult sample capture module is used to automatically capture difficult samples when at least one of the following occurs: low model confidence, excessively long uncertainty window, excessively high probability distribution entropy, or inconsistent sentiment trend. The incremental learning module is used for personalized fine-tuning of the model based on difficult samples; The communication module is used to upload data (including de-identified data and incremental model parameters) to the cloud and receive models sent from the cloud. like Figure 3 As shown, the cloud subsystem includes: The multi-vehicle data aggregation module is used to receive data uploaded from multiple vehicles. The global model training and update module is used to train and update the global emotion recognition model. The global model distribution module is used to distribute the updated global emotion recognition model to the vehicle. The vehicle terminal system and the cloud subsystem are connected through a communication network to form a dual-closed-loop self-evolving architecture.

[0071] In this invention, the vehicle-side small-sample fine-tuning addresses the issue of differences in facial expression styles among different drivers, enabling the vehicle's ability to recognize the emotions of the same driver to gradually improve with continuous use of the emotion recognition model (i.e., the vehicle-side emotion recognition model becomes more accurate with use).

[0072] In this invention, the cloud-based global model training is based on the features of difficult samples uploaded by all vehicles and the fine-tuned incremental parameters. It absorbs common patterns from the data of all drivers. It assumes that the vehicle is used by multiple drivers for a long time and that different drivers have significant differences in facial expression amplitude, movement habits, and inter-class confusion patterns. The global model update and iteration enable the vehicle to quickly adapt to emotion recognition tasks after the driver is changed.

[0073] The core technical solution of the above-mentioned adaptive update method for driver emotion recognition model includes the following key points: 1) Vehicle-side emotion recognition of driver's facial images: Collect driver's facial images, extract facial texture features and facial key point geometric features, and fuse multi-domain features to output an emotion probability vector, providing a preliminary general model for subsequent credibility assessment; 2) Evaluate model credibility: Quantitatively evaluate the reliability of the model output results through four types of indicators, and construct a unified model credibility scoring calculation method to determine whether the model identification has ambiguity, uncertainty or potential errors; 3) Automatic extraction of difficult samples: When the model credibility evaluation results meet the preset conditions, the system automatically captures the current time segment as a "difficult sample" for subsequent model learning; 4) Personalized incremental learning on the device side: The vehicle-side performs small-sample incremental fine-tuning on the pre-trained emotion recognition model based on difficult samples to obtain a personalized model; 5) Cloud-based collaborative model update: Each vehicle uploads the relevant features of the de-identified difficult samples and the incremental parameters for model fine-tuning to the cloud. The cloud aggregates data from multiple vehicles and trains and updates the global general emotion recognition model.

[0074] 6) Cloud-based distribution of updated model: The cloud distributes the updated global model to the vehicle. The vehicle receives the updated global model, which is used for subsequent emotion recognition or as a basis for fine-tuning.

[0075] The beneficial effects of the aforementioned adaptive update method for driver emotion recognition models mainly include: acquiring driver facial images through in-vehicle cameras and introducing a multi-domain feature fusion mechanism during emotion recognition, comprehensively utilizing facial texture features, key point geometric structure features, and short-term emotion trends to improve the initial recognition ability of negative emotions; compared with the traditional single-frame maximum probability confidence judgment method, the multi-index model credibility evaluation system can effectively reduce the confusion probability of multiple categories of negative emotions and improve the stability of emotion recognition; hard sample capture is automatically triggered based on the model's own behavioral characteristics, without relying on external labels or intervention effects, and can automatically collect valuable samples in real driving scenarios; introducing an end-side small sample individualized incremental learning mechanism driven by hard samples, the system uses a small amount of the latest data to fine-tune some parameters on the vehicle to obtain a personalized model. When network conditions are available, the desensitized statistical features can also be selectively uploaded to the cloud for aggregation analysis and general model upgrades, forming a two-layer optimization architecture of vehicle-side personalized fine-tuning combined with cloud-based global aggregation.

[0076] Based on the above-described adaptive update method for driver emotion recognition models, this invention also provides an adaptive update device for driver emotion recognition models, see [link to relevant documentation]. Figure 4 As shown, the device may include the following modules: The fusion module 402 is used to construct texture feature vectors and geometric feature vectors based on the face detection results of the driver's face video frame sequence, and to fuse the texture feature vectors and geometric feature vectors.

[0077] Evaluation module 404 is used to perform emotion recognition on the fused vector through the personalized emotion recognition model on the vehicle, and to perform multi-dimensional credibility evaluation based on the emotion recognition results.

[0078] The first update module 406 is used to mark difficult samples from the driver's facial video frame sequence based on the multi-dimensional credibility assessment results and preset threshold information, and to iteratively update the parameters of the personalized emotion recognition model based on the obtained difficult samples.

[0079] The second update module 408 is used to desensitize the target data corresponding to the difficult samples, and send the desensitized data and the updated parameters of the personalized emotion recognition model to the cloud, so that the cloud updates the global emotion recognition model based on the desensitized data and the updated parameters and sends the updated model to the vehicle.

[0080] The aforementioned adaptive update device for driver emotion recognition model uses an in-vehicle camera to capture driver facial images and performs feature vector fusion after face detection. A multi-dimensional credibility assessment mechanism is then introduced to mark difficult samples and iteratively update the parameters of the personalized emotion recognition model on the vehicle side, improving the accuracy and stability of emotion recognition. Furthermore, difficult samples are used to send relevant data to the cloud so that the cloud can update the global emotion recognition model before sending it back to the vehicle, forming a collaborative model optimization architecture between the vehicle and the cloud. This enables the emotion recognition model to have adaptive update capabilities, ensuring the universality of accurate emotion recognition for different drivers on the vehicle side.

[0081] The driver emotion recognition model adaptive update device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned driver emotion recognition model adaptive update method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0082] This invention also provides an electronic device, such as... Figure 5 The diagram shows the structure of the electronic device, which includes a processor 51 and a memory 50. The memory 50 stores computer-executable instructions that can be executed by the processor 51. The processor 51 executes the computer-executable instructions to implement the aforementioned adaptive update method for the driver emotion recognition model.

[0083] exist Figure 5 In the illustrated embodiment, the electronic device further includes a bus 52 and a communication interface 53, wherein the processor 51, the communication interface 53, and the memory 50 are connected via the bus 52.

[0084] The memory 50 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 53 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 52 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 52 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0085] The processor 51 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the aforementioned driver emotion recognition model adaptive update method can be completed through the integrated logic circuitry in the hardware of the processor 51 or through software instructions. The processor 51 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the driver emotion recognition model adaptive update method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory. The processor 51 reads the information in the memory and, in conjunction with its hardware, completes the steps of the driver emotion recognition model adaptive update method of the aforementioned embodiment.

[0086] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0087] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0089] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An adaptive update method for a driver emotion recognition model, characterized in that, include: Based on the face detection results of the driver's facial video frame sequence, texture feature vectors and geometric feature vectors are constructed, and the texture feature vectors and geometric feature vectors are fused together. The vehicle-mounted personalized emotion recognition model is used to perform emotion recognition on the fused vectors, and a multi-dimensional credibility assessment is conducted based on the emotion recognition results. Difficult samples are identified from the driver's facial video frame sequence based on multi-dimensional credibility assessment results and preset threshold information, and the parameters of the personalized emotion recognition model are iteratively updated based on the obtained difficult samples. The target data corresponding to the difficult samples is anonymized, and the anonymized data and the updated parameters of the personalized emotion recognition model are sent to the cloud, so that the cloud can update the global emotion recognition model based on the anonymized data and the updated parameters and send the updated model to the vehicle.

2. The adaptive update method for the driver emotion recognition model according to claim 1, characterized in that, The face detection results include a set of face detection bounding boxes; Based on the face detection results of the driver's facial video frame sequence, texture feature vectors and geometric feature vectors are constructed, including: An initial image sequence is generated based on the face detection box set; The texture feature vector corresponding to each initial image in the initial image sequence is obtained by calculating using a convolutional neural network; Facial landmark detection is performed on each initial image in the initial image sequence, and a geometric feature vector corresponding to each initial image is constructed based on the facial landmark detection results of each initial image.

3. The adaptive update method for the driver emotion recognition model according to claim 2, characterized in that, The emotion recognition results include the fused vector corresponding to each initial image, which corresponds to the category and classification probability of each emotion. A multi-dimensional credibility assessment is conducted based on emotion recognition results, including: The maximum classification probability corresponding to each initial image is calculated as the single-frame confidence score. Based on the classification probability corresponding to each initial image, the probability distribution entropy corresponding to each initial image is calculated. The number of initial images whose confidence level is continuously lower than a preset confidence threshold within each first sliding window is counted as the number of consecutive low-confidence frames. The average classification probability of each emotion within each second sliding window is calculated as the emotion intensity value, and trend analysis is performed on the emotion intensity values ​​of each emotion to obtain the trend of emotion intensity change. Based on the obtained single-frame confidence, probability distribution entropy, number of consecutive low-confidence frames, and emotion intensity change trend, the credibility of the personalized emotion recognition model corresponding to each emotion is evaluated.

4. The adaptive update method for the driver emotion recognition model according to claim 3, characterized in that, Based on the obtained single-frame confidence, probability distribution entropy, number of consecutive low-confidence frames, and emotion intensity change trend, the reliability of the personalized emotion recognition model corresponding to each emotion is evaluated, including: After normalizing the obtained single-frame confidence, probability distribution entropy, number of consecutive low-confidence frames, and emotion intensity change trend, the data are input into a preset confidence scoring model. The confidence score of each emotion is calculated and output through the preset confidence scoring model.

5. The adaptive update method for the driver emotion recognition model according to claim 4, characterized in that, The multi-dimensional credibility assessment results include probability distribution entropy, number of consecutive low confidence frames, trend of emotion intensity change, and credibility score of each emotion. The preset threshold information includes probability distribution entropy threshold, number of consecutive low confidence frames threshold, range of emotion intensity change trend and credibility score threshold for each emotion. Difficult samples were identified from the driver's facial video frame sequence based on multi-dimensional credibility assessment results and preset threshold information, including: The segments in the driver's facial video frame sequence that correspond to the multi-dimensional credibility assessment results meeting preset conditions are marked as difficult samples; wherein, the multi-dimensional credibility assessment results meeting preset conditions include at least one of the following: credibility score is less than its corresponding credibility score threshold, probability distribution entropy is greater than the probability distribution entropy threshold, the number of consecutive low confidence frames is greater than the number of consecutive low confidence frames threshold, and the emotional intensity change trend is not within its corresponding emotional intensity change trend range.

6. The adaptive update method for the driver emotion recognition model according to claim 3, characterized in that, Before iteratively updating the personalized emotion recognition model based on the obtained difficult samples, the process also includes: The anonymized data and the updated parameters are written into a preset data pool, and the parameters of the personalized emotion recognition model are iteratively updated based on the obtained difficult samples when the amount of data in the preset data pool reaches a preset data volume threshold.

7. The adaptive update method for the driver emotion recognition model according to claim 3, characterized in that, The parameters of the personalized emotion recognition model are iteratively updated based on the obtained difficult samples, including: Based on the obtained difficult samples, the parameters of the personalized emotion recognition model are iteratively updated using a low-rank adaptation method.

8. The adaptive update method for the driver emotion recognition model according to claim 7, characterized in that, The basic topology of the global emotion recognition model is the same as that of the personalized emotion recognition model. The target data includes at least one of the following data corresponding to the difficult samples: texture feature vector, geometric feature vector, fused vector, classification probability, single-frame confidence, number of consecutive low-confidence frames, and emotion intensity change trend. Updating the global emotion recognition model based on the desensitized data and the updated parameters includes: The global emotion recognition model is iteratively trained using the anonymized data, and the parameters of the global emotion recognition model are fine-tuned according to the updated parameters during the iterative training process.

9. An adaptive update device for a driver emotion recognition model, characterized in that, include: The fusion module is used to construct texture feature vectors and geometric feature vectors based on the face detection results of the driver's face video frame sequence, and to fuse the texture feature vectors and geometric feature vectors. The evaluation module is used to perform emotion recognition on the fused vectors through the personalized emotion recognition model on the vehicle, and to conduct multi-dimensional credibility evaluation based on the emotion recognition results. The first update module is used to mark difficult samples from the driver's facial video frame sequence based on the multi-dimensional credibility assessment results and preset threshold information, and to iteratively update the parameters of the personalized emotion recognition model based on the obtained difficult samples. The second update module is used to desensitize the target data corresponding to the difficult samples, and send the desensitized data and the updated parameters of the personalized emotion recognition model to the cloud, so that the cloud updates the global emotion recognition model based on the desensitized data and the updated parameters and sends the updated model to the vehicle.

10. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the driver emotion recognition model adaptive update method according to any one of claims 1 to 8.

Images