Driver emotion recognition method for vehicles with different privacy levels

Abstract

The invention relates to a driver emotion recognition method for vehicles with different privacy levels. For the non-privacy vehicle, the optimal task unloading strategy is obtained by taking the distance between the non-privacy vehicle and the roadside unit, computing resources given to the non-privacy vehicle by the roadside unit and the average time delay of the task as states, taking the task unloading strategy as an action and taking a negative value of the average time delay of the task as an award; the average time delay of the unloading system is reduced to the maximum extent; a private vehicle locally trains a model, and then only model parameters are unloaded to a roadside unit to be summarized; the non-privacy vehicles and the roadside units upload parameters to the central cloud server, and the central cloud server aggregates all model parameters and distributes updated model parameters. By adopting the driver emotion recognition method for the vehicles with different privacy levels, the privacy of the driver can be protected, and the training efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of vehicle networking, and in particular to a method for identifying driver emotions for vehicles with different privacy levels. Background Art

[0002] The driver's emotional state is considered to be one of the most important factors affecting traffic safety.

[0003] Driver expression recognition based on facial expression provides an efficient and accurate method for real-time automatic recognition of driver emotions by utilizing neural networks.

[0004] In the new Internet of Vehicles-Mobile Edge Computing environment, the advantages of driver expression recognition are further enhanced, but it also poses the following two challenges.

[0005] The first challenge is data privacy protection. Facial image data, especially facial image data of private car drivers, is highly sensitive, and improper data exchange or leakage may be used for unauthorized commercial activities or even endanger driving safety. The second challenge is how to comprehensively consider vehicle / roadside computing capabilities, vehicle network link conditions, and user privacy requirements to make efficient task offloading decisions.

[0006] Based on this, the inventor of this application discovered that by classifying vehicles into two types, non-privacy vehicles and privacy vehicles, according to different privacy levels, and designing driver emotion recognition offloading solutions for each type, computing efficiency may be improved while protecting driver privacy.

[0007] Since non-privacy vehicles can share driver data, while privacy vehicles are highly sensitive to privacy, the present invention designs different unloading schemes for vehicles with different privacy levels. By setting the unloading driver data or model parameters to the roadside unit and reasonably allocating the unloading amount, the training efficiency of the system model can be improved while protecting the driver's privacy. Summary of the invention

[0008] To solve the above problems, the purpose of the present invention is to provide a driver emotion recognition method for vehicles with different privacy levels. The technical solution adopted covers non-privacy vehicles, privacy vehicles, roadside units and central cloud servers, so that the system can improve training efficiency while ensuring reasonable accuracy.

[0009] In a first aspect, an embodiment of the present invention provides a delay-aware partial task offloading method for a non-privacy vehicle, the solution comprising:

[0010] Non-private vehicle and roadside unit communication models;

[0011] Non-privacy vehicle local computing latency and edge computing latency;

[0012] Optimization objectives for non-private vehicle task offloading.

[0013] First, the upload rate in the communication model between a non-privacy vehicle n (n∈N={1,2,...,N}, N is the set of non-privacy vehicles, N is the total number of non-privacy vehicles) and a roadside unit k (k∈K={1,2,...,K}, K is the set of roadside units, K is the total number of roadside units) is Where B is the communication link bandwidth, P is the transmission power of non-private vehicles, h is the instantaneous channel gain, σ 2 is the Gaussian white noise power, ξ is the communication interference, L k,n is the path loss fading, defined as Where, d k,n is the distance between the non-privacy vehicle n and the roadside unit k, δ is the path loss factor;

[0014] Secondly, D n The amount of data for the training task for the non-privacy vehicle n. This task can be computed locally or on a roadside unit through task offloading, and the task can be offloaded in any proportion;

[0015] The task offloading strategy for all non-private vehicles N is Λ={λ 1 ,λ 2 ,...,λ N};

[0016] The non-private vehicle n will be n (λ n ∈[0,1]) part of the task is offloaded to the roadside unit for calculation, (1-λ n ) Some tasks are calculated locally;

[0017] When n = 0, indicating that the task is calculated locally; when λ n =1, indicating that the task is completely unloaded to the roadside unit; when 0<λ n <1, indicating that the task has λ n Part of the task is offloaded to the roadside unit, (1-λ n ) Some tasks are calculated locally;

[0018] The local computation delay of non-privacy vehicle n is In the formula, C n is the number of computation cycles required for non-privacy vehicle n to process 1 bit of data, is the available computing resources of the non-privacy vehicle n;

[0019] The connection relationship between non-privacy vehicle n and roadside unit k can be expressed as φ k,n Indicates that φ k,n= 1 means that the non-privacy vehicle n is connected to the roadside unit k, φ k,n = 0 means that the non-privacy vehicle n is not connected to the roadside unit k;

[0020] When a non-privacy vehicle n requests task offloading, λ n The total delay of some tasks includes data upload delay and roadside unit k calculation delay;

[0021] Non-privacy vehicle n unloads λ to roadside unit k n The upload data delay of some tasks is

[0022]

[0023] The computation delay of roadside unit k is In the formula, C k is the number of computation cycles required for roadside unit k to process 1 bit of data, is the computing resource allocated by roadside unit k to the upload task of non-privacy vehicle n, which is determined by the ratio of the data volume of the upload task of non-privacy vehicle n to the total data volume of roadside unit k, that is, is the total computational resources available to roadside unit k;

[0024] Due to the parallelism of tasks, the total delay is determined by the task’s λ n and (1-λ n ) is determined by the maximum value of the partial delay, so the total delay of the non-privacy vehicle n task is

[0025] Finally, the optimization goal of non-private vehicle task offloading is to minimize the average delay of completing all non-private vehicle tasks, and establish an optimization model

[0026] A non-private vehicle can only connect to one roadside unit, i.e.

[0027] For each task offloaded to RSU k, the computing resources allocated to RSU k are positive and should not exceed the total computing resources available to RSU k. Right now

[0028] The total amount of computing resources of the unloading tasks allocated by roadside unit k to all connected non-private vehicles should not exceed the total amount of available computing resources of roadside unit k, that is,

[0029] The completion delay of the task of the non-privacy vehicle should not exceed the set maximum delay T max .

[0030] In a second aspect, an embodiment of the present invention provides a federated learning method for a privacy-sensitive vehicle, the method comprising:

[0031] Training for privacy vehicles;

[0032] Aggregation of roadside units;

[0033] Aggregation of central cloud servers.

[0034] First, at the bottom layer, each privacy vehicle generates and processes raw data while sharing a unified deep learning model;

[0035] Second, in the middle layer, the roadside unit collects and aggregates the local model parameters of all connected privacy vehicles within its coverage area and offloads the aggregated model parameters to the central cloud server for further global aggregation;

[0036] Finally, at the top level, the central cloud server e is responsible for performing global caching and model aggregation tasks;

[0037] The central cloud server receives aggregated model parameters from multiple intermediate layers (roadside units) and globally aggregates these parameters to generate an updated global model;

[0038] As feedback, the central cloud server sends the updated global model parameters back to each roadside unit, which then distributes them to each non-privacy vehicle.

[0039] In a third aspect, an embodiment of the present invention provides a cross-vehicle collaboration model aggregation method, the solution comprising:

[0040] Driver expression recognition pre-training model;

[0041] Transfer learning methods;

[0042] Model aggregation of non-private vehicles and private vehicles.

[0043] First, non-privacy vehicles, privacy vehicles, and roadside units deployed convolutional neural network models as pre-trained models for driver expression recognition;

[0044] Secondly, for different drivers, since there are obvious data differences in facial expressions, lighting, angles, etc., their local models also have their own feature information. Therefore, the local model training task is processed using feature extractors and task-specific layers. Specifically, the feature extractor of the pre-trained model is inherited and partially frozen during the transfer learning process, and the prediction of the new task is learned in the task-specific layer.

[0045] Finally, in cross-vehicle collaborative model aggregation, the model training process for non-privacy vehicles and privacy vehicles is different;

[0046] The training process of non-private vehicles can be viewed as a single-layer federated learning architecture, where non-private vehicles and roadside units serve as training entities to jointly train the model, and the central cloud server is responsible for the final aggregation of model parameters;

[0047] In contrast, the training process of privacy-preserving vehicles uses a two-layer federated learning architecture, where the privacy-preserving vehicles serve as training entities and the roadside units are responsible for the initial aggregation of model parameters, which are then offloaded to the central cloud server for final aggregation. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the prior art description. The drawings are only used to illustrate the implementation methods of driver emotion recognition and are not considered to be limitations of this specification.

[0049] Figure 1 Schematic diagram of a scenario for training a driver emotion recognition model in a connected vehicle environment according to the present invention.

[0050] Figure 2 The figure is a flowchart of a delay-aware partial task offloading method according to the present invention.

[0051] Figure 3 The figure is a flowchart of a federated learning method according to the present invention.

[0052] Figure 4 The figure is a schematic flow chart of a cross-vehicle collaborative model aggregation method according to the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] Embodiment 1

[0055] Figure 2 The figure is a flowchart of a delay-aware partial task offloading method according to the present invention.

[0056] This example uses the distance between the non-privacy vehicle and the roadside unit, the computing resources provided by the roadside unit to the non-privacy vehicle, and the average delay of the task as the state, the task offloading strategy as the action, and the negative value of the average task delay as the reward to obtain the optimal task offloading strategy to minimize the average delay of the offloading system.

[0057] Step 201 inputs the distance between the non-privacy vehicle and the roadside unit, the computing resources provided by the roadside unit to the non-privacy vehicle, and the average delay of the task as the state; the task offloading strategy as the action; and the negative value of the average delay of the task as the reward.

[0058] Step 202 inputs the state, action, and reward into the SAC model to obtain the optimal task offloading strategy.

[0059] The principle of the SAC model is: first, take action based on the current strategy and value function estimate, then update the strategy by maximizing the expected reward and entropy (the randomness of the action), and at the same time update the value function to more accurately evaluate the value of future states. Through this adaptive adjustment, the performance and exploration efficiency of the strategy are gradually improved.

[0060] The expected return function that maximizes the entropy term is

[0061]

[0062] In the formula, π represents the decision strategy, represents the state and action sequence (s) generated by the strategy π t ,a t ), ρ π represents the joint distribution of states and actions under strategy π, r(s t ,a t ) is in state s t Take action in t The direct reward obtained, γ is the discount factor, which reflects the importance of future rewards, α is the temperature parameter, which is used to adjust the entropy weight, H(π(·|s t )) is the state s t The entropy of the strategy.

[0063] To maximize the above goals, the SAC model adopts an actor-critic architecture, where the critic estimates the value of the state-action pair (i.e., the soft Q function) and the actor is responsible for optimizing the strategy.

[0064] The baseline strategies of the SAC model are defined as the complete unloading strategy and the complete non-unloading strategy. When the performance of the current strategy exceeds that of the baseline strategy, that is, when the direct reward of the current strategy is greater than the direct reward of the baseline strategy, a positive additional reward is given; if the direct reward of the current strategy is less than or equal to the direct reward of the baseline strategy, the reward will not be reduced.

[0065] The purpose of this adjustment is to incentivize policies that outperform these defined baseline policies, thereby promoting more effective learning and adaptation in complex environments.

[0066] Through experimental observation, the number of iterations is set to 1500, the learning rate is set to 0.001, the discount factor γ is set to 0.99, and the temperature parameter α is set to 0.12.

[0067] In step 203, the non-privacy vehicle obtains the optimal task offloading strategy to minimize the average delay of the offloading system.

[0068] Embodiment 2

[0069] Figure 3 The figure is a flowchart of a federated learning method according to the present invention.

[0070] In this example, the privacy vehicle uploads the trained driver expression recognition model parameters to the roadside unit for aggregation.

[0071] In step 301, the privacy vehicle uses the local data set to train the driver emotion recognition model and uploads the model parameters.

[0072] Step 302 : The roadside unit aggregates model parameters of multiple privacy vehicles within the coverage area.

[0073] The formula for J (J≤M, M is the total number of private vehicles) private vehicles within the coverage of roadside unit k is:

[0074] In the formula, is the model parameter of the private vehicle m (m∈M={1,2,…,M}, M is the set of private vehicles) at time step t, obtained by 300 rounds of transfer learning iterations, δ m is the weight coefficient of the privacy vehicle m, where δ m is the ratio of the data volume of the training task of private vehicle m to the data volume of the training task of private vehicle J, and are the model parameters after aggregation of roadside unit k at time step t.

[0075] Embodiment 3

[0076] Figure 4 The figure is a schematic flow chart of a cross-vehicle collaborative model aggregation method according to the present invention.

[0077] In this example, non-privacy vehicles, privacy vehicles, and roadside units perform transfer learning, and then the non-privacy vehicles and roadside units upload parameters to the central cloud server. Finally, the central cloud server aggregates all model parameters and distributes the updated model parameters.

[0078] Step 401: Non-privacy vehicles, privacy vehicles, and roadside units deploy driver emotion recognition pre-training models.

[0079] In step 402, the non-privacy vehicle offloads part of the task to the roadside unit according to the partial task offloading scheme, and the two migrate and train the pre-trained model in parallel; the privacy vehicle first migrates and trains the pre-trained model, and then uses the federated learning method to offload the model parameters to the roadside unit.

[0080] In step 403, non-privacy vehicles and roadside units upload model parameters to the central cloud server.

[0081] Step 404: The central cloud server aggregates all model parameters and distributes updated model parameters.

[0082] The formula for the central cloud server e to aggregate non-privacy vehicles, roadside units, and model parameters aggregated by roadside units at time step t is:

[0083] In the formula, and They represent the non-privacy vehicle n, roadside unit k and the model parameters aggregated by roadside unit k at time step t, respectively. and The weight coefficient δ of the non-privacy vehicle n is obtained by 300 rounds of transfer learning iterations. n is the ratio of the data volume of the training task of the non-privacy vehicle n to the total data volume, and the weight coefficient δ of the roadside unit k k ′ is the ratio of the data volume of the roadside unit k training task to the total data volume, and the weight coefficient δ aggregated by the roadside unit k k is the ratio of the data volume aggregated by roadside unit k to the total data volume, where the total data volume is the sum of the data volume of N non-privacy vehicle training tasks, the data volume of K roadside unit training tasks, and the data volume aggregated by K roadside units, and satisfies

[0084]

[0085] After aggregation, the central cloud server e updates the global model parameters Sent back to the client (including non-privacy vehicles, privacy vehicles and roadside units).

[0086] Clients update their local models to reflect the results of global learning.

[0087] The description of the invention is given for the purpose of illustration and description, and is not intended to be exhaustive or to limit the invention to the disclosed form. For those of ordinary skill in the art, the invention may be subject to many modifications and variations. Any modifications, equivalent substitutions, and improvements made within the method of the invention shall be included within the scope of protection of the invention.

Claims

1. A driver emotion recognition method for vehicles with different privacy levels, characterized by: First, define non-privacy vehicles n, n∈N={1,2,...,N}, N is the set of non-privacy vehicles, N is the total number of non-privacy vehicles; roadside unit k (k∈K={1,2,...,K}, K is the set of roadside units, K is the total number of roadside units; the upload rate in the communication model is Where B is the communication link bandwidth, P is the transmission power of non-private vehicles, h is the instantaneous channel gain, σ 2 is the Gaussian white noise power, ξ is the communication interference, L k,n is the path loss fading, defined as Where, d k,n is the distance between the non-privacy vehicle n and the roadside unit k, δ is the path loss factor; Secondly, D n The amount of data for the non-private vehicle n training task; The task offloading strategy for all non-private vehicles N is Λ={λ1,λ2,...,λ N }; The non-private vehicle n will be n Part of the task is offloaded to the roadside unit for calculation, (1-λ n ) Some tasks are calculated locally; n ∈[0,1]; When n = 0, indicating that the task is calculated locally; when λ n =1, indicating that the task is completely unloaded to the roadside unit; when 0<λ n <1, indicating that the task has λ n Part of the task is offloaded to the roadside unit, (1-λ n ) Some tasks are calculated locally; The local computation delay of non-privacy vehicle n is In the formula, C n is the number of computation cycles required for non-privacy vehicle n to process 1 bit of data, is the available computing resources of the non-privacy vehicle n; The connection relationship between non-privacy vehicle n and roadside unit k can be expressed as φ k,n Indicates that φ k,n = 1 means that the non-privacy vehicle n is connected to the roadside unit k, φ k,n = 0 means that the non-privacy vehicle n is not connected to the roadside unit k; When a non-privacy vehicle n requests task offloading, λ n The total delay of some tasks includes data upload delay and roadside unit k calculation delay; Non-privacy vehicle n unloads λ to roadside unit k n The upload data delay of some tasks is The computation delay of roadside unit k is In the formula, C k is the number of computation cycles required for roadside unit k to process 1 bit of data, is the computing resource allocated by roadside unit k to the upload task of non-privacy vehicle n, which is determined by the ratio of the data volume of the upload task of non-privacy vehicle n to the total data volume of roadside unit k, that is, is the total computational resources available to roadside unit k; Due to the parallelism of tasks, the total delay is determined by the task’s λ n and (1-λ n ) is determined by the maximum value of the partial delay, so the total delay of the non-privacy vehicle n task is Finally, the optimization goal of non-private vehicle task offloading is to minimize the average delay of completing all non-private vehicle tasks, and establish an optimization model A non-private vehicle can only connect to one roadside unit, i.e. For each task offloaded to RSU k, the computing resources allocated to RSU k are positive and should not exceed the total computing resources available to RSU k. Right now The total amount of computing resources of the unloading tasks allocated by roadside unit k to all connected non-private vehicles should not exceed the total amount of available computing resources of roadside unit k, that is, The completion delay of the task of the non-privacy vehicle should not exceed the set maximum delay T max .

2. The method according to claim 1, characterized in that The program includes: Training for privacy vehicles; Aggregation of roadside units; Aggregation of central cloud servers; First, at the bottom layer, each privacy vehicle generates and processes raw data while sharing a unified deep learning model; Second, in the middle layer, the roadside unit collects and aggregates the local model parameters of all connected privacy vehicles within its coverage area and offloads the aggregated model parameters to the central cloud server for further global aggregation; Finally, at the top level, the central cloud server e is responsible for performing global caching and model aggregation tasks; The central cloud server receives aggregated model parameters from multiple intermediate layers, i.e., roadside units, and globally aggregates these parameters to generate an updated global model; As feedback, the central cloud server sends the updated global model parameters back to each roadside unit, which then distributes them to each non-privacy vehicle.

3. The method according to claim 1, characterized in that The program includes: Driver expression recognition pre-training model; Transfer learning methods; Model aggregation of non-private vehicles and private vehicles; First, non-privacy vehicles, privacy vehicles, and roadside units deployed convolutional neural network models as pre-trained models for driver expression recognition; Second, for different drivers, local model training tasks are handled using feature extractors and task-specific layers. Specifically, the feature extractor of the pre-trained model is inherited and partially frozen during the transfer learning process, and the prediction of the new task is learned in the task-specific layer. Finally, in cross-vehicle collaborative model aggregation, the model training process for non-privacy vehicles and privacy vehicles is different; The training process of non-private vehicles can be viewed as a single-layer federated learning architecture, where non-private vehicles and roadside units serve as training entities to jointly train the model, and the central cloud server is responsible for the final aggregation of model parameters; The training process of privacy-preserving vehicles uses a two-layer federated learning architecture, where the privacy-preserving vehicles serve as training entities and the roadside units are responsible for the initial aggregation of model parameters, which are then offloaded to the central cloud server for final aggregation.

Images