Driving behavior recognition method and system and storage medium

Abstract

The invention discloses a driving behavior recognition method and system and a storage medium, and relates to the technical field of image recognition, and the driving behavior recognition method comprises the steps: sending a preset global model to a plurality of vehicle ends, and receiving target local models fed back by the plurality of vehicle ends based on the preset global model; aggregating the plurality of target local models to obtain a new preset global model; returning to the step of sending the preset global model to the plurality of vehicle ends and receiving the target local model fed back by the plurality of vehicle ends based on the preset global model until the preset global model meets the preset convergence requirement, and determining the preset global model as the target global model; and sending the target global model to the plurality of vehicle ends, so that the plurality of vehicle ends input the acquired current cabin image data into the target global model when acquiring the current cabin image data, thereby obtaining a driving behavior recognition result and feeding back the driving behavior recognition result. The user data privacy is protected in the process of training the driving behavior recognition model.

Description

Technical Field

[0001] The present application relates to a specific technical field, and in particular to a driving behavior recognition method, system and storage medium. Background Art

[0002] Cars are an important tool for people's travel and life. According to the Chinese government website, the number of motor vehicles in China will reach 435 million in 2023, and the number of drivers will reach 523 million. With the popularization of new energy vehicles, the number of motor vehicles will continue to rise. However, with the increase in vehicles, the number of vehicle accidents is also increasing. According to statistics, the number of deaths in automobile traffic accidents in my country in 2022 is 42,012, the number of injuries in traffic accidents is 149,650, and the direct economic losses reach 9.9 billion yuan. In addition to external factors such as roads and environment, the main cause of traffic accidents is the driver's own improper driving behavior. Improper driving behavior will not only harm the safety of drivers and passengers themselves, but also threaten the lives of pedestrians, causing significant loss of life and property. Therefore, the identification of driving behavior has always been a hot topic in academia and industry. By identifying the driver's improper driving behavior and giving reminders, the accident rate can be greatly reduced and the traffic safety of the entire society can be improved.

[0003] Existing deep learning-based driving behavior recognition methods regularly collect user data to cloud data centers, and use these fresh and diverse data as training sets to continuously adjust the parameters of the recognition model to adapt to dynamic changes in the environment, but they cannot guarantee user privacy.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a driving behavior recognition method, system and storage medium, aiming to solve the technical problem of how to protect user data privacy during the training of a driving behavior recognition model.

[0006] To achieve the above objectives, the present application proposes a driving behavior recognition method, which is applied in the cloud and includes:

[0007] Sending a preset global model to multiple vehicle terminals, and receiving multiple target local models fed back by the vehicle terminals based on the preset global model;

[0008] Aggregating the plurality of target local models to obtain an aggregated global model;

[0009] The aggregated global model is used as a preset global model, and the step of sending the preset global model to multiple vehicle terminals and receiving multiple target local models fed back by the vehicle terminals based on the preset global model is returned until the preset global model meets the preset convergence requirement, and the preset global model is determined to be the target global model;

[0010] The target global model is sent to the multiple vehicle terminals, so that when the multiple vehicle terminals collect current cockpit image data, the collected current cockpit image data is input into the target global model to obtain a driving behavior recognition result and feed back the driving behavior recognition result.

[0011] In one embodiment, the step of sending a preset global model to a plurality of vehicle terminals and receiving a plurality of target local models fed back by the vehicle terminals based on the preset global model includes:

[0012] Sending a preset global model to multiple vehicle terminals and obtaining calculation rates of the multiple vehicle terminals;

[0013] Determine a plurality of local iteration numbers of the vehicle ends based on a plurality of the calculation rates;

[0014] Sending the corresponding local iteration times to the multiple vehicle terminals, so that the multiple vehicle terminals train the preset global model based on the corresponding local iteration times, obtain the target local model and provide feedback;

[0015] Receive multiple target local models fed back by the vehicle end.

[0016] In one embodiment, the step of determining a plurality of local iteration numbers of the vehicle-side based on a plurality of the calculation rates comprises:

[0017] Obtaining the sizes of multiple vehicle-side data sets;

[0018] A plurality of vehicle-side local iteration numbers are determined based on a plurality of the data set sizes and a plurality of the calculation rates.

[0019] In one embodiment, the step of determining a plurality of vehicle-side local iteration numbers based on a plurality of data set sizes and a plurality of calculation rates comprises:

[0020] Determine the computing requirements of the plurality of vehicle terminals based on the plurality of data set sizes and the plurality of computing rates to obtain a computing requirement set;

[0021] Determining a target computing requirement from the set of computing requirements;

[0022] The number of local iterations of the plurality of vehicle ends is determined based on the target computing requirements.

[0023] In one embodiment, the step of aggregating the plurality of target local models to obtain an aggregated global model includes:

[0024] Determine aggregation weights corresponding to the plurality of target local models based on the size of the data set;

[0025] The plurality of target local models are aggregated based on the aggregation weight to obtain an aggregated global model.

[0026] In addition, to achieve the above purpose, the present application also proposes a driving behavior recognition method, which is applied to multiple vehicle terminals and includes:

[0027] When receiving the preset global model sent by the cloud, training the preset global model based on the local cockpit image data set to obtain a target local model;

[0028] Sending the target local model to the cloud, and receiving an aggregated global model fed back by the cloud based on the target local model, wherein the aggregated global model is an updated preset global model;

[0029] Returning to the step of training the preset global model based on the local cockpit image data set to obtain a target local model when the preset global model sent by the cloud is received, until the target global model fed back by the cloud based on the preset global model is received;

[0030] When the current cockpit image data is collected, the collected current cockpit image data is input into the target global model to obtain a driving behavior recognition result.

[0031] In one embodiment, when receiving the preset global model sent by the cloud, the step of training the preset global model based on the local cockpit image dataset to obtain the target local model includes:

[0032] When receiving the preset global model sent by the cloud, sending a calculation rate to the cloud, and receiving a number of local iterations fed back by the cloud based on the calculation rate;

[0033] The preset global model is trained based on the local iteration number and the local cockpit image data set to obtain a target local model.

[0034] In one embodiment, the step of training the preset global model based on the local iteration number and the local cockpit image dataset to obtain the target local model includes:

[0035] Inputting the local cockpit image data set into the preset global model, and iteratively training the preset global model based on the local iteration number to obtain a preset local model;

[0036] The model parameters of the preset local model are quantized based on the quantization function to obtain a target local model.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a driving behavior recognition system, which includes the cloud mentioned above and multiple vehicle terminals mentioned above.

[0038] In addition, to achieve the above purpose, the present application also proposes a driving behavior recognition device, which is applied to the cloud. The driving behavior recognition device includes:

[0039] A model distribution module, used to send a preset global model to multiple vehicle terminals, and receive multiple target local models fed back by the vehicle terminals based on the preset global model;

[0040] A model aggregation module, used for aggregating the plurality of target local models to obtain an aggregated global model;

[0041] A model iteration module, used to use the aggregated global model as a preset global model, and return to the step of sending the preset global model to multiple vehicle terminals and receiving multiple target local models fed back by the vehicle terminals based on the preset global model, until the preset global model meets the preset convergence requirement, and determines that the preset global model is the target global model;

[0042] The result feedback module is used to send the target global model to the multiple vehicle terminals, so that when the multiple vehicle terminals collect the current cockpit image data, they input the collected current cockpit image data into the target global model, obtain the driving behavior recognition result and feedback the driving behavior recognition result.

[0043] In addition, to achieve the above purpose, the present application also proposes a driving behavior recognition device, which is applied to multiple vehicle terminals, and the driving behavior recognition device includes:

[0044] A local training module, configured to train the preset global model based on a local cockpit image data set to obtain a target local model when the preset global model is received from the cloud;

[0045] A model feedback module, used to send a target local model to the cloud, and receive an aggregated global model fed back by the cloud based on the target local model, wherein the aggregated global model is an updated preset global model;

[0046] a training iteration module, configured to return to the step of training the preset global model based on the local cockpit image data set to obtain a target local model when the preset global model sent by the cloud is received, until the target global model fed back by the cloud based on the preset global model is received;

[0047] The result recognition module is used to input the collected current cockpit image data into the target global model when collecting the current cockpit image data to obtain the driving behavior recognition result.

[0048] In addition, to achieve the above-mentioned purpose, the present application also proposes a driving behavior recognition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the driving behavior recognition method described above.

[0049] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the driving behavior recognition method described above are implemented.

[0050] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the driving behavior recognition method described above are implemented.

[0051] One or more technical solutions proposed in this application have at least the following technical effects:

[0052] Send a preset global model to multiple vehicle terminals, and receive target local models fed back by multiple vehicle terminals based on the preset global model; aggregate multiple target local models to obtain a new preset global model; return to the step of sending the preset global model to multiple vehicle terminals, and receiving the target local models fed back by multiple vehicle terminals based on the preset global model, until the preset global model meets the preset convergence requirements, and the preset global model is determined to be the target global model; send the target global model to multiple vehicle terminals, so that when multiple vehicle terminals collect the current cockpit image data, they input the collected current cockpit image data into the target global model, obtain the driving behavior recognition result and feedback the driving behavior recognition result. In the process of training the driving behavior recognition model, the target local model obtained by training the preset global model with the local data collected by the vehicle terminal is aggregated. In the process of aggregating the model, only the model parameters are processed without obtaining the specific data of any single user, so that the user data privacy is protected while obtaining a high-precision driving behavior recognition model. By aggregating the local model updates from multiple vehicles, the global model can learn a variety of driving behaviors and scenarios, and improve the generalization ability of the model in different vehicles and environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0055] Figure 1 This is a flow chart of the first embodiment of the driving behavior recognition method applied in the cloud of the present application;

[0056] Figure 2 This is a sensor data processing framework diagram of the first embodiment of the driving behavior recognition method applied to the cloud in this application;

[0057] Figure 3 This is a diagram of the image data processing framework of the first embodiment of the driving behavior recognition method applied to the cloud in this application;

[0058] Figure 4 This is a diagram of the vehicle network federated learning framework of the first embodiment of the driving behavior recognition method applied to the cloud in this application;

[0059] Figure 5 This is a flow chart of a second embodiment of the driving behavior recognition method applied in the cloud of the present application;

[0060] Figure 6 This is a flow chart of the first embodiment of the driving behavior recognition method applied to the vehicle side of the present application;

[0061] Figure 7 A schematic diagram of a brief process of the driving behavior recognition method provided in the second embodiment of the present application;

[0062] Figure 8 This is a schematic diagram of the module structure of a driving behavior recognition device applied to the cloud in an embodiment of the present application;

[0063] Fig. 9 This is a schematic diagram of the module structure of a driving behavior recognition device applied to a vehicle side according to an embodiment of the present application;

[0064] Fig.10 Schematic diagram of the device structure of the hardware operating environment involved in the driving behavior recognition method in the embodiment of the present application.

[0065] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0066] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0067] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0068] In this embodiment, for ease of description, the following description is made with the driving behavior recognition device as the execution subject.

[0069] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a driving behavior recognition device, etc. The following takes the driving behavior recognition device as an example to illustrate this embodiment and the following embodiments.

[0070] Based on this, the embodiment of the present application provides a driving behavior recognition method, which is applied to the cloud. Figure 1 , Figure 1 This is a flow chart of the first embodiment of the driving behavior recognition method applied in the cloud of the present application.

[0071] In this embodiment, the driving behavior recognition method applied in the cloud includes steps S10 to S40:

[0072] Step S10, sending a preset global model to multiple vehicle terminals, and receiving multiple target local models fed back by the vehicle terminals based on the preset global model;

[0073] It should be noted that driving behavior can be divided into three categories: regular driving, irregular driving, and distracted driving. Regular driving refers to driving behavior in which the driver remains awake and abides by traffic rules. Irregular driving refers to abnormal driving behavior of the driver, such as sudden lane changes, driving very close to the vehicle in front, etc. Distracted driving means that the driver's attention is not on driving, such as drunk driving, fatigue driving, or making phone calls and sending text messages while driving. The latter two driving behaviors may cause major traffic accidents. Driving behavior recognition is a kind of pattern recognition. There are currently two main methods: recognition based on sensor data and recognition based on existing cockpit image data sets. Both methods are based on machine learning methods, but the data sets and data processing methods used are different.

[0074] For example, refer to Figure 2 , Figure 2This is a sensor data processing framework diagram of the first embodiment of the driving behavior recognition method applied to the cloud in this application. Driving behavior recognition based on sensor data is a widely used method at present, in which sensor data includes vehicle speed, engine load, engine speed, three-axis acceleration, three-axis gyroscope, etc. After the OBU collects the sensor data, it is necessary to select a feature set according to artificially defined rules, and then use a classifier, such as a support vector machine, a hidden Markov model, a Gaussian mixture model, etc., to classify the data. However, the amount of information contained in the sensor data is relatively limited, and it is not very capable of identifying distracted driving behavior. For example, when the driver is driving fatigued, before a car accident occurs, the driver still drives according to normal operation, so the sensor data cannot reflect whether the driver is in a fatigued state. Only in the first few seconds of the car accident will the sensor data reflect the abnormality, but it is too late to remind the driver at this time. This is a major defect of the prior art one. The reason for this defect is the characteristics of the car accident event. The car accident event often occurs only within a few seconds, and the sensor data cannot reflect the driver's driving state in real time.

[0075] For example, refer to Figure 3 , Figure 3 This is an image data processing framework diagram of the first embodiment of the driving behavior recognition method applied to the cloud in this application. There are two types of driving behavior recognition methods based on image data. One is the traditional pattern recognition method based on artificial features, such as using HOG, SIFT and other operators to extract image features, and then inputting them into the classifier to recognize the image; the other is an image classification method based on deep learning, which automatically extracts image features through a deep convolutional neural network and then inputs them into the fully connected layer to achieve classification. Many experiments have shown that methods based on deep learning can achieve better results in recognition accuracy and robustness than pattern recognition methods based on artificial features. However, there is a huge flaw in the driving behavior recognition method based on deep learning in practical applications, namely the data privacy problem. In order to solve the data privacy problem caused by the image classification method based on deep learning, this application adopts the method of federated learning to adjust the parameters of the recognition model. In the federated learning of the Internet of Vehicles, the vehicle does not need to upload its own data, and the cloud trains the recognition model in a distributed manner.

[0076] Reference Figure 4 , Figure 4 This is a diagram of the vehicle network federated learning framework of the first embodiment of the driving behavior recognition method applied to the cloud in this application. Figure 4 As shown, multiple cars are connected to RSU (Road Side Unit) through wireless links, and RSU is connected to the data center of the car manufacturer through the core network.

[0077] It should be understood that the cloud can specifically be a cloud server in the data center of an automobile enterprise. In the initial stage of training, the cloud server of the data center initializes a global model, that is, a preset global model, which will be sent to the vehicle terminals participating in the training to receive the target local model fed back by all vehicle terminals based on the preset global model. Among them, the target local model is a driving behavior recognition model obtained by each vehicle terminal after training the preset global model based on the local cockpit image data set. Each sample image of the local cockpit image data set carries a driving behavior label. Since the training process of each vehicle terminal for the preset global model is independent and does not need to communicate with each other, the cloud can obtain multiple driving behavior recognition models trained based on fresh and diverse data sets while ensuring user privacy.

[0078] In a feasible implementation, step S10 may include steps S11 to S14:

[0079] Step S11, sending a preset global model to multiple vehicle terminals, and obtaining calculation rates of the multiple vehicle terminals;

[0080] It should be noted that during the global iteration, since the vehicle is moving at high speed, the channel between the vehicle and the RSU is dynamically changing, which makes the communication rate of the vehicle low. The currently widely used federated learning method is synchronous, which means that subsequent operations can only be performed after the cloud receives the training results uploaded by all vehicle terminals, that is, the cloud will only perform aggregation operations when it receives the local models of all vehicles. Due to this characteristic, the convergence speed of federated learning is unstable, and its training time is determined by the vehicle with the slowest training speed. That is, in the process of synchronous federated learning, the time of a global iteration is determined by the device with the slowest computing speed. At this time, if the training amount of all devices is set to the same, the training time of all devices is different. At this time, the device with the fastest training speed needs to wait for the device with the slowest training speed to complete the training. This not only causes a waste of computing resources, but also increases the training time. In order to improve the convergence speed of federated learning in the Internet of Vehicles, this application adopts a local iteration number optimization strategy during the training process.

[0081] In the specific implementation, when the training starts, the cloud sends the preset global model to the vehicle and collects the calculation rate of all vehicle terminals, so that the vehicle The calculation rate is c n , then the calculation of all vehicles

[0082] The rate is

[0083] Step S12, determining a plurality of local iteration times of the vehicle end based on the plurality of calculation rates;

[0084] It should be understood that the computing capacity of the vehicle side can be determined based on the collected computing rate of the vehicle side, and the corresponding number of local iterations can be allocated to different vehicle sides according to the computing capacity of the vehicle side, so that the vehicle side can undertake the appropriate amount of training tasks. This can make the training time of each vehicle side for the preset model roughly the same, avoid the situation where the fast train waits for the slow train, and improve the overall training efficiency.

[0085] In a feasible implementation, step S12 may include steps A11 to A12:

[0086] Step A11, obtaining the sizes of multiple vehicle-side data sets;

[0087] It should be understood that since the amount of data will directly affect the complexity of model training and the required computing resources, before calculating the number of local iterations, it is necessary to first collect the data set size of each vehicle.

[0088] It should be noted that, assuming that the vehicle set participating in the training The number of vehicles is N. For the vehicle side Its local dataset is The size of the dataset is where |·| is the cardinality of the set.

[0089] Step A12, determining multiple local iteration numbers of the vehicle side based on multiple data set sizes and multiple calculation rates.

[0090] It should be understood that the computing rate of the vehicle and the size of the data set jointly determine the amount of training that the vehicle can undertake. For vehicles with a large amount of data but limited computing power, fewer local iterations may be required; conversely, for vehicles with strong computing power but small data volumes, the corresponding local iterations can be appropriately increased.

[0091] In a feasible implementation, step A12 may include: determining the computing requirements of multiple vehicle ends based on multiple data set sizes and multiple computing rates to obtain a computing requirement set; determining the target computing requirement from the computing requirement set; and determining the number of local iterations of multiple vehicle ends based on the target computing requirement.

[0092] It should be noted that for vehicle-side n, the formula for calculating the duration is as follows:

[0093]

[0094] Among them, a is the amount of computation required to train a sample, L n is the number of local iterations of vehicle n. The training time of all vehicles is as equal as possible. The formula of this constraint can be expressed as:

[0095]

[0096] where L1, L2, …, L N is an integer. In order to find the smallest possible number of local iterations, this application proposes a local iteration number solution strategy. By setting the local iteration number, the training task amount of the vehicle is indirectly changed, so that the training time of all vehicles is as close as possible, thereby avoiding the waste of computing resources. The details are as follows:

[0097] First, based on multiple datasets of size D n and multiple computation rates c n Determine the computing requirements of multiple vehicle terminals Get the computational requirements set Then, from the calculation requirement set The target computing requirement is determined in , and the target computing requirement refers to the computing requirement with the largest value. Assuming that the subscript of this number is j, then After that, based on the target calculation requirements Determine the number of local iterations of multiple vehicle ends. For vehicle end n, the number of local iterations is L N Specifically, the number of local iterations of each vehicle end can be determined according to the following calculation formula:

[0098]

[0099] Among them, the round() function means rounding.

[0100] It should be understood that this local iteration number solution strategy takes into account the computing power and data volume of each vehicle in the process of optimizing the local iteration number, so that the training time of all vehicles is close to the training time of the vehicle with the maximum computing demand, ensuring that the training time of each vehicle matches its resource contribution, thereby protecting data privacy while optimizing resource utilization and training efficiency in the federated learning process.

[0101] Step S13, sending the corresponding local iteration number to the multiple vehicle terminals, so that the multiple vehicle terminals train the preset global model based on the corresponding local iteration number, obtain the target local model and provide feedback;

[0102] It should be noted that after determining the number of local iterations of each vehicle end, the corresponding number of local iterations will be sent to each vehicle end, so that each vehicle end uses its own local cockpit image data set to perform local iterative training on the preset global model. In the process of one global iteration, after the number of iterative training of the preset global model on each vehicle end reaches its corresponding number of local iterations, the obtained local model will be fed back to the cloud as the target local model.

[0103] Step S14, receiving multiple target local models fed back by the vehicle end.

[0104] It should be understood that when the vehicle-side training times for the preset global model reaches the corresponding number of local iterations, the trained target local model will be sent to the cloud. The cloud will collect the target local models sent by each vehicle-side to prepare for the subsequent aggregation of the target local models. Model aggregation will only be performed when the cloud confirms that it has received the target local models returned by all vehicles.

[0105] Step S20, aggregating the plurality of target local models to obtain an aggregated global model;

[0106] It should be noted that when the cloud receives the target local models fed back by all vehicles based on the preset global model, it will aggregate the received target local models to obtain a new global model, namely the aggregated global model, which reflects the data characteristics of all vehicles by merging local updates.

[0107] It should be understood that the cloud sends a preset global model to multiple vehicle terminals, receives target local models fed back by multiple vehicle terminals based on the preset global model, aggregates multiple target local models, and obtains an aggregated global model. Such a back-and-forth process is called global iteration.

[0108] Step S30, taking the aggregated global model as the preset global model, and returning to the step of sending the preset global model to multiple vehicle terminals and receiving multiple target local models fed back by the vehicle terminals based on the preset global model, until the preset global model meets the preset convergence requirement, and determining that the preset global model is the target global model;

[0109] It should be noted that when the preset global model does not meet the preset convergence requirements, the global model obtained after the last aggregation of the local models will be sent to each vehicle end for a new round of global iteration until the preset global model obtained in the new round of global iteration meets the preset convergence requirements. The preset global model will be used as the target global model, and the target global model will be used in the actual driving behavior recognition process.

[0110] Step S40, sending the target global model to the multiple vehicle terminals, so that when the multiple vehicle terminals collect current cockpit image data, they input the collected current cockpit image data into the target global model, obtain driving behavior recognition results and feed back the driving behavior recognition results.

[0111] It should be understood that after obtaining the target global model, the cloud will send the target global model to each vehicle end, so that each vehicle end can analyze the real-time collected cockpit image data based on the target global model, identify driving behavior, and obtain driving behavior recognition results.

[0112] In the process of training the driving behavior recognition model, this embodiment aggregates the target local model obtained by collecting the vehicle end using its own local data to train the preset global model. In the process of model aggregation, only the model parameters are processed without obtaining the specific data of any single user, thereby obtaining a high-precision driving behavior recognition model while protecting the privacy of user data. By aggregating local model updates from multiple vehicles, the global model can learn a variety of driving behaviors and scenarios, improving the generalization ability of the model in different vehicles and environments.

[0113] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 5 , step S20 may include steps S21 to S22:

[0114] Step S21, determining aggregation weights corresponding to a plurality of target local models respectively based on the size of the data set;

[0115] It should be noted that in the process of federated learning, the amount of data from different vehicles may vary greatly. Since vehicles with larger data volumes provide more information, their contribution to the global model should be greater. By assigning each local model a weight proportional to its data volume, it can be ensured that the global model better reflects the data characteristics of all vehicles.

[0116] Step S22, aggregating the plurality of target local models based on the aggregation weight to obtain an aggregated global model.

[0117] It should be understood that in order to ensure that all vehicle-side contributions to the global model are fair, when aggregating the target global model, the cloud can aggregate multiple target local models through weighted model aggregation operations to obtain a new global model, namely, the aggregated global model. The specific aggregation formula is as follows:

[0118]

[0119] Among them, w r Indicates the global model parameters sent by the cloud server to the set of vehicles participating in the training during the rth global iteration; Identifies the model parameters of vehicle-side n after the kth local iteration of the rth global iteration; Q(·) is the quantization function; represents the number of all training data samples; w r+1 is the r+1th global model parameter.

[0120] This embodiment determines the aggregation weight corresponding to the target local model of the corresponding vehicle end according to the size of the data set, ensuring that the weight distribution during the model aggregation process is proportional to the amount of data on each vehicle end, thereby ensuring the fairness of the contribution of each vehicle end.

[0121] In addition, the present application embodiment provides a driving behavior recognition method, which is applied to the vehicle side, please refer to Figure 6 , Figure 6 This is a flow chart of the first embodiment of the driving behavior recognition method applied to the vehicle side of the present application.

[0122] In this embodiment, the driving behavior recognition method applied to the vehicle side includes steps S50 to S80:

[0123] Step S50, when receiving the preset global model sent by the cloud, training the preset global model based on the local cockpit image data set to obtain a target local model;

[0124] It should be understood that when the vehicle receives the preset global model sent by the cloud, it will train the preset global model based on the local cockpit image dataset to obtain the target local model. Each sample image in the local cockpit image dataset is labeled with a driving behavior. For example, for the local cockpit image dataset n of the vehicle x n,i is the i-th sample image of device n. n,i is the label corresponding to this sample image (for example, label 0 represents normal driving behavior, 1 represents making a phone call, 2 represents sending a text message, etc.).

[0125] In a feasible implementation, step S50 may include steps S51 to S52:

[0126] Step S51, when receiving the preset global model sent by the cloud, sending the calculation rate to the cloud, and receiving the number of local iterations fed back by the cloud based on the calculation rate;

[0127] It should be understood that when the cloud sends the preset global model to the vehicle, it will also send a calculation rate acquisition signal to the vehicle. When the vehicle receives the calculation rate acquisition signal, it will send its calculation rate to the cloud, so that the cloud can determine the number of iterations of the corresponding vehicle when training the preset global model in one global iteration process, that is, the number of local iterations, according to the received calculation rate, and send the corresponding local iteration number to each vehicle.

[0128] Step S52: training the preset global model based on the local iteration number and the local cockpit image data set to obtain a target local model.

[0129] It should be understood that after receiving the local iteration number, the vehicle side will train the preset global model according to the local iteration number to ensure that the model is fully trained on the local data and obtain the target local model. In the process of local iteration of the preset global model on the vehicle side, the local cockpit image dataset will be used for training to obtain the target local model with local data features.

[0130] In a feasible implementation, step S52 may include: inputting the local cockpit image data set into the preset global model, and iteratively training the preset global model based on the local iteration number to obtain a preset local model; and quantizing the model parameters of the preset local model based on a quantization function to obtain a target local model.

[0131] It should be noted that after the vehicle completes the local iterative training of the preset global model, it will obtain the preset local model. By quantizing the parameters of the preset local model obtained by training, the target local model with reduced data size is obtained. Specifically, in the local training stage, the model parameters are updated as follows:

[0132]

[0133] in, is the model parameter of vehicle n after the kth local iteration of the rth global iteration; μ is the learning rate of the local iteration; represents the gradient. It should be noted that when k = 1, After the local training is completed, the model parameters of the preset local model obtained by the vehicle end n are After obtaining the preset local model The vehicle side will preset the model parameters of the local model based on the quantization function Q(·) Quantify and update the quantized model Send to the cloud. Specifically, the quantization function Q(·) is expressed as follows:

[0134] Q(x p ; s) = ||x||·sign(x p )·ξ(x p ,s)

[0135] Where x=(x1,x2,…,x p ,…) T is the vector to be quantized, x p is the pth dimension of vector x, ||x|| is the 2-norm of vector x; sign(x p ) is x p The sign of x p ≥0, sign(x p )=1, otherwise sign(xp )=-1;ξ(x p ,s) is the same as x p , s related random variables, ξ(x p ,s) is expressed as follows:

[0136]

[0137] This expression means that ξ(x p ,s) The probability value is by The probability value is Where s is an integer greater than 0, which is a manually defined quantization level. When s is determined, l satisfies two conditions: 1) l is an integer; 2)

[0138] Step S60, sending the target local model to the cloud, and receiving the aggregated global model fed back by the cloud based on the target local model, wherein the aggregated global model is an updated preset global model;

[0139] It should be understood that after the vehicle sends the target local model to the cloud, the cloud will check the received target local model. Only when it is determined that the target local model obtained by all vehicle-side training based on the preset global model has been received, the cloud will aggregate the target local model to obtain a new preset global model, that is, the aggregated global model, and send the updated preset global model to all vehicle-sides. Only then can the vehicle-side receive the aggregated global model sent by the cloud.

[0140] Step S70, returning to the step of training the preset global model based on the local cockpit image data set to obtain a target local model when the preset global model sent by the cloud is received, until the target global model fed back by the cloud based on the preset global model is received;

[0141] It should be understood that the model sent by the cloud carries label information, and the vehicle side can determine whether the cloud side sends the preset global model or the target global model based on the label information. In the process of federated learning, when the vehicle side receives the model sent by the cloud, it will first identify the label information of the model. When the vehicle side identifies that the received model is the preset global model, it will iteratively train the preset global model to obtain a new target local model; when the vehicle side identifies that the received model is the target global model, it will save the target global model locally to identify driving behavior based on the target global model.

[0142] Step S80, when collecting the current cockpit image data, inputting the collected current cockpit image data into the target global model to obtain a driving behavior recognition result.

[0143] It should be understood that the target global model can achieve high-precision driving behavior recognition. After the vehicle receives the target global model, it will continue to collect current cockpit image data while the vehicle is driving, and input the current cockpit image data into the trained target global model to achieve real-time recognition of the driver's driving behavior and obtain real-time driving behavior recognition results. The driving behavior recognition results can be used to identify the driver's improper behavior (such as drowsiness, distracted driving, etc.) in real time, and immediately issue a warning when the driver's improper behavior is identified, reminding the driver to pay attention to safety and ensure traffic safety.

[0144] In this embodiment, each vehicle end uses its local data set to independently train the preset global model to obtain the target local model. The global model obtained through local training has strong performance, and the data of each vehicle end contributes to the target global model. By iteratively training the global model using fresh and diverse cockpit image data from each vehicle end, the target global model can achieve higher accuracy in identifying driving behavior.

[0145] For example, in order to help understand the implementation process of the driving behavior recognition method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 7 , Figure 7 A brief flowchart of a driving behavior recognition method is provided, specifically:

[0146] The cloud server selects the vehicles that need to participate in the training; the cloud server sends the machine learning model to the vehicles participating in the training; the vehicles use their own data sets for local training, as shown in the formula As shown; the vehicle quantizes the local model, as shown in the formula Q(x p ; s) = ||x||·sign(x p )·ξ(x p ,s), and then sent to the server; the server performs model aggregation, as shown in formula as shown and then sent to the vehicle.

[0147] Exemplarily, in order to verify the convergence of federated learning, this application uses the State Farm Dataset for simulation experiments. In the experiment, there are 20 vehicles, and the number of samples for each vehicle is uniformly distributed between [200,500]. The trained recognition model is ResNet18, the learning rate is 0.01, the batch size is set to 128, and the number of local iterations is set according to the above strategy. In each global iteration, the server randomly selects different devices to participate in the training. As the training progresses, the loss function value will continue to decrease, and the accuracy of the verification set will increase. In the experiment, 1000 data samples are first used to train the model in a centralized manner (used to simulate offline data sets in actual scenarios), and then 2000 data samples are used to train the model in a federated learning manner (used to simulate user vehicle-side data sets in actual scenarios). Federated learning can learn more obvious features from the vehicle-side data, thereby improving recognition accuracy and reducing training loss values.

[0148] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the driving behavior recognition method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0149] In addition, to achieve the above-mentioned purpose, the present application also proposes a driving behavior recognition system, which includes the cloud mentioned above and multiple vehicle terminals mentioned above.

[0150] Since the present driving behavior recognition system adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described one by one here.

[0151] This application also provides a driving behavior recognition device, please refer to Figure 8 , the driving behavior recognition device is applied in the cloud, and the driving behavior recognition device applied in the cloud includes:

[0152] A model distribution module 10 is used to send a preset global model to multiple vehicle terminals, and receive multiple target local models fed back by the vehicle terminals based on the preset global model;

[0153] A model aggregation module 20, used for aggregating the plurality of target local models to obtain an aggregated global model;

[0154] A model iteration module 30, configured to use the aggregated global model as a preset global model, and return to the step of sending the preset global model to multiple vehicle terminals, and receiving multiple target local models fed back by the vehicle terminals based on the preset global model, until the preset global model meets the preset convergence requirement, and determine that the preset global model is the target global model;

[0155] The result feedback module 40 is used to send the target global model to the multiple vehicle terminals, so that when the multiple vehicle terminals collect the current cockpit image data, they input the collected current cockpit image data into the target global model, obtain the driving behavior recognition result and feedback the driving behavior recognition result.

[0156] This application also provides a driving behavior recognition device, please refer to Fig. 9 The driving behavior recognition device is applied to multiple vehicle terminals, and the driving behavior recognition device applied to multiple vehicle terminals includes:

[0157] A local training module 50 is used to train the preset global model based on the local cockpit image data set to obtain a target local model when the preset global model sent by the cloud is received;

[0158] A model feedback module 60, configured to send a target local model to the cloud, and receive an aggregated global model fed back by the cloud based on the target local model, wherein the aggregated global model is an updated preset global model;

[0159] The training iteration module 70 is used to return to the step of training the preset global model based on the local cockpit image data set to obtain the target local model when the preset global model sent by the cloud is received, until the target global model fed back by the cloud based on the preset global model is received;

[0160] The result recognition module 80 is used to input the collected current cockpit image data into the target global model when collecting the current cockpit image data, so as to obtain the driving behavior recognition result.

[0161] The driving behavior recognition device provided by the present application adopts the driving behavior recognition method in the above embodiment, which can solve the technical problem of how to protect the privacy of user data in the process of training the driving behavior recognition model. Compared with the prior art, the beneficial effects of the driving behavior recognition device provided by the present application are the same as the beneficial effects of the driving behavior recognition method provided by the above embodiment, and the other technical features in the driving behavior recognition device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0162] The present application provides a driving behavior recognition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the driving behavior recognition method in the above-mentioned embodiment one.

[0163] Reference below Fig.10, which shows a schematic diagram of the structure of a driving behavior recognition device suitable for implementing the embodiment of the present application. The driving behavior recognition device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The driving behavior recognition device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0164] like Fig.10 As shown, the driving behavior recognition device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the driving behavior recognition device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the driving behavior recognition device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a driving behavior recognition device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided alternatively.

[0165] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0166] The driving behavior recognition device provided by the present application adopts the driving behavior recognition method in the above embodiment, which can solve the technical problem of how to protect the privacy of user data in the process of training the driving behavior recognition model. Compared with the prior art, the beneficial effects of the driving behavior recognition device provided by the present application are the same as the beneficial effects of the driving behavior recognition method provided by the above embodiment, and the other technical features in the driving behavior recognition device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0167] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0168] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0169] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the driving behavior recognition method in the above-mentioned embodiment.

[0170] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0171] The computer-readable storage medium may be included in the driving behavior recognition device; or may exist independently without being assembled into the driving behavior recognition device.

[0172] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the driving behavior recognition device, the driving behavior recognition device: sends a preset global model to multiple vehicle terminals, and receives multiple target local models fed back by the vehicle terminals based on the preset global model; aggregates the multiple target local models to obtain an aggregated global model; uses the aggregated global model as the preset global model, and returns to the step of sending the preset global model to multiple vehicle terminals and receiving multiple target local models fed back by the vehicle terminals based on the preset global model, until the preset global model meets the preset convergence requirements, and determines that the preset global model is the target global model; sends the target global model to the multiple vehicle terminals, so that when the multiple vehicle terminals collect current cockpit image data, they input the collected current cockpit image data into the target global model to obtain a driving behavior recognition result and feed back the driving behavior recognition result.

[0173] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0174] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0175] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0176] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned driving behavior recognition method, and can solve the technical problem of how to protect the privacy of user data in the process of training the driving behavior recognition model. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the driving behavior recognition method provided in the above-mentioned embodiment, and will not be repeated here.

[0177] The present application also provides a computer program product, including a computer program, which implements the steps of the driving behavior identification method as described above when executed by a processor.

[0178] The computer program product provided by this application can solve the technical problem of how to protect the privacy of user data during the training of the driving behavior recognition model. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the driving behavior recognition method provided by the above embodiment, which will not be repeated here.

[0179] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A driving behavior recognition method, characterized in that: The driving behavior recognition method is applied to the cloud, and the method includes: Sending a preset global model to multiple vehicle terminals, and receiving multiple target local models fed back by the vehicle terminals based on the preset global model; Aggregating the multiple target local models to obtain an aggregated global model; The aggregated global model is used as a preset global model, and the step of sending the preset global model to multiple vehicle terminals and receiving multiple target local models fed back by the vehicle terminals based on the preset global model is returned until the preset global model meets the preset convergence requirement, and the preset global model is determined to be the target global model; The target global model is sent to the multiple vehicle terminals, so that when the multiple vehicle terminals collect current cockpit image data, the collected current cockpit image data is input into the target global model to obtain a driving behavior recognition result and feed back the driving behavior recognition result.

2. The method according to claim 1, characterized in that The step of sending a preset global model to a plurality of vehicle terminals and receiving a plurality of target local models fed back by the vehicle terminals based on the preset global model comprises: Sending a preset global model to multiple vehicle terminals and obtaining calculation rates of the multiple vehicle terminals; Determine a plurality of local iteration numbers of the vehicle ends based on a plurality of the calculation rates; Sending the corresponding local iteration times to the multiple vehicle terminals, so that the multiple vehicle terminals train the preset global model based on the corresponding local iteration times, obtain the target local model and provide feedback; Receive multiple target local models fed back by the vehicle end.

3. The method according to claim 2, characterized in that The step of determining a plurality of local iteration numbers of the vehicle ends based on a plurality of the calculation rates comprises: Obtaining the sizes of multiple vehicle-side data sets; A plurality of vehicle-side local iteration numbers are determined based on a plurality of the data set sizes and a plurality of the calculation rates.

4. The method according to claim 3, characterized in that The step of determining a plurality of local iteration numbers of the vehicle-side based on a plurality of the data set sizes and a plurality of the calculation rates comprises: Determine the computing requirements of the plurality of vehicle terminals based on the plurality of data set sizes and the plurality of computing rates to obtain a computing requirement set; Determining a target computing requirement from the set of computing requirements; The number of local iterations of the plurality of vehicle ends is determined based on the target computing requirements.

5. The method according to claim 1, characterized in that The step of aggregating the plurality of target local models to obtain an aggregated global model comprises: Determine aggregation weights corresponding to the plurality of target local models based on the size of the data set; The plurality of target local models are aggregated based on the aggregation weight to obtain an aggregated global model.

6. A driving behavior recognition method, characterized in that: The driving behavior recognition method is applied to multiple vehicle terminals, and the method includes: When receiving the preset global model sent by the cloud, training the preset global model based on the local cockpit image data set to obtain a target local model; Sending the target local model to the cloud, and receiving an aggregated global model fed back by the cloud based on the target local model, wherein the aggregated global model is an updated preset global model; Returning to the step of training the preset global model based on the local cockpit image data set to obtain a target local model when the preset global model sent by the cloud is received, until the target global model fed back by the cloud based on the preset global model is received; When the current cockpit image data is collected, the collected current cockpit image data is input into the target global model to obtain a driving behavior recognition result.

7. The method according to claim 6, characterized in that The step of training the preset global model based on the local cockpit image data set to obtain the target local model when the preset global model sent by the cloud is received includes: When receiving the preset global model sent by the cloud, sending a calculation rate to the cloud, and receiving a number of local iterations fed back by the cloud based on the calculation rate; The preset global model is trained based on the local iteration number and the local cockpit image data set to obtain a target local model.

8. The method according to claim 7, characterized in that The step of training the preset global model based on the local iteration number and the local cockpit image data set to obtain a target local model includes: Inputting the local cockpit image data set into the preset global model, and iteratively training the preset global model based on the local iteration number to obtain a preset local model; The model parameters of the preset local model are quantized based on the quantization function to obtain a target local model.

9. A driving behavior recognition system, characterized in that: The driving behavior recognition system includes: a cloud and multiple vehicle terminals, the cloud terminal executing the steps of the driving behavior recognition method as described in any one of claims 1 to 5, and the multiple vehicle terminals executing the steps of the driving behavior recognition method as described in any one of claims 6 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the driving behavior recognition method applied to the cloud as described in any one of claims 1 to 5 or the steps of the driving behavior recognition method applied to multiple vehicle terminals as described in any one of claims 6 to 8 are implemented.

Images