Model training method, device and system and vehicle

Abstract

The invention provides a model training method, device and system and a vehicle. Relates to the technical field of vehicles. The method comprises the following steps: acquiring driver state data and vehicle driving data, inputting the driver state data into a pre-trained evaluation model, outputting an evaluation label of a driver on a current vehicle driving state, and combining the evaluation label with the corresponding vehicle driving data to obtain a model training data set; and finally, inputting the model training data set into the target model, and training the target model to obtain a trained target model. According to the method provided by the invention, the problem of insufficient efficiency and accuracy of automatic driving model training in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a model training method, device, system and vehicle. Background Art

[0002] In the field of autonomous driving, the use of neural network algorithm models to realize system functions and improve intelligence has become an increasingly important field. Neural network algorithm models require big data for model training. Among them, screening valuable data and establishing model evaluation criteria are crucial links in training models.

[0003] Related technologies, the determination of training data for model training and the evaluation criteria of training objectives mostly rely on the subjective judgment of R&D engineers. However, vehicle driving scenarios are extremely complex, and this subjective evaluation standard has obvious deficiencies in its generalization ability when facing diverse scenarios. It is difficult to ensure that data value and model performance can be accurately evaluated in various driving scenarios, thus restricting the efficiency and accuracy of autonomous driving model training. Summary of the invention

[0004] The present application provides a model training method, device, system and vehicle to solve the problem of insufficient efficiency and accuracy of autonomous driving model training in the prior art.

[0005] In a first aspect, the present application provides a model training method, the model training method comprising:

[0006] Collect driver status data and vehicle driving data, the driver status data includes voice data and expression data;

[0007] Input the driver status data into the pre-trained evaluation model, and output the driver's evaluation label of the current vehicle driving status, which includes a positive evaluation label and a negative evaluation label;

[0008] Combine the evaluation labels with the corresponding vehicle driving data to obtain the model training data set;

[0009] The model training data set is input into the target model, the target model is trained, and a trained target model is obtained.

[0010] In a second aspect, the present application provides a model training device, comprising:

[0011] A collection module is used to collect driver status data and vehicle driving data, where the driver status data includes voice data and expression data;

[0012] An analysis module, used to input the driver state data into a pre-trained evaluation model and output the driver's evaluation label of the current vehicle driving state, wherein the evaluation label includes a positive evaluation label and a negative evaluation label;

[0013] A processing module, used to combine the evaluation label with the corresponding vehicle driving data to obtain a model training data set;

[0014] The training module is used to input the model training data set into the target model, train the target model, and obtain a trained target model.

[0015] In a third aspect, an embodiment of the present application provides a model training system, the model training system comprising: an image sensor, a voice sensor, and a controller;

[0016] The controller is connected to the image sensor and the voice sensor, and executes the model training method as described in the first aspect and various possible designs of the first aspect;

[0017] The image sensor is used to collect the driver's voice data, and the image sensor is used to collect the driver's expression data.

[0018] In a fourth aspect, an embodiment of the present application provides a vehicle, which includes the model training system provided in the third aspect of the present disclosure.

[0019] The model training method, device, system and vehicle provided in the present application collect driver status data and vehicle driving data, input the driver status data into a pre-trained evaluation model, output the driver's evaluation label for the current vehicle driving status, and then combine the evaluation label with the corresponding vehicle driving data to obtain a model training data set; finally, the model training data set is input into the target model, the target model is trained, and a trained target model is obtained. Thus, by collecting the driver's voice and expression data and combining it with the vehicle driving data, a more objective model training data set is generated. Using the pre-trained evaluation model, the driver's status data is converted into positive or negative evaluation labels, thereby effectively reflecting the driver's real feedback on the driving status. Thereby reducing the reliance on the subjective judgment of R&D engineers and improving the objectivity and accuracy of data evaluation. In this way, the trained target model has stronger generalization ability in diversified driving scenarios, which significantly improves the efficiency and accuracy of autonomous driving model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0021] Figure 1An application scenario diagram of the model training method provided in the embodiment of the present application;

[0022] Figure 2 A flowchart of a model training method provided in an embodiment of the present application;

[0023] Figure 3 A flowchart of a model training method provided in yet another embodiment of the present application;

[0024] Figure 4 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of the model training system provided in an embodiment of the present application.

[0026] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0028] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0029] With the development of autonomous driving technology, neural network algorithm models play a key role in realizing system functions and improving the degree of intelligence. However, the challenge in this field is how to train these models efficiently and accurately to cope with complex and changeable vehicle driving scenarios. The training of neural network algorithm models relies on big data, among which screening valuable data and establishing model evaluation criteria are crucial links. In traditional methods, the selection of training data and the evaluation criteria of training targets mainly rely on the subjective judgment of R&D engineers. However, the diversity and complexity of vehicle driving scenarios make this evaluation standard based on subjective judgment obviously insufficient in generalization ability. This deficiency makes it difficult to ensure accurate evaluation of data value and model performance when facing different driving scenarios, thus restricting the efficiency and accuracy of autonomous driving model training.

[0030] In actual applications, the complexity and diversity of autonomous driving systems make the manifestation of problems diverse and often hidden. And because autonomous driving involves complex processes such as the fusion of multiple sensor data, real-time decision-making and control, deviations in any link may lead to a decline in overall performance. In addition, R&D engineers may be influenced by personal experience and prejudice when subjectively judging the value of data and model evaluation criteria, making it difficult to discover technical problems. Especially in an application scenario such as autonomous driving that requires high reliability, the dynamic changes in the environment and the high frequency of operations make the identification and location of problems more challenging.

[0031] Solving this problem is also difficult. How to select standards to replace the subjective evaluation standards of R&D personnel and ensure that the alternatives meet the requirements of various driving scenarios to ensure their generalization ability and accuracy is difficult to solve. Therefore, there is a lack of effective solutions in related technologies.

[0032] In order to solve the above problems, the embodiments of the present disclosure provide a model training method, device, car key and vehicle, which collects the driver's voice and expression data and inputs them into a pre-trained evaluation model to generate an evaluation label reflecting the driver's current driving status. These labels are divided into positive and negative, and are combined with the corresponding vehicle driving data to form an objective model training data set. By inputting this data set into the target model for training, the reliance on the subjective judgment of R&D personnel is reduced, and the generalization ability of the model in a variety of scenarios is enhanced.

[0033] The application scenarios of the embodiments of the present disclosure are explained below:

[0034] Figure 1 This is an application scenario diagram of the model training method provided in the embodiment of the present disclosure. Figure 1As shown, during the startup and operation of the vehicle 100, the on-board controller 110 will collect the driver-related data collected by the image sensor 120 and the voice sensor 130, and send it to the cloud server 140 in combination with the vehicle driving data, so that the cloud server 140 can train the autonomous driving vehicle control function model in combination with the driver-related data and the vehicle driving data.

[0035] It should be noted that Figure 1 In the scenario shown, only one vehicle, on-board controller, image sensor, voice sensor and cloud server is used as an example for illustration, but the present disclosure is not limited to this. That is, the number of vehicles, on-board controllers, image sensors, voice sensors and cloud servers can be arbitrary.

[0036] The model training method provided by the present disclosure is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0037] Figure 2 This is a flow chart of a model training method provided in one embodiment of the present application. Figure 2 As shown, including but not limited to the following steps:

[0038] Step S201: Collect driver status data and vehicle driving data.

[0039] Among them, the driver status data includes voice data and expression data.

[0040] Specifically, this embodiment is used to illustrate the overall implementation process of the model training method.

[0041] The execution entity of the embodiment of the present disclosure may be a cloud server that is communicatively connected to the vehicle control system or the main control unit, and for the convenience of description, it will be collectively referred to as a server hereinafter.

[0042] The model training process in the disclosed embodiments can be a process of collecting test driver-related data and training the model of the corresponding autonomous driving function during the vehicle development stage when the test driver is driving, or it can be a process of collecting vehicle driver-related data and optimizing a bunch of autonomous driving function models during the application stage.

[0043] Driver status data refers to information that reflects the driver's current physiological and psychological state, including voice data and expression data, while vehicle driving data refers to the dynamic information generated by the vehicle during driving, such as speed, acceleration, direction, position, etc.

[0044] Voice data can be collected through the microphone in the car to capture the driver's tone, volume, speed and other characteristics. Expression data is captured by the camera to capture the driver's facial expression changes and analyzed using computer vision technology. Vehicle driving data is obtained through on-board sensors and GPS systems to record the real-time dynamic information of the vehicle.

[0045] These data are initially processed and stored by the on-board computing system and then uploaded to the server for subsequent analysis and use.

[0046] In some embodiments, in order to improve the accuracy and reliability of data collection, multimodal sensor fusion technology can be used (during the vehicle development stage) to combine voice and facial expression data with other physiological signals (such as heart rate, skin electrical response) to provide a more comprehensive driver status assessment.

[0047] In some embodiments, in order to ensure the privacy and security of data, encryption technology can be used during data collection and transmission.

[0048] In some embodiments, for vehicle driving data, in addition to traditional sensors, V2X (Vehicle-to-Everything) communication technology can also be used to obtain more extensive traffic information and improve the comprehensiveness and accuracy of the data.

[0049] Step S202: input the driver status data into a pre-trained evaluation model, and output the driver's evaluation label for the current vehicle driving status.

[0050] Among them, the evaluation labels include positive evaluation labels and negative evaluation labels.

[0051] Specifically, the pre-trained evaluation model is a model trained by machine learning, which can output corresponding evaluation labels according to the input driver status data, and the evaluation labels are quantitative representations of the driver's subjective feelings about the current vehicle driving state, which are divided into positive and negative.

[0052] The server will pre-process the collected driver status data, including denoising, normalization, and feature extraction, in order to adapt to the input requirements of the evaluation model.

[0053] The pre-processed data is fed into a pre-trained evaluation model based on a deep learning neural network that can recognize and classify complex patterns. The model outputs the driver's evaluation label of the current driving status by analyzing features such as voice intonation and facial expressions.

[0054] A positive rating label might indicate that the driver feels safe and comfortable, while a negative rating label might indicate that the driver feels nervous or uneasy.

[0055] The training of the evaluation model can be based on a large amount of historical data, combining supervised learning and unsupervised learning methods to improve the accuracy and robustness of the model.

[0056] Step S203: Combine the evaluation label with the corresponding vehicle driving data to obtain a model training data set.

[0057] Specifically, after obtaining the evaluation label, it needs to be matched and combined with the corresponding vehicle driving data to obtain a data set for model training (ie, a model training data set).

[0058] The combination process needs to ensure the time synchronization of the data, that is, each evaluation label corresponds to the vehicle driving data at the same time point or time period.

[0059] The combined data is organized into a structured dataset, which usually includes multiple feature dimensions, such as the vehicle's speed, acceleration, direction, position, and corresponding evaluation labels.

[0060] In some embodiments, the combined data set needs to be cleaned and organized before it can be used as a model training data set to ensure the integrity and consistency of the data and prepare for subsequent model training. In order to improve the quality of the data set, data enhancement technology can be used to generate more diverse training samples.

[0061] Step S204: input the model training data set into the target model, train the target model, and obtain a trained target model.

[0062] Specifically, the target model refers to the autonomous driving algorithm model that needs to be trained, and the trained target model is a model that can be applied in actual scenarios after training.

[0063] The training process can begin by inputting the constructed model training data set into the target model. The target model may be a complex deep learning network with multiple layers and nodes, which is used to identify and learn the relationship between vehicle driving data and evaluation labels. During the training process, the model continuously adjusts its internal parameters to minimize the error between the predicted output and the actual evaluation label.

[0064] As a result, the target model will optimize the autonomous driving function based on the real-time actual experience of the actual person driving the vehicle for the specific vehicle driving status as the evaluation standard, rather than based on the subjective evaluation of R&D engineers, so that the model training data is more accurate in the corresponding driving scenarios, and the accuracy and versatility of the autonomous driving model after training are better.

[0065] The model training method provided in this embodiment collects driver status data and vehicle driving data, then inputs the driver status data into a pre-trained evaluation model, outputs the driver's evaluation label for the current vehicle driving state, and then combines the evaluation label with the corresponding vehicle driving data to obtain a model training data set; finally, the model training data set is input into the target model, and the target model is trained to obtain a trained target model. Thus, by collecting the driver's voice and expression data and combining it with the vehicle driving data, a more objective model training data set is generated. Using the pre-trained evaluation model, the driver's status data is converted into positive or negative evaluation labels, thereby effectively reflecting the driver's real feedback on the driving state. This reduces the reliance on the subjective judgment of R&D engineers and improves the objectivity and accuracy of data evaluation. In this way, the trained target model has stronger generalization ability in a variety of driving scenarios, significantly improving the efficiency and accuracy of autonomous driving model training.

[0066] Figure 3 The process of the model training method provided in the embodiment of the present application Figure 2 .like Figure 3 As shown, the model training method includes the following steps:

[0067] Step S301: Collect in-vehicle voice data based on a voice sensor.

[0068] Specifically, the embodiments of the present disclosure are used to further explain the model training method in detail.

[0069] In this embodiment, the voice sensor array is arranged in the vehicle near the driver.

[0070] To facilitate the collection of voice data, a voice collection unit with a high-sensitivity microphone array can be deployed in the car and set near the driver. Combined with a signal processing algorithm, voice directional enhanced capture can be achieved to obtain the driver's voice signal.

[0071] Step S302: extracting the driver's voice data from the in-vehicle voice data based on a voice processing algorithm.

[0072] Specifically, in order to accurately extract the driver's voice data, the voice data collected in the car needs to be further processed, and the driver's voice data is extracted in combination with the driver's voice characteristics and speaking habits to ensure the accuracy of the analysis based on the voice data.

[0073] Furthermore, the method for extracting driver voice data includes:

[0074] Step A1: filter the in-car voice data.

[0075] Specifically, the collection of in-car voice data is usually accompanied by various background noises, such as engine noise, wind noise, road noise, and conversations of other passengers, etc. Therefore, it is necessary to effectively filter the in-car voice data to ensure the accuracy of subsequent voice recognition and analysis.

[0076] The filtering process can use frequency domain filtering technology to analyze the spectral characteristics of the speech signal, remove low-frequency and high-frequency noise components, and retain the speech signal in the mid-frequency band. Common frequency domain filtering methods include Fourier transform and wavelet transform. Alternatively, time domain filtering technology can also be used to process speech signals, and short-term burst noise can be eliminated by smoothing and denoising the time series of the speech signal.

[0077] The effect of filtering processing directly affects the quality of voice data, so it needs to be optimized and adjusted according to the specific in-vehicle environment and noise characteristics.

[0078] Step A2: extracting feature information of the filtered in-car voice data.

[0079] The feature information includes intonation, speaking speed and volume.

[0080] Specifically, after completing the filtering process of the in-car voice data, the next task is to extract the feature information in the voice signal, which is used to reflect the driver's emotional state and attention level.

[0081] Among them, the extraction of intonation is usually achieved through fundamental frequency analysis. The fundamental frequency is the basic vibration frequency of the speech signal, which can reflect the speaker's emotional state and tone changes.

[0082] The extraction of speech rate involves the analysis of the temporal characteristics of speech signals, which is usually achieved by calculating the number of syllables per unit time or the duration of a speech segment.

[0083] Volume extraction is to analyze the amplitude characteristics of the speech signal and evaluate the speaker's speech intensity by calculating the energy or amplitude of the signal.

[0084] In the feature extraction process, methods such as linear predictive coding (LPC), Mel-frequency cepstral coefficients (MFCC) and perceptual linear prediction (PLP) can be used. These methods can effectively capture the spectrum and time characteristics of speech signals, providing a rich information basis for subsequent emotion analysis and speech recognition.

[0085] Step A3: converting the filtered voice data into text data.

[0086] Specifically, the filtered voice signal needs to be converted into readable text data, which will be used for subsequent semantic analysis and emotion recognition.

[0087] The specific conversion process is achieved through speech recognition technology. The speech recognition system needs to be trained with a large amount of speech data to improve its recognition accuracy and efficiency.

[0088] Commonly used speech recognition technologies include Hidden Markov Model (HMM), Deep Neural Network (DNN) and Long Short-Term Memory Network (LSTM). The combined use of these technologies can significantly improve the accuracy of speech recognition to ensure the reliability of subsequent analysis.

[0089] Step A4: based on the feature information and text data, extract the voice data corresponding to the driver from the in-vehicle voice data.

[0090] Specifically, after obtaining the filtered feature information and text data, it is necessary to extract the voice data related to the driver. The accuracy and relevance of the extraction result can be ensured by comprehensively considering the feature information and text content of the voice signal.

[0091] In specific implementation, by analyzing the characteristic information of the voice signal, such as intonation, speaking speed and volume, the voice segments that may belong to the driver can be initially screened out. Then, through semantic analysis of the text data, the instructions and conversation content related to the driving task can be identified. This process can be achieved through natural language processing (NLP) technology, including algorithms such as lexical analysis, syntactic analysis and semantic analysis.

[0092] By combining the characteristic information of the speech signal and the semantic content of the text data, the driver's speech data can be effectively extracted without interference from other passengers or background sounds.

[0093] Step S303: Based on the image sensor, collect the corresponding facial image data of the driver.

[0094] Specifically, in this embodiment, there are at least two image sensors, which are located at different positions; the image sensors include infrared fill light sources.

[0095] In order to accurately collect image data corresponding to the driver, multiple high-definition cameras with infrared fill light function (that is, infrared fill light source) can be configured in the car to capture the driver's facial images from multiple angles to reduce the impact of lighting differences such as day, night, and backlight, making it easier to extract relevant information such as the driver's emotions and fatigue status from the facial images.

[0096] Step S304: extracting the facial expression data of the driver from the facial image data corresponding to the driver based on the image processing algorithm.

[0097] Specifically, by combining the driver's facial image with an image processing algorithm, the driver's facial expression data can be recognized and analyzed.

[0098] Furthermore, the method for extracting driver expression data includes:

[0099] Step B1: extracting the eyebrow image, mouth image and eye image from the driver's corresponding facial image data.

[0100] Specifically, extracting the driver's facial features is the basis for expression analysis. Eyebrows, mouth and eyes are the areas with the richest facial expressions, so it is necessary to accurately extract images of these parts from the driver's image data.

[0101] In specific implementation, the driver's face position can be located through a face detection algorithm, which is usually implemented through a convolutional neural network (CNN) or other deep learning models. After locating the face, it is further subdivided into the eyebrow, mouth and eye areas.

[0102] Extraction of eyebrow images requires identification of the outline and position of the eyebrows, which can be achieved through edge detection and shape analysis.

[0103] Extraction of mouth images requires identification of the shape and opening and closing state of the lips, which is usually achieved through color segmentation and morphological processing.

[0104] Extraction of eye images requires identification of the eye contour and pupil position, which can be achieved through Hough transform and template matching.

[0105] The extraction of these feature areas needs to take into account the driver's facial posture changes and lighting conditions to ensure the accuracy and stability of the extraction results.

[0106] The extracted facial features will be used for subsequent expression analysis and emotion recognition, so their integrity and representativeness need to be ensured.

[0107] Step B2: input the eyebrow image, mouth image and eye image into a feature extraction algorithm to extract the corresponding expression data of the driver.

[0108] Specifically, after extracting the driver's eyebrows, mouth and eye images, these images can be analyzed by a feature extraction algorithm to obtain the driver's expression data.

[0109] Exemplarily, these images can be input into corresponding models respectively. For example, the extracted eyebrow image can be input into the feature extraction algorithm to analyze the shape, position and movement state of the eyebrows to identify emotion-related features, such as frowning, raising eyebrows, etc. Next, the mouth image is input into the feature extraction algorithm to analyze the shape, opening and closing state and movement pattern of the lips to identify facial expressions such as smiling, anger or surprise. Finally, the eye image is input into the feature extraction algorithm to analyze the opening and closing state of the eyes, the position of the pupils and the blinking frequency to identify emotional states such as concentration, fatigue or confusion.

[0110] In some embodiments, the image data may be spliced ​​and input into a feature extraction algorithm to quickly identify the driver's facial expression data.

[0111] The design of feature extraction algorithms needs to take into account the diversity and individual differences of facial features to ensure the accuracy and universality of recognition results.

[0112] The extracted expression data will be used for subsequent emotion analysis and driving behavior evaluation, so its accuracy and representativeness need to be ensured.

[0113] Step S305: extract the vehicle driving data received in the vehicle control system.

[0114] Specifically, in combination with the above evaluation results, the driver's real-time feedback on the current vehicle driving state can be obtained. Therefore, it is necessary to obtain vehicle driving data for reflecting the vehicle driving state.

[0115] The server can quickly identify and capture valuable scene data including vehicle camera image streams, operating trajectories and parameters through the on-board controller, and eliminate data that is not related to the vehicle's driving status, such as the content played by the current on-board entertainment device. The eliminated data is then encrypted and uploaded to the cloud storage and the corresponding cases are recorded in detail. This not only accumulates materials for subsequent large-scale model training, but also provides diversified data support for traffic management, automobile manufacturing, insurance assessment and other fields, helping to optimize strategies and upgrade performance in multiple industries.

[0116] By encrypting uploaded data, strictly controlling the data upload process, and accurately identifying and discarding invalid data, we can prevent it from occupying resources in storage, transmission and subsequent processing, reduce operational expenses such as server load and network bandwidth usage, and significantly reduce the total cost of development and operation, achieve the goal of reducing costs and increasing efficiency, and improve overall operational efficiency.

[0117] Step S306: input the expression data into a pre-trained emotion analysis model, and output the driver's emotion score and fatigue score corresponding to the expression data.

[0118] Specifically, the sentiment analysis model is developed based on deep learning technology to identify and evaluate the driver's emotional state.

[0119] In the specific implementation, the extracted expression data is input into the pre-trained emotion analysis model to obtain the driver's emotion score and fatigue score. The emotion score reflects the driver's current emotional state, such as joy, tension or confusion, and the fatigue score reflects the driver's fatigue level.

[0120] The training of the sentiment analysis model requires a large amount of labeled data to ensure its generalization and accuracy in different situations. The output of the model will be used in subsequent evaluation models to provide a more comprehensive assessment of the driver's status.

[0121] Based on the driver's own real-time evaluation system, we deeply explore their subjective perception and objective operation feedback, and build an evaluation mechanism closely around the user's daily driving habits, style and actual scenarios. Based on this, we carry out targeted model training, so that the intelligent driving model can accurately match the user's personalized driving mode, calmly deal with complex and changeable daily road conditions, continuously improve the model's practicality and adaptability, and enhance driving safety and experience.

[0122] In a specific application, the positions of steps S306, S307 and step S305 may be arbitrarily exchanged without affecting the effective implementation of the solution.

[0123] Step S307: input the driver's corresponding voice data, driver emotion score and fatigue score into the evaluation model, and output the evaluation label.

[0124] Specifically, the evaluation model is used to comprehensively analyze the driver's voice data, emotion score and fatigue score to generate an evaluation label for the current vehicle driving status. The evaluation label is used to evaluate the current vehicle operation status (the current vehicle operation status refers to whether there is a risk in the vehicle operation at this time or whether the vehicle's control in this scenario meets expectations), and instantly and accurately outputs evaluation information (good or bad) to the intelligent driving domain, injecting key references into subsequent decisions.

[0125] If the driver is satisfied with the vehicle operating status output by the current target model (whether the driver is satisfied with the vehicle operating status can be identified through facial expressions and voice input, etc.), then it is marked as a good case (or positive label), and subsequent data collection and labeling are carried out.

[0126] If the driver is not satisfied with the vehicle operating status output by the current target model, it is marked as a bad case (or negative label) scenario, and data collection and corresponding annotation are performed subsequently.

[0127] The input data of the evaluation model needs to be standardized and feature extracted to ensure data compatibility and model stability.

[0128] In some embodiments, the evaluation model generally uses machine learning techniques, such as support vector machines (SVM) or random forests, to improve the accuracy and robustness of classification. The output of the model will be used to guide the vehicle's autonomous driving strategy adjustment and safety warning mechanism.

[0129] Step S308: Determine the evaluation tag corresponding to the vehicle driving data based on the timestamp of the vehicle driving data and the driver status data corresponding to the evaluation tag.

[0130] Specifically, the vehicle driving data can be matched with the corresponding evaluation tag through the timestamp to determine the driving status evaluation at each moment.

[0131] In the specific implementation, the collected data needs to be time synchronized to eliminate the time deviation between different data sources, and then the vehicle driving data is associated with the corresponding evaluation label according to the timestamp to generate a complete data set.

[0132] Compared with traditional intelligent driving model training that relies on manual screening and collection of vehicle operation extreme scenario data, which requires large manpower investment, is time-consuming, costly and inefficient, this solution relies on the evaluation labels extracted in the previous steps and combined with real-time collected vehicle driving data to automatically complete data labeling without large-scale manual intervention, greatly reducing the cost of collecting data from different scenarios and different user feedback, and freeing up R&D resources.

[0133] Step S309: Based on the evaluation labels, the vehicle driving data is divided into two categories to obtain a model training data set corresponding to each category of evaluation labels.

[0134] Specifically, according to the corresponding evaluation labels, the vehicle driving data can be divided into two categories: positive evaluation and negative evaluation.

[0135] The positive evaluation dataset is used to train the model’s normal driving status, while the negative evaluation dataset is used to identify and handle abnormal driving status.

[0136] The classification of data needs to take into account the accuracy and representativeness of the evaluation labels to ensure the diversity and coverage of the training data. The classified datasets will be used for different model training strategies to improve the generalization ability and robustness of the model.

[0137] Step S310: input the model training data set corresponding to each type of evaluation label into the target model respectively, train the target model, and obtain a trained target model.

[0138] Specifically, by accurately dividing the collected scene data into good and bad categories based on driving effects (driver evaluation labels), focusing on extreme characteristic scenarios (such as vehicle driving data corresponding to scenarios with negative driver evaluations), and conducting targeted special training for large models (inputting good and bad data into the model for training, so that the model also knows what is good and what is bad), the model's decision-making logic and response accuracy in complex and rare scenarios are deeply optimized to enhance overall robustness.

[0139] In some embodiments, the trained target model can be tailored and adapted as needed, and can be robustly deployed to the vehicle control system, seamlessly embedded in the driving process, and continuously iterated to improve the comprehensive capabilities of intelligent driving, thereby enhancing the safety and user experience of the entire driving process.

[0140] Therefore, this solution deeply mines and screens data resources that match the vehicle driver's unique driving habits and style, and then relies on these exclusive data to complete targeted model training and optimization upgrades, so that the functional model of the autonomous driving vehicle can closely fit the individual driving characteristics, and comprehensively improve the safety, stability and user experience of autonomous driving.

[0141] The model training method provided in this embodiment significantly improves the efficiency and accuracy of autonomous driving model training by comprehensively collecting and analyzing the driver's voice and expression data and combining it with vehicle driving data. First, through the multimodal fusion of voice and expression data, it is possible to more comprehensively capture the driver's emotions and fatigue status, providing the model with richer training data. Secondly, the evaluation labels are automatically generated using a pre-trained evaluation model, which reduces the reliance on the subjective judgment of R&D engineers and improves the objectivity and consistency of data annotation. In addition, by matching the vehicle driving data with the evaluation labels by timestamp, the accuracy and relevance of the training data are ensured, which not only improves the generalization ability of the target model in complex driving scenarios, but also enhances the ability to recognize abnormal driving conditions, thereby improving the safety and user experience of the autonomous driving system.

[0142] Figure 4 This is a schematic diagram of the structure of the model training device provided in the embodiment of the present application. Figure 4 As shown, the model training device 400 includes: a collection module 410, an analysis module 420, a processing module 430 and a training module 440;

[0143] The acquisition module 410 is used to collect driver status data and vehicle driving data, and the driver status data includes voice data and expression data;

[0144] The analysis module 420 is used to input the driver state data into the pre-trained evaluation model and output the driver's evaluation label for the current vehicle driving state, where the evaluation label includes a positive evaluation label and a negative evaluation label;

[0145] A processing module 430 is used to combine the evaluation label with the corresponding vehicle driving data to obtain a model training data set;

[0146] The training module 440 is used to input the model training data set into the target model, train the target model, and obtain a trained target model.

[0147] In some embodiments, the acquisition module 410 is specifically used to collect in-vehicle voice data based on a voice sensor; extract the driver's voice data from the in-vehicle voice data based on a voice processing algorithm; collect the driver's corresponding facial image data based on an image sensor; extract expression data from the driver's corresponding facial image data based on an image processing algorithm; and extract vehicle driving data received in the vehicle control system.

[0148] In some embodiments, the acquisition module 410 specifically includes: a voice sensor array arranged in a position adjacent to the driver in the vehicle; at least two image sensors, which are located in different positions; and the image sensor includes an infrared fill light source.

[0149] In some embodiments, the acquisition module 410 is specifically used to filter the in-vehicle voice data; extract feature information of the filtered in-vehicle voice data, the feature information including intonation, speaking speed and volume; convert the filtered voice data into text data; and extract the voice data corresponding to the driver from the in-vehicle voice data based on the feature information and text data.

[0150] In some embodiments, the acquisition module 410 is specifically used to extract the eyebrow image, mouth image and eye image from the driver's corresponding facial image data; input the eyebrow image, mouth image and eye image into the feature extraction algorithm to extract the driver's corresponding expression data.

[0151] In some embodiments, the acquisition module 410 is also used to input the eyebrow image, mouth image and eye image into the feature extraction algorithm, and after extracting the corresponding expression data of the driver, input the expression data into a pre-trained emotion analysis model to output the driver's emotion score and fatigue score corresponding to the expression data; the analysis module 420 is also used to input the corresponding voice data, driver emotion score and fatigue score of the driver into the evaluation model, and output the evaluation label.

[0152] In some embodiments, the processing module 430 is specifically used to determine the evaluation label corresponding to the vehicle driving data based on the timestamp of the vehicle driving data and the driver status data corresponding to the evaluation label; based on the evaluation label, the vehicle driving data is divided into two categories to obtain a model training data set corresponding to each category of evaluation labels; the training module 440 is specifically used to input the model training data set corresponding to each category of evaluation labels into the target model respectively, train the target model, and obtain a trained target model.

[0153] The model training device provided in the embodiment of the present application can be used to execute the technical solution of the model training method in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0154] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the acquisition module 410 can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above acquisition module 410. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0155] Figure 5 A schematic diagram of the structure of a model training system provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the model training system 500 includes: an image sensor 510, a voice sensor 520, and a controller 530.

[0156] The controller 530 is connected to the image sensor 510 and the voice sensor 520, and executes the model training method shown in the above embodiment;

[0157] The image sensor 510 is used to collect the driver's voice data, and the image sensor 520 is used to collect the driver's expression data.

[0158] The relevant instructions can be understood by referring to the relevant descriptions and effects corresponding to the method embodiments, which will not be repeated here.

[0159] An embodiment of the present application provides a vehicle, which includes the model training system provided in the above embodiments.

[0160] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0161] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0162] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0163] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0164] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0165] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0166] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0167] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0168] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A model training method, characterized in that: include: Collecting driver status data and vehicle driving data, wherein the driver status data includes voice data and expression data; Inputting the driver status data into a pre-trained evaluation model, and outputting the driver's evaluation label for the current vehicle driving status, wherein the evaluation label includes a positive evaluation label and a negative evaluation label; Combining the evaluation label with the corresponding vehicle driving data to obtain a model training data set; The model training data set is input into the target model, and the target model is trained to obtain a trained target model.

2. The method according to claim 1, characterized in that The collecting of driver status data and vehicle driving data includes: Based on voice sensors, collect voice data in the car; Extracting the driver's voice data from the in-vehicle voice data based on a voice processing algorithm; Based on the image sensor, the corresponding facial image data of the driver is collected; Extracting expression data from the driver's corresponding facial image data based on an image processing algorithm; Extract vehicle driving data received from the vehicle control system.

3. The method according to claim 2, characterized in that The voice sensor array is arranged in a position near the driver in the vehicle; There are at least two image sensors located at different positions; the image sensor includes an infrared fill light source.

4. The method according to claim 2, characterized in that: The extracting the driver's voice data from the in-vehicle voice data based on the voice processing algorithm includes: Filtering the in-vehicle voice data; Extracting feature information of the filtered in-vehicle voice data, wherein the feature information includes intonation, speech speed and volume; Convert the filtered voice data into text data; Based on the feature information and the text data, voice data corresponding to the driver is extracted from the in-vehicle voice data.

5. The method according to claim 4, characterized in that The step of extracting expression data from the facial image data corresponding to the driver based on an image processing algorithm includes: Extracting an eyebrow image, a mouth image and an eye image from the facial image data corresponding to the driver; The eyebrow image, mouth image and eye image are input into a feature extraction algorithm to extract the expression data corresponding to the driver.

6. The method according to claim 5, characterized in that After inputting the eyebrow image, mouth image and eye image into a feature extraction algorithm to extract the expression data corresponding to the driver, the method further includes: Inputting the expression data into a pre-trained emotion analysis model, and outputting the driver's emotion score and fatigue score corresponding to the expression data; The step of inputting the driver status data into a pre-trained evaluation model and outputting the driver's evaluation label of the current vehicle driving status includes: The voice data corresponding to the driver, the driver's emotion score and fatigue score are input into the evaluation model, and the evaluation label is output.

7. The method according to any one of claims 1 to 6, characterized in that The step of combining the evaluation label with the corresponding vehicle driving data to obtain a model training data set includes: Determining the evaluation tag corresponding to the vehicle driving data based on the timestamp of the vehicle driving data and the driver status data corresponding to the evaluation tag; Based on the evaluation labels, the vehicle driving data is divided into two categories to obtain a model training data set corresponding to each category of evaluation labels; The step of inputting the model training data set into the target model, training the target model, and obtaining a trained target model comprises: The model training data set corresponding to each type of evaluation label is respectively input into the target model, and the target model is trained to obtain a trained target model.

8. A model training device, characterized in that: include: A collection module, used to collect driver status data and vehicle driving data, wherein the driver status data includes voice data and expression data; An analysis module, used to input the driver state data into a pre-trained evaluation model, and output the driver's evaluation label for the current vehicle driving state, wherein the evaluation label includes a positive evaluation label and a negative evaluation label; A processing module, used to combine the evaluation label with the corresponding vehicle driving data to obtain a model training data set; The training module is used to input the model training data set into the target model, train the target model, and obtain a trained target model.

9. A model training system, characterized in that: include: Image sensor, voice sensor, controller; The controller is connected to the image sensor and the voice sensor, and executes the method according to any one of claims 1 to 7; The image sensor is used to collect the driver's voice data, and the image sensor is used to collect the driver's expression data.

10. A vehicle, characterized in that: The vehicle includes the model training system described in claim 9.

Images