Driving intervention method, device and equipment based on digital twinning and vehicle

Abstract

The invention discloses a driving intervention method, device and equipment based on digital twinning and a vehicle, and relates to the technical field of vehicles. The digital twinning-based driving intervention method comprises the following steps: acquiring a biological characteristic signal of a driver, wherein the biological characteristic signal is used for representing characteristics of the driver in different physiological and psychological states; generating a digital twin of the driver based on the biometric signal; the digital twin is monitored through a pattern recognition model, monitoring result information is obtained, and the pattern recognition model is used for extracting key features of the biological feature signals and monitoring the fatigue degree, emotion and health state of the digital twin; and performing active intervention on the driving process according to the monitoring result information. According to the embodiment of the invention, the digital twin of the driver is generated through the biological characteristic signal, a new thought is provided for monitoring the state of the driver, the monitoring accuracy and real-time performance of the driver can be improved, and then the driving safety of the vehicle is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a driving intervention method, device, equipment and vehicle based on digital twins. Background Art

[0002] The driver's state is crucial to the driving safety of the car. The driver's attention, emotions and physical condition directly affect driving behavior and his response to the road environment. For example, fatigue driving, alcoholism or drug influence can significantly reduce the driver's reaction speed and judgment, thereby increasing the risk of accidents. In addition, the driver's psychological state such as anxiety and stress can also lead to unstable driving behavior.

[0003] Driver monitoring systems and vehicle assistance systems that monitor the driver's status can help identify and correct potential driving risks, thereby reducing the occurrence of traffic accidents.

[0004] The driver monitoring system in related technologies cannot accurately monitor the driver's status, resulting in the vehicle assistance system being unable to accurately and timely intervene in driving risks, which has a certain impact on driving safety. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to provide a driving intervention method, device, equipment and vehicle based on digital twins, aiming to improve the accuracy of driver monitoring to improve the driving safety of the vehicle.

[0006] To achieve the above objectives, one aspect of an embodiment of the present application proposes a driving intervention method based on digital twins, comprising the following steps:

[0007] Acquire a biometric signal of the driver, wherein the biometric signal is used to characterize the characteristics of the driver in different physiological and psychological states;

[0008] generating a digital twin of the driver based on the biometric signal;

[0009] Monitoring the digital twin through a pattern recognition model to obtain monitoring result information, wherein the pattern recognition model is used to extract key features of the biometric signal and monitor the fatigue level, emotion and health status of the digital twin;

[0010] Actively intervene in the driving process based on the monitoring result information.

[0011] In some embodiments, the biometric signal includes an electroencephalogram signal, an electromyography signal, a facial feature, and an exhalation signal of the driver, and generating a digital twin of the driver based on the biometric signal includes:

[0012] A digital twin of the driver is generated based on the driver's electroencephalogram signal, the electromyography signal, the facial features, and the exhalation signal.

[0013] In some embodiments, the digital twin is monitored by a pattern recognition model to obtain monitoring result information, including:

[0014] Extracting key features of the biometric signal respectively by using a plurality of feature extraction methods;

[0015] Performing feature fusion on the key features to obtain feature fusion results;

[0016] The feature fusion results are analyzed by using a plurality of monitoring algorithms to obtain monitoring result information of the fatigue level, monitoring result information of the emotion and monitoring result information of the health status.

[0017] In some embodiments, the biometric signal includes an electroencephalogram signal, an electromyography signal, a facial feature, and an exhalation signal of the driver, and the key features of the biometric signal are extracted by using a plurality of feature extraction methods, including:

[0018] Extract key features of the EEG signal and the EMG signal using a wavelet transform method;

[0019] Extracting key features of the facial features using a histogram of oriented gradients method;

[0020] The key features of the exhalation signal are extracted using the average method.

[0021] In some embodiments, the analyzing the feature fusion results by using a plurality of monitoring algorithms includes:

[0022] Monitoring the fatigue level of the digital twin using a back propagation neural network model;

[0023] Monitoring the emotions of the digital twin using a support vector machine model combined with a particle swarm optimization algorithm;

[0024] The health status of the digital twin is monitored using a random forest model combined with a particle swarm optimization algorithm.

[0025] In some embodiments, the monitoring result information includes monitoring result information of fatigue level, and the actively intervening in the driving process according to the monitoring result information includes:

[0026] Obtaining monitoring result information of the fatigue level;

[0027] According to the monitoring result information of the fatigue level, one or more modules among the seat adjustment control module, the audio control module and the display module of the vehicle are controlled to provide corresponding functional services.

[0028] In some embodiments, the monitoring result information includes emotion monitoring result information, and the actively intervening in the driving process according to the monitoring result information includes:

[0029] Obtaining monitoring result information of the emotion;

[0030] According to the emotion monitoring result information, one or more modules among the vehicle's audio control module, lighting control module, temperature control module, powertrain control module or auxiliary driving module are controlled to provide corresponding functional services.

[0031] In some embodiments, the monitoring result information includes monitoring result information of health status, and the actively intervening in the driving process according to the monitoring result information includes:

[0032] Obtaining monitoring result information of the health status;

[0033] According to the monitoring result information of the health status, one or more of the vehicle's automatic braking control module, auxiliary driving module, vehicle alarm system, on-board communication module, positioning system, telematics system or health detection module are controlled to provide corresponding functional services.

[0034] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a driving intervention device based on digital twins, and the driving intervention device based on digital twins includes:

[0035] An acquisition module, used to acquire a biometric signal of the driver, wherein the biometric signal is used to characterize the characteristics of the driver in different physiological and psychological states;

[0036] A generation module, configured to generate a digital twin of the driver based on the biometric signal;

[0037] A monitoring module, used to monitor the digital twin through a pattern recognition model to obtain monitoring result information, wherein the pattern recognition model is used to extract key features of the biometric signal and monitor the fatigue level, mood and health status of the digital twin;

[0038] The active intervention module is used to actively intervene in the driving process according to the monitoring result information.

[0039] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the driving intervention method based on digital twins of the above-mentioned embodiment is implemented.

[0040] To achieve the above objectives, another aspect of an embodiment of the present application proposes a vehicle, which includes a driving intervention device based on digital twins of the above embodiment or an electronic device of the above embodiment.

[0041] The embodiments of the present application include at least the following beneficial effects:

[0042] The present application provides a driving intervention method, device, equipment and vehicle based on digital twins. In the embodiments of the present application, first, the driver's biometric signal is obtained, and the biometric signal is used to characterize the characteristics of the driver under different physiological and psychological states; then, the driver's digital twin is generated based on the biometric signal; then, the digital twin is monitored through a pattern recognition model to obtain monitoring result information, wherein the pattern recognition model is used to extract the key features of the biometric signal and monitor the fatigue level, emotion and health status of the digital twin; finally, the driving process is actively intervened according to the monitoring result information. The embodiment of the present application generates the driver's digital twin through biometric signals, which provides a new idea for monitoring the driver's status. The monitoring of the fatigue level, emotion and health status of the digital twin through the pattern recognition model can improve the accuracy and real-time performance of driver monitoring, and active intervention in the driving process according to the monitoring result information can improve the driving safety of the vehicle.

[0043] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0045] Figure 1 is a flowchart of a driving intervention method based on digital twins provided in some embodiments of the present application;

[0046] Figure 2 is a flowchart of a method for monitoring a digital twin through a pattern recognition model provided by some embodiments of the present application;

[0047] Figure 3 is a flow chart of a method for actively intervening in a driving process according to monitoring result information provided by some embodiments of the present application;

[0048] Figure 4 is a schematic diagram of a system framework of a driving intervention method based on digital twins provided in some embodiments of the present application;

[0049] Figure 5 is a schematic block diagram of a module of a driving intervention device based on digital twins provided in some embodiments of the present application;

[0050] Figure 6 It is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the reference to "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of this phrase in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0052] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0053] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0056] In order to make the inventive concept of the present application easy to understand, before describing the embodiments of the present application in detail, the English abbreviations (terms) / related concepts involved in the embodiments of the present application are first described. The English abbreviations (terms) / related concepts involved in the embodiments of the present application are subject to the following explanations.

[0057] Digital twin technology: Digital twin refers to a virtual model of a physical object, system or process created through digital technology. This model can reflect the state and behavior of its physical counterpart in real time, and can simulate, predict and optimize the physical counterpart. Through data collection and analysis, digital twins can provide real-time monitoring and early warning, which is helpful for system maintenance and improvement. Digital twins connect the physical world and the information world, providing a new direction for drivers to monitor abnormal conditions during driving.

[0058] The driver's state is crucial to the safety of automobile driving. The driver's attention, emotions and physical condition directly affect driving behavior and his response to the road environment. For example, fatigue driving, alcoholism or drug influence can significantly reduce the driver's reaction speed and judgment, thereby increasing the risk of accidents. In addition, the driver's psychological state such as anxiety and stress may also lead to unstable driving behavior. In order to improve road safety, it is key to understand and monitor the driver's state information during driving. Driver monitoring systems and vehicle assistance systems can help identify and correct potential driving risks, thereby reducing the occurrence of traffic accidents.

[0059] At present, the monitoring of the driver's state has received attention from all parties. Among them, the research review on the detection and mitigation of driving fatigue in the human-machine co-driving environment summarizes the current methods for detecting the fatigue state of automobile drivers, including questionnaire-based subjective detection methods, detection methods based on physiological signals, detection methods based on driver behavior characteristics, and detection methods based on driver facial features. The research review on the detection and mitigation of driving fatigue in the human-machine co-driving environment pointed out that the current methods for detecting driver emotions are mainly based on physiological signals, including electroencephalogram, electromyogram, electrocardiogram eye movement information, skin electrical activity, etc. The driver's cerebrovascular disease identification model based on the PPG pulse signal feature value proposed that the driver's possible cardiovascular and cerebrovascular health can be determined based on the pulse signal response. However, current research only collects part of the driver's information for driver status monitoring, resulting in low monitoring accuracy and failure to fully guarantee the safety of automobile driving.

[0060] The driver monitoring system in the related technology cannot accurately monitor the driver's status, resulting in the vehicle assistance system being unable to accurately and timely intervene in driving risks, which has a certain impact on driving safety.

[0061] In view of this, the present application proposes a driving intervention method, device, equipment and vehicle based on digital twins. The scheme obtains the driver's biometric signal, which is used to characterize the characteristics of the driver in different physiological and psychological states; then, the driver's digital twin is generated based on the biometric signal; then, the digital twin is monitored through a pattern recognition model to obtain monitoring result information, wherein the pattern recognition model is used to extract the key features of the biometric signal and monitor the fatigue level, emotion and health status of the digital twin; finally, the driving process is actively intervened according to the monitoring result information. The embodiment of the present application generates a digital twin of the driver through biometric signals, which provides a new idea for monitoring the driver's status. The monitoring of the fatigue level, emotion and health status of the digital twin by the pattern recognition model can improve the accuracy and real-time performance of driver monitoring. Active intervention in the driving process according to the monitoring result information can improve the driving safety of the vehicle.

[0062] Furthermore, the driver's digital twin constructed through digital twin technology can realize real-time analysis of the driver's status, generate real-time data, and accurately monitor the driver's various states through real-time data and algorithm models, so as to take measures in advance when the driver has dangerous behaviors or health problems. In addition, the digital twin method can also record the driver's behavior data, which can be used to analyze and improve the driver's driving habits, improve driving safety, and realize intelligent vehicle services.

[0063] The method provided in the embodiment of the present application can be applied to the electronic device provided in the embodiment of the present application, wherein the electronic device can be a terminal, a server capable of communicating with a vehicle, or a vehicle-mounted device in the vehicle.

[0064] The terminal may be a tablet computer, a laptop computer, a car computer, etc., but is not limited thereto.

[0065] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, as well as big data and artificial intelligence platforms.

[0066] For example, a vehicle intervention system integration module can be designed in the vehicle system to identify dangerous driving behaviors of the driver and actively intervene in driving, thereby improving the driving safety of the car. For example, the vehicle intervention system integration module can collect the driver's EEG signals, EMG signals, facial features and exhalation features during driving, and build a digital twin of the driver in the car in combination with intelligent algorithms. The vehicle intervention system integration module uses feature extraction and data fusion technology to process the acquired driver information, monitors the driver's digital twin in multiple states through a pattern recognition model, and provides real-time feedback on the monitoring results of the driver's digital twin in multiple states. Based on the monitoring results, the vehicle intelligent algorithm is used to implement the intervention of the car's automatic driving.

[0067] The following will describe in detail the implementation steps of a driving intervention method based on digital twins provided in an embodiment of the present application in conjunction with the accompanying drawings.

[0068] Please refer to Figure 1 , Figure 1 A flowchart of a driving intervention method based on digital twins is provided for some embodiments of the present application. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0069] The method of the embodiment of the present application comprises the following steps:

[0070] Step 101: Acquire a driver's biometric signal, where the biometric signal is used to characterize the driver's characteristics under different physiological and psychological states;

[0071] Step 102: Generate a digital twin of the driver based on the biometric signal;

[0072] Step 103: Monitor the digital twin through a pattern recognition model to obtain monitoring result information, wherein the pattern recognition model is used to extract key features of the biometric signal and monitor the fatigue level, emotion and health status of the digital twin;

[0073] Step 104: Actively intervene in the driving process based on the monitoring result information.

[0074] Steps 101 to 104 shown in the embodiment of the present application generate a digital twin of the driver through biometric signals, which provides a new idea for monitoring the driver's status. The monitoring of the digital twin's fatigue level, emotions and health status through pattern recognition models can improve the accuracy and real-time performance of driver monitoring. Active intervention in the driving process based on the monitoring result information can improve the driving safety of the vehicle.

[0075] The specific implementation methods of the above steps are introduced below.

[0076] In step 101 , a biometric signal of a driver is acquired, where the biometric signal is used to characterize the characteristics of the driver in different physiological and psychological states.

[0077] Optionally, the biometric signal may include the driver's EEG signal, EMG signal, facial features, and exhalation feature signal. It should be understood that these signals can characterize the driver's characteristics under different physiological and psychological states. For example, EEG signals can reflect the activity state of the brain, EMG signals can reflect the activity state of the muscles, facial features can reflect the person's expression and emotional state, and exhalation features can reflect the person's breathing state and health status. By analyzing these feature signals, a deeper understanding and analysis of the driver's psychological and physiological characteristics can be performed.

[0078] Specifically, for electroencephalogram (EEG), when the driver is in a relaxed state, alpha waves (frequency of 8-12Hz) will appear in the EEG, indicating that the brain is in a relatively calm and relaxed state; when the driver is in a focused state, beta waves (frequency of 13-30Hz) will appear in the EEG, indicating that the brain is active and alert; when the driver is fatigued, the overall activity of the EEG signal may decrease, the synchronization between EEG signals in different brain regions may decrease, and the spectrum of the EEG signal may change, manifested as an increase in low-frequency components (such as delta waves and theta waves) and a decrease in high-frequency components (such as beta waves).

[0079] For electromyographic signals (EMG), when muscles are tense or contracted, EMG signals will show higher levels of electrical activity; when muscles are relaxed, EMG signals will show lower levels of electrical activity; after long-term muscle use, EMG signals may show fatigue characteristics, such as a decrease in signal amplitude and changes in frequency.

[0080] As for facial features, when under stress, the face may show certain expressions, such as tight brows or tightly closed lips; when tired for a long time, the face may have droopy eyelids, dull eyes and other features.

[0081] Regarding exhalation characteristics, the breathing rate may increase when you are nervous or anxious; when you are relaxed, your breathing may be deeper and slower; certain physiological or psychological states may affect the composition of exhaled gas, such as anxiety may cause an increase in the concentration of exhaled carbon dioxide. In addition, the analysis of exhalation characteristics can be used to monitor respiratory diseases or psychological stress.

[0082] Optionally, the driver's electroencephalogram signals, electromyographic signals, facial features, and exhalation characteristic signals may be acquired and stored in a database.

[0083] Exemplarily, EEG signals and EMG signals can be collected through an electroencephalogram (EEG) device and an electromyogram (EMG) device, the driver's facial features can be captured using a high-resolution camera, the driver's breath can be monitored using a gas sensor array, and sensor response data can be collected.

[0084] The embodiment provided in the present application provides data support for subsequent driver status monitoring by acquiring the driver's biometric signals, and also provides a basis for generating a digital twin of the driver.

[0085] In step 102 , a digital twin of the driver is generated based on the biometric signal.

[0086] In some embodiments, a digital twin corresponding to the driver may be generated through the driver's biometric signal. For example, the following is a possible implementation step:

[0087] Data collection: First, it is necessary to collect the driver's biometric signals, such as heart rate, blood pressure, brain waves, electromyography, etc. These signals can be obtained through various sensors and transmitted to the computer for processing.

[0088] Data preprocessing: Preprocess the collected biometric signals, including filtering, denoising, feature extraction, etc. The purpose of this step is to remove noise and interference in the signal and extract useful feature information.

[0089] Model building: Based on the preprocessed biometric signals, a digital twin model of the driver is built. This model can be a physics-based model or a data-based model. For example, a machine learning algorithm can be used to train a model to associate biometric signals with the driver's behavior or status.

[0090] Model Validation: The established digital twin model is validated to ensure that it accurately reflects the driver’s real-world status. This can be achieved by comparing it with the actual driver’s behavior.

[0091] In some embodiments of the present application, the biometric signals may include the driver's EEG signals, EMG signals, facial features, and exhalation signals, and a digital twin of the driver may be generated based on the driver's EEG signals, EMG signals, facial features, and exhalation signals.

[0092] For example, the preprocessed EEG signals, EMG signals, facial features, and exhalation signals can be fused to form a comprehensive feature vector. This can be achieved through a variety of methods, such as weighted average, principal component analysis, etc. Based on the fused feature vector, a digital twin model of the driver is established. This can be achieved through a variety of methods, such as a physics-based model, a data-based model, etc.

[0093] Physics-based models can be implemented by establishing a physical relationship between biometric signals and the driver's physiological state. For example, EEG signals can reflect the driver's nervousness, and EMG signals can reflect the driver's fatigue, etc.

[0094] Data-based models can be trained through machine learning algorithms, which use biometric signals as input and the driver's physiological state or behavior as output, and establish a mapping relationship between the two by learning a large amount of data. This type of model usually does not require a deep understanding of the generation mechanism of biometric signals, but rather builds the model in a data-driven way.

[0095] It should be understood that the appropriate model type can be selected according to specific needs and scenarios. For example, the two models can be combined to establish a more comprehensive and accurate digital twin model.

[0096] The embodiment of the present application provides a new direction for monitoring abnormal conditions of the driver during driving by generating a digital twin of the driver based on biometric signals. The digital twin can provide real-time monitoring and early warning, facilitate accurate monitoring of the driver's various conditions, and take measures in advance when the driver exhibits dangerous behaviors or health problems, thereby improving driving safety.

[0097] In step 103, the digital twin is monitored through a pattern recognition model to obtain monitoring result information, wherein the pattern recognition model is used to extract key features of biometric signals and monitor the fatigue level, emotion and health status of the digital twin.

[0098] Optionally, a pattern recognition model can be built that is able to extract key features from the biometric signal. This can be achieved through machine learning algorithms such as support vector machines, decision trees, neural networks, etc.

[0099] Afterwards, through the key features extracted by the pattern recognition model, a variety of monitoring algorithms can be used to monitor the fatigue level of the digital twin. For example, heart rate variability analysis can be used to assess the driver's cardiac fatigue, or electromyography analysis can be used to assess the driver's muscle fatigue.

[0100] Similarly, key features extracted by pattern recognition models can be used to monitor the emotional state of the digital twin. For example, a facial expression recognition algorithm can be used to analyze the driver’s facial features to determine his or her emotional state, such as anger, sadness, happiness, etc.

[0101] In addition, key features extracted by pattern recognition models can be used to monitor the health status of the digital twin. For example, breathing signal analysis can be used to assess the driver's respiratory health, or EEG signal analysis can be used to assess the driver's brain health.

[0102] Through the above monitoring algorithm, the digital twin’s fatigue level, emotions and health status can be monitored. This information can be used to evaluate the driver’s status and take corresponding measures, such as adjusting driving strategies and issuing warnings.

[0103] In some embodiments, the digital twin is monitored through a pattern recognition model to obtain monitoring result information, which can be to extract key features of biometric signals separately through multiple feature extraction methods; perform feature fusion on the key features to obtain feature fusion results; and analyze the feature fusion results through multiple monitoring algorithms to obtain monitoring result information on fatigue level, monitoring result information on emotions, and monitoring result information on health status.

[0104] Optionally, a variety of feature extraction methods can be used to extract key features of biometric signals. These methods may include time domain feature extraction, frequency domain feature extraction, time-frequency domain feature extraction, etc. For example, for EEG signals, their time domain features such as mean, variance, standard deviation, etc. can be extracted; for EMG signals, their frequency domain features such as power spectral density, etc. can be extracted.

[0105] The extracted key features are subjected to feature fusion to obtain feature fusion results. Feature fusion can be achieved through a variety of methods, such as weighted average, principal component analysis, singular value decomposition, etc. Feature fusion can improve the accuracy and reliability of monitoring.

[0106] A variety of monitoring algorithms can be used to analyze the feature fusion results to obtain monitoring results information of fatigue level, emotion and health status. These monitoring algorithms can include support vector machines, decision trees, neural networks, etc. For example, for fatigue level monitoring, we can use support vector machine algorithms to analyze feature fusion results to determine the driver's fatigue level.

[0107] Through the above monitoring algorithm, the monitoring result information of fatigue level, emotion and health status can be obtained. This information can be used to evaluate the driver's status and take corresponding measures, such as adjusting driving strategy, issuing warnings, etc.

[0108] Please refer to Figure 2 , Figure 2 It is a flow chart of a method for monitoring a digital twin through a pattern recognition model provided by some embodiments of the present application. In some embodiments of the present application, the biometric signal may include the driver's EEG signal, EMG signal, facial features and exhalation signal, and the key features of the biometric signal may be extracted by a plurality of feature extraction methods, such as using a wavelet transform method to extract the key features of the EEG signal and the EMG signal; using a directional gradient histogram method to extract the key features of the facial features; and using an average value method to extract the key features of the exhalation signal.

[0109] For example, for EEG signals and EMG signals, the main features thereof can be collected using the wavelet transform method, as shown in formula (1):

[0110]

[0111] Among them, W x (τ,σ) are wavelet coefficients, τ is the translation parameter, σ is the scaling parameter, is the conjugate complex number of the mother wavelet, and t is a variable.

[0112] For the driver's facial features, the directional gradient histogram (HOG) method can be used for feature extraction. The method mainly includes gradient calculation, creation of directional histogram and normalization. The calculation steps are as follows:

[0113] Gradient calculation:

[0114] G x =I(x+1,y)-I(x-1,y) (2)

[0115] G y =I(x,y+1)-I(x,y-1) (3)

[0116] Where: I(x,y) is the gray value of the image, G xis the horizontal gradient, G y Vertical grayscale.

[0117] Gradient magnitude and gradient direction calculation:

[0118]

[0119] Among them: Mag is the gradient magnitude, Ori is the gradient direction.

[0120] Direction histogram and normalization:

[0121] The driver's facial image is divided into small cells. In each small cell, the histogram of each direction range (bins) is calculated and normalized, see formula (6).

[0122]

[0123] Among them: His is the value of the histogram of each direction range in the cell, ∈ is a very small constant, and Norm is the normalized value.

[0124] For the driver's breath characteristics, the average value method can be used to process each sensor in the sensor array. The calculation formula is as follows:

[0125]

[0126] Where: i is the i-th sensor, x(t) is the sensor response value.

[0127] Afterwards, the extracted features can be fused for input into subsequent models. For example, principal component analysis can be used as a method for data fusion.

[0128] In some embodiments, the feature fusion results are analyzed through multiple monitoring algorithms, such as using a back-propagation neural network model to monitor the fatigue level of the digital twin; using a support vector machine model combined with particle swarm optimization to monitor the emotions of the digital twin; and using a random forest model combined with particle swarm optimization to monitor the health status of the digital twin.

[0129] For example, for monitoring the fatigue level of the driver's digital twin, a back-propagation neural network model can be used to monitor whether the driver is not tired, slightly tired, or severely tired.

[0130] For the emotional monitoring of the driver’s digital twin, a support vector machine model can be used to realize the recognition of emotions, including emotional stability, road rage, tension, and anxiety. The particle swarm optimization algorithm is used to optimize the parameters of the support vector machine to achieve a more accurate recognition effect.

[0131] For disease monitoring of the driver’s digital twin, the random forest model is used to monitor and identify whether the driver has a sudden illness (myocardial infarction or epilepsy), and the particle swarm optimization algorithm is used to optimize the parameters in the random forest model.

[0132] The embodiment provided in this application monitors the digital twin through a pattern recognition model to obtain monitoring result information, which provides a basis for subsequent active intervention in the vehicle through the monitoring result information. That is, when the driver's digital twin is detected to be in a state that is not conducive to safe driving, the state of the twin at this time is fed back to the vehicle in real time, and the vehicle intervention system integration module takes corresponding intervention measures after identifying the dangerous driving state.

[0133] In step 104, active intervention is performed on the driving process based on the monitoring result information.

[0134] Optionally, the monitoring result information may include monitoring result information of fatigue level, monitoring result information of emotion and monitoring result information of health status.

[0135] Please refer to Figure 3 , Figure 3 This is a flowchart of a method for actively intervening in the driving process according to monitoring result information provided by some embodiments of the present application. Active intervention in the driving process can be performed based on the monitoring result information of the degree of fatigue, the monitoring result information of the emotion, and the monitoring result information of the health status. It should be understood that if the monitoring results show that the driver's digital twin is not currently fatigued, emotionally stable, or healthy, no operation is performed.

[0136] Optionally, the vehicle intervention system integration module can make corresponding intervention measures after identifying dangerous driving conditions, which is mainly connected to some modules through the central control unit of the intervention system integration module, including the audio control module, seat adjustment control module, automatic brake control module, vehicle auxiliary driving module, instrument panel display, lights and warning lights, GPS module, communication module and network interface, etc. By controlling these modules, the vehicle intervention system integration module can realize active intervention in vehicle driving.

[0137] In some embodiments, the monitoring result information of the fatigue degree may be obtained, including no fatigue, mild fatigue and severe fatigue. According to the monitoring result information of the fatigue degree, one or more modules of the seat adjustment control module, the audio control module and the display module of the vehicle are controlled to provide corresponding functional services.

[0138] For example, if the monitoring result is mild fatigue, the central control unit of the intervention system integration module controls the dashboard display screen to pop up a warning icon or message to remind the driver to pay attention to the fatigue state. The seat adjustment control module controls the seat motor to automatically adjust the seat angle to improve the driver's comfort and help the driver stay awake.

[0139] If the monitoring result is severe fatigue, the central control unit of the intervention system integration module will issue a command to the audio control module, emitting a high-intensity sound alarm, forcing the driver to pay attention and take measures immediately, and recommending the driver to stop and rest. The dashboard display screen will be controlled to flash the warning icon and emergency reminder, and the vehicle assisted driving mode will be turned on. It is forced to recommend stopping and entering the rest mode, which can lock the car doors or prompt the driver to stop and rest immediately.

[0140] In some embodiments, the monitoring result information of emotions can be obtained, including emotional stability, road rage, tension or anxiety. According to the monitoring result information of emotions, one or more modules of the vehicle's audio control module, lighting control module, temperature control module, powertrain control module and auxiliary driving module are controlled to provide corresponding functional services.

[0141] For example, if the monitoring result is road rage, the intervention system integration module controls the voice system through the audio control module to issue a prompt "Keep calm, safety first" and play music; adjusts the color and brightness of the vehicle lights to create a relaxing environment; switches to a more comfortable driving mode to reduce the vehicle's power response and reduce the driver's driving pressure; controls the vehicle to automatically play soft and soothing music to help the driver relax, while providing voice comfort to encourage the driver to take deep breaths and think calmly.

[0142] If the monitoring result is tension, the intervention system integration module controls the voice system through the audio control module to issue a prompt "You are doing well, stay focused", play calm music and encouraging voice messages to help the driver relax; provide additional driving assistance, such as lane keeping assist and blind spot monitoring, to help the driver better cope with driving stress.

[0143] If the monitoring result is anxiety, the intervention system integration module controls the voice system through the audio control module to issue a prompt "Everything is under control, please stay calm", play relaxing music or guided meditation audio to help the driver reduce anxiety; provide deep breathing guidance or mindfulness exercises to help the driver relax; provide tips and guidance on driving safety, such as road information and navigation prompts, to increase the driver's confidence; adjust the temperature in the car to maintain a comfortable environment to help reduce anxiety.

[0144] In some embodiments, health status monitoring result information is obtained, and based on the health status monitoring result information, one or more of the vehicle's automatic braking control module, auxiliary driving module, vehicle alarm system, on-board communication module, positioning system, remote information processing system, and health detection module are controlled to provide corresponding functional services.

[0145] For example, if the monitoring result is myocardial infarction or epilepsy, the central control unit of the intervention system integration module controls the automatic braking control module to perform automatic braking and turns on the vehicle assisted driving mode to park the vehicle. The system automatically dials the preset emergency contact or service number, such as the emergency service center or family member. Activate the in-vehicle alarm system to send an emergency signal to the outside world to attract the attention of surrounding vehicles and pedestrians; turn on the GPS positioning function to provide real-time location to emergency services to ensure that rescuers can quickly find the location of the vehicle; collect and transmit the driver's health data (such as heart rate, blood pressure) to the emergency service center to help quickly assess the health status.

[0146] The embodiment of the present application actively intervenes in the driving process based on the monitoring result information. When the driver's digital twin is in a state that is not conducive to safe driving, the state is promptly fed back to the driver and the vehicle, and the vehicle is controlled to actively intervene in driving, thereby improving driving safety.

[0147] The following is a detailed description and explanation of the solution of the embodiment of the present invention in conjunction with a specific application example:

[0148] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the system framework of a driving intervention method based on digital twins provided in some embodiments of the present application. The system framework of the driving intervention method based on digital twins includes a physical layer, a twin layer and an application layer.

[0149] At the physical layer, the driver's biometric data is collected through multiple types of sensors; at the twin layer, the driver's twin is created through the collected biometric data and multiple algorithm models are used to monitor the various states of the driver's twin, and the monitoring results are fed back to the physical entity in real time; at the application layer, the feedback monitoring results are used to determine whether driving intervention is needed.

[0150] Specifically, a driver virtual model system based on digital twin technology can be developed and implemented. This digital twin can accurately simulate and reflect the driver's fatigue, emotions and health status. Through real-time data collection and analysis, the system can continuously monitor the driver's physiological and psychological state and feed this information back to the vehicle control system. The driver multi-state monitoring process can be seen in Figure 2The digital twin not only provides a comprehensive digital view of the driver's status, but also supports dynamic adjustment and precise driving intervention, thus significantly improving driving safety and driving experience.

[0151] Exemplarily, the driver's EEG signal, electromyographic signal, facial features and exhalation signal are first collected to construct the driver's digital twin. Next, different feature extraction methods are used to extract features of the driver's digital twin, obtain key features and perform feature data fusion. Finally, the algorithm model is used to analyze the fused data to achieve monitoring of multiple states of the driver's digital twin. The algorithm model can include three models, namely, a back propagation neural network model, a support vector machine model and a random forest model. The back propagation neural network is used to monitor the driver's fatigue level. The model is a three-classification model, and the output is three fatigue states of the driver (mild fatigue, moderate fatigue, and severe fatigue). The support vector machine model is used as a classification model for driver emotion monitoring. The particle swarm method is used to optimize the parameters of the model to achieve monitoring of four emotions (stable emotions, road rage, tension, and anxiety). The random forest model is used to monitor sudden illnesses (myocardial infarction, cerebral infarction) of drivers, and the particle swarm optimization algorithm is used to optimize its parameters to improve the classification accuracy of the model.

[0152] In the twin space, the driver's digital twin's driving ability is evaluated in real time based on the monitoring results of the driver's twin, and the evaluation results are fed back to the physical space in a timely manner. When the driver's twin is in a state that is not conducive to safe driving, the state is fed back to the driver and the vehicle in a timely manner, and the vehicle is controlled to actively intervene in driving to improve driving safety.

[0153] The above is an introduction to the embodiment of the driving intervention method based on digital twin provided in this application.

[0154] The implementation scheme of the driving intervention device based on digital twin provided by the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0155] For the driving intervention method based on digital twin provided in the above embodiment, the embodiment of the present application also provides a driving intervention device based on digital twin, which is used to implement the above method, such as Figure 5 As shown, Figure 5 This is a module schematic block diagram of a driving intervention device based on digital twins according to an embodiment of the present application. The driving intervention device based on digital twins 500 includes:

[0156] An acquisition module 501 is used to acquire a driver's biometric signal, where the biometric signal is used to characterize the driver's characteristics under different physiological and psychological states;

[0157] A generating module 502, for generating a digital twin of the driver based on the biometric signal;

[0158] A monitoring module 503 is used to monitor the digital twin through a pattern recognition model to obtain monitoring result information, wherein the pattern recognition model is used to extract key features of the biometric signal and monitor the fatigue level, mood and health status of the digital twin;

[0159] The active intervention module 504 is used to actively intervene in the driving process according to the monitoring result information.

[0160] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0161] like Figure 6 As shown, the embodiment of the present application also provides an electronic device, the electronic device 600 includes a memory 601, one or more processors 602 ( Figure 6 Only one is shown) and a computer program stored in the memory 601 and executable on the processor 602. Wherein: the memory 601 is used to store software programs and units, and the processor 602 executes various functional applications and data processing by running the software programs and units stored in the memory 601 to obtain the resources corresponding to the above-mentioned preset events. Optionally, the processor 602 implements the above-mentioned driving intervention method based on digital twins by running the above-mentioned computer program stored in the memory 601.

[0162] The memory 601, as a non-transitory computer-readable medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 601 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 601 may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor 602 via a network.

[0163] It can be understood that the contents of the above method embodiments are all applicable to the electronic device embodiment, the functions specifically implemented by the electronic device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0164] The embodiment of the present application also provides a vehicle, which includes the above-mentioned driving intervention device based on digital twin or the electric drive assembly of the above-mentioned electronic device. Specifically, the vehicle can be a private car, such as a sedan, SUV, MPV or pickup truck. The vehicle can also be an operating vehicle, such as a van, bus, small truck or large trailer. The vehicle can be a gasoline vehicle or a new energy vehicle. When the vehicle is a new energy vehicle, it can be a hybrid vehicle or a pure electric vehicle.

[0165] It can be understood that the contents of the above method embodiments are all applicable to the present vehicle embodiment, the functions specifically implemented by the present vehicle embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0166] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it can implement the steps of the above-mentioned driving intervention method based on digital twins.

[0167] It can be understood that the contents of the above method embodiments are all applicable to the present computer program product, the functions specifically implemented by the present computer program product embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0168] The driving intervention method, device, electronic device, vehicle and computer program product based on digital twin provided in the embodiment of the present application obtain the driver's biometric signal, wherein the biometric signal is used to characterize the characteristics of the driver under different physiological and psychological states; then, the driver's digital twin is generated based on the biometric signal; then, the digital twin is monitored by a pattern recognition model to obtain monitoring result information, wherein the pattern recognition model is used to extract the key features of the biometric signal and monitor the fatigue level, emotion and health status of the digital twin; finally, the driving process is actively intervened according to the monitoring result information. The embodiment of the present application generates the driver's digital twin through biometric signals. The digital twin connects the physical world and the information world, and provides a new idea for monitoring the driver's state. The monitoring of the fatigue level, emotion and health status of the digital twin through the pattern recognition model can improve the accuracy and real-time performance of driver monitoring. When the driver has dangerous behavior or health problems, measures can be taken in advance, such as actively intervening in the driving process to improve the driving safety of the vehicle. In addition, the digital twin method can also record the driver's behavior data, which can be used to analyze and improve their driving habits, improve driving safety, and realize intelligent services for vehicles.

[0169] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0170] Although specific embodiments are described herein, those of ordinary skill in the art will recognize that many other modifications or alternative embodiments are also within the scope of the present disclosure. For example, any of the functions and / or processing capabilities described in conjunction with a particular device or component may be performed by any other device or component. In addition, although various exemplary implementations and architectures have been described according to embodiments of the present disclosure, those of ordinary skill in the art will recognize that many other modifications to the exemplary implementations and architectures described herein are also within the scope of the present disclosure.

[0171] Some aspects of the present disclosure are described above with reference to the block diagram and flow chart of the system, method, system and / or computer program product according to the exemplary embodiments.It should be understood that the combination of one or more blocks in the block diagram and the flow chart and the blocks in the block diagram and the flow chart can be realized by executing computer executable program instructions respectively.Equally, according to some embodiments, some blocks in the block diagram and the flow chart may not need to be executed in the order shown, or may not need to be executed in full.In addition, additional components and / or operations beyond those components and / or operations shown in the blocks in the block diagram and the flow chart may be present in certain embodiments.

[0172] Therefore, the blocks in the block diagrams and flow charts support the combination of devices for performing the specified functions, the combination of elements or steps for performing the specified functions, and the program instruction devices for performing the specified functions. It should also be understood that each block in the block diagrams and flow charts and the combination of blocks in the block diagrams and flow charts can be implemented by a special-purpose hardware computer system or a combination of special-purpose hardware and computer instructions that performs a specific function, element or step.

[0173] The program modules, applications, etc. described herein may include one or more software components, including, for example, software objects, methods, data structures, etc. Each such software component may include computer-executable instructions that, in response to execution, cause at least a portion of the functionality described herein (e.g., one or more operations of the illustrative methods described herein) to be performed.

[0174] Software components can be encoded with any of various programming languages. An exemplary programming language can be a low-level programming language, such as an assembly language associated with a specific hardware architecture and / or an operating system platform. Software components comprising assembly language instructions may need to be converted to executable machine code by an assembler before being executed by hardware architecture and / or platform. Another exemplary programming language can be a higher-level programming language, which can be transplanted across multiple architectures. Software components comprising a higher-level programming language may need to be converted to an intermediate representation by an interpreter or a compiler before execution. Other examples of programming languages ​​include but are not limited to macro languages, shells or command languages, job control languages, scripting languages, database queries or search languages ​​or report writing languages. In one or more exemplary embodiments, a software component comprising the instructions of one of the above-mentioned programming language examples can be directly executed by an operating system or other software components, without first being converted into another form.

[0175] Software components may be stored as files or other data storage structures. Software components of similar type or related functions may be stored together, such as in a specific directory, folder, or library. Software components may be static (e.g., preset or fixed) or dynamic (e.g., created or modified at execution time).

[0176] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A driving intervention method based on digital twin, characterized in that: The following steps are involved: Acquire a biometric signal of the driver, where the biometric signal is used to characterize the characteristics of the driver in different physiological and psychological states; generating a digital twin of the driver based on the biometric signal; Monitoring the digital twin through a pattern recognition model to obtain monitoring result information, wherein the pattern recognition model is used to extract key features of the biometric signal and monitor the fatigue level, emotion and health status of the digital twin; Actively intervene in the driving process based on the monitoring result information.

2. The driving intervention method based on digital twin according to claim 1, characterized in that: The biometric signal includes an electroencephalogram signal, an electromyographic signal, a facial feature, and an exhalation signal of the driver, and generating a digital twin of the driver based on the biometric signal includes: A digital twin of the driver is generated based on the driver's electroencephalogram signal, the electromyography signal, the facial features, and the exhalation signal.

3. The driving intervention method based on digital twin according to claim 1, characterized in that: The digital twin is monitored by the pattern recognition model to obtain monitoring result information, including: Extracting key features of the biometric signal respectively by using a plurality of feature extraction methods; Performing feature fusion on the key features to obtain feature fusion results; The feature fusion results are analyzed by using a plurality of monitoring algorithms to obtain monitoring result information of the fatigue level, monitoring result information of the emotion and monitoring result information of the health status.

4. The driving intervention method based on digital twin according to claim 3, characterized in that: The biometric signal includes the driver's electroencephalogram signal, electromyography signal, facial features and exhalation signal. The key features of the biometric signal are extracted by multiple feature extraction methods, including: Extract key features of the EEG signal and the EMG signal using a wavelet transform method; Extracting key features of the facial features using a histogram of oriented gradients method; The key features of the exhalation signal are extracted using the average method.

5. The driving intervention method based on digital twin according to claim 3, characterized in that: The analyzing the feature fusion results by using a plurality of monitoring algorithms includes: Monitoring the fatigue level of the digital twin using a back propagation neural network model; Monitoring the emotions of the digital twin using a support vector machine model combined with a particle swarm optimization algorithm; The health status of the digital twin is monitored using a random forest model combined with a particle swarm optimization algorithm.

6. The driving intervention method based on digital twin according to claim 1, characterized in that: The monitoring result information includes monitoring result information of fatigue level, and the actively intervening in the driving process according to the monitoring result information includes: Obtaining monitoring result information of the fatigue level; According to the monitoring result information of the fatigue level, one or more modules among the seat adjustment control module, the audio control module and the display module of the vehicle are controlled to provide corresponding functional services.

7. The driving intervention method based on digital twin according to claim 1, characterized in that: The monitoring result information includes the monitoring result information of emotions, and the actively intervening in the driving process according to the monitoring result information includes: Obtaining monitoring result information of the emotion; According to the emotion monitoring result information, one or more modules among the vehicle's audio control module, lighting control module, temperature control module, powertrain control module or auxiliary driving module are controlled to provide corresponding functional services.

8. The driving intervention method based on digital twin according to claim 1, characterized in that: The monitoring result information includes monitoring result information of the health status, and the actively intervening in the driving process according to the monitoring result information includes: Obtaining monitoring result information of the health status; According to the monitoring result information of the health status, one or more of the vehicle's automatic braking control module, auxiliary driving module, vehicle alarm system, on-board communication module, positioning system, telematics system or health detection module are controlled to provide corresponding functional services.

9. A driving intervention device based on digital twin, characterized in that: The device comprises: An acquisition module, used to acquire a biometric signal of the driver, wherein the biometric signal is used to characterize the characteristics of the driver in different physiological and psychological states; A generation module, configured to generate a digital twin of the driver based on the biometric signal; A monitoring module, used to monitor the digital twin through a pattern recognition model to obtain monitoring result information, wherein the pattern recognition model is used to extract key features of the biometric signal and monitor the fatigue level, mood and health status of the digital twin; The active intervention module is used to actively intervene in the driving process according to the monitoring result information.

10. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the digital twin-based driving intervention method as described in any one of claims 1 to 8 when executing the computer program.

11. A vehicle, characterized in that: The vehicle includes the digital twin-based driving intervention device as claimed in claim 9 or the electronic device as claimed in claim 10.

Images