Driving state monitoring method and device, vehicle, electronic equipment and storage medium

Abstract

The invention provides a driving state monitoring method and device, a vehicle, electronic equipment and a storage medium. The driving state monitoring method comprises the steps of obtaining driving state data of a driver and vehicle driving state data; performing fusion processing on the driving state data and the vehicle driving state data according to the time sequence to obtain a target fusion feature; analyzing the target fusion feature based on a preset safe driving model algorithm obtained from the cloud server, and determining whether the driving state of the driver and the driving state of the vehicle meet respective corresponding safe driving conditions or not according to an analysis result, and when the driving state of the driver and / or the driving state of the vehicle do not meet the corresponding safe driving conditions, determining that the vehicle is in non-safe driving. The driving state data and the vehicle driving state data are analyzed based on the preset safe driving model algorithm acquired from the cloud server, so that the transmission time delay of uploading the driving state data and the vehicle driving state data to the cloud server is avoided, and the vehicle driving state is monitored in time.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of vehicles, and particularly relates to a driving state monitoring method and device, a vehicle, an electronic device and a storage medium. BACKGROUND

[0002] In a vehicle monitoring system for monitoring a driving state of a vehicle, large-scale artificial intelligence algorithm models and centralized analysis of large-scale data are performed on a cloud server. Because the cloud server has a large amount of computing resources, the cloud server can complete data processing in a very short time. However, relying only on the computing power of the cloud server to provide services for the vehicle monitoring system has some disadvantages, for example, the operation of the artificial intelligence algorithm model relies on a large amount of real-time data generated by the vehicle during driving. If the real-time data is transmitted to the artificial intelligence algorithm model of the cloud server through the core network for remote processing, the transmission delay of the real-time data is large. SUMMARY

[0003] The present disclosure provides a driving state monitoring method, device, vehicle, electronic device and storage medium. The main purpose is to solve the problem of long delay when the artificial intelligence algorithm model running on the cloud server remotely acquires real-time data generated by the vehicle during driving.

[0004] According to a first aspect of the present disclosure, a driving state monitoring method is provided, which comprises:

[0005] obtaining driving state data of a driver and vehicle driving state data;

[0006] performing fusion processing on the driving state data and the vehicle driving state data in time sequence to obtain target fusion features;

[0007] analyzing the target fusion features based on a preset safe driving model algorithm, and determining whether the driving state of the driver and the driving state of the vehicle both meet the respective corresponding safe driving conditions according to the analysis result, the preset safe driving model algorithm being a safe driving model algorithm obtained from a cloud server;

[0008] in the case that the driving state of the driver and / or the driving state of the vehicle does not meet the respective corresponding safe driving conditions, determining that the vehicle is in unsafe driving.

[0009] Optionally, the analyzing the target fusion features based on the preset safe driving model algorithm and determining whether the driving state of the driver and the driving state of the vehicle both meet the respective corresponding safe driving conditions according to the analysis result comprises:

[0010] analyze the target fusion feature based on the preset safe driving model algorithm, wherein the preset safe driving model algorithm is a safe driving model algorithm obtained from the cloud server according to a preset update period;

[0011] determine whether the driving state of the driver and the driving state of the vehicle conform to the respective corresponding safe driving conditions according to the analysis result.

[0012] Optionally, the fusion processing of the driving state data and the vehicle driving state data according to the time sequence to obtain the target fusion feature comprises:

[0013] respectively performing data preprocessing on the driving state data and the vehicle driving state data to obtain processed driving state data and processed vehicle driving state data;

[0014] respectively performing feature extraction on the processed driving state data and the processed vehicle driving state data to obtain driving state features and vehicle driving state features;

[0015] performing feature fusion on the driving state features and the vehicle driving state features with the same timestamp based on a preset fusion algorithm to obtain the target fusion feature.

[0016] Optionally, after determining that the vehicle is in unsafe driving, the method comprises:

[0017] rejudging whether the driving state of the driver and the driving state of the vehicle conform to the respective corresponding safe driving conditions after a preset time period;

[0018] if the driving state of the driver and / or the driving state of the vehicle does not conform to the respective corresponding safe driving conditions, controlling the vehicle according to a preset braking strategy.

[0019] Optionally, after analyzing the target fusion feature based on the preset safe driving model algorithm and determining whether the driving state of the driver and the driving state of the vehicle conform to the respective corresponding safe driving conditions according to the analysis result, the method further comprises:

[0020] if the driving state of the driver and the driving state of the vehicle conform to the respective corresponding safe driving conditions, determining that the vehicle is in safe driving, and continuing to obtain the driving state data of the driver and the driving state data of the vehicle.

[0021] According to a second aspect of the present disclosure, a device for monitoring driving state is provided, comprising:

[0022] an obtaining unit configured to obtain driving state data of a driver and vehicle driving state data.

[0023] a processing unit, configured to perform fusion processing on the driving state data and the vehicle driving state data in a time sequence to obtain target fusion features;

[0024] an analysis unit, configured to analyze the target fusion features based on a preset safe driving model algorithm, and determine whether the driving state of the driver and the driving state of the vehicle both meet respective corresponding safe driving conditions according to an analysis result, the preset safe driving model algorithm being a safe driving model algorithm obtained from a cloud server;

[0025] a determination unit, configured to determine that the vehicle is in unsafe driving in a case where the driving state of the driver and / or the driving state of the vehicle does not meet the respective corresponding safe driving conditions.

[0026] Optionally, the analysis unit comprises:

[0027] an analysis module, configured to analyze the target fusion features based on the preset safe driving model algorithm to obtain the analysis result, the preset safe driving model algorithm being a safe driving model algorithm obtained from the cloud server according to a preset update period;

[0028] a determination module, configured to determine whether the driving state of the driver and the driving state of the vehicle both meet the respective corresponding safe driving conditions according to the analysis result.

[0029] Optionally, the processing unit comprises:

[0030] a processing module, configured to perform data preprocessing on the driving state data and the vehicle driving state data respectively to obtain processed driving state data and processed vehicle driving state data;

[0031] an extraction module, configured to perform feature extraction on the processed driving state data and the processed vehicle driving state data respectively to obtain driving state features and vehicle driving state features;

[0032] a fusion module, configured to perform feature fusion on the driving state features and the vehicle driving state features with the same time stamp based on a preset fusion algorithm to obtain the target fusion features.

[0033] Optionally, the device comprises:

[0034] a judgment unit, configured to re-determine whether the driving state of the driver and the driving state of the vehicle both meet the respective corresponding safe driving conditions after a preset time period;

[0035] The control unit is configured to control the vehicle according to a preset braking strategy when the driving state of the driver and / or the driving state of the vehicle does not meet the respective corresponding safe driving condition.

[0036] Optionally, the determination unit is further configured to, after analyzing the target fusion feature based on the preset safe driving model algorithm and determining whether the driving state of the driver and the driving state of the vehicle meet the respective corresponding safe driving condition according to the analysis result, determine that the vehicle is in safe driving and continue to acquire the driving state data of the driver and the driving state data of the vehicle when the driving state of the driver and the driving state of the vehicle meet the respective corresponding safe driving condition.

[0037] According to a third aspect of the present disclosure, a vehicle is provided, comprising the driving state monitoring apparatus of the foregoing second aspect.

[0038] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising:

[0039] at least one processor; and

[0040] a memory connected with the at least one processor in communication; wherein,

[0041] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the foregoing first aspect.

[0042] According to a fifth aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the method of the foregoing first aspect.

[0043] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method of the foregoing first aspect.

[0044] The method, device, vehicle, electronic device and storage medium provided by the present disclosure obtain driving state data and vehicle driving state data of a driver; perform fusion processing on the driving state data and the vehicle driving state data in a time sequence to obtain target fusion features; analyze the target fusion features based on a preset safe driving model algorithm, determine whether the driving state of the driver and the driving state of the vehicle meet respective corresponding safe driving conditions according to an analysis result, and the preset safe driving model algorithm is a safe driving model algorithm obtained from a cloud server; in the case that the driving state of the driver and / or the driving state of the vehicle does not meet the respective corresponding safe driving conditions, determine that the vehicle is in unsafe driving. Compared with the related art, after the vehicle obtains the driving state data and the vehicle driving state data, and analyzes the driving state data and the vehicle driving state data based on the preset safe driving model algorithm obtained from the cloud server to determine whether the vehicle is in safe driving, the driving state data and the vehicle driving state data are directly obtained from the vehicle end, thereby avoiding the transmission delay of uploading the driving state data and the vehicle driving state data to the cloud server, and timely monitoring of the driving state of the vehicle is realized.

[0045] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0047] Figure 1 A flowchart of a driving state monitoring method provided by an embodiment of the present disclosure;

[0048] Figure 2 A flowchart of another driving state monitoring method provided by an embodiment of the present disclosure;

[0049] Figure 3 A structural diagram of a driving state monitoring device provided by an embodiment of the present disclosure;

[0050] Figure 4 A structural diagram of another driving state monitoring device provided by an embodiment of the present disclosure

[0051] Figure 5 A schematic block diagram of an example electronic device 300 provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are cited as illustrative examples. Various details of the embodiments of the present disclosure are described herein in order to provide a thorough understanding of the embodiments. It will be understood by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted herein.

[0053] Noun explanation: edge computing refers to a computing model that performs computing at the edge of the network. Its operation object comes from the downlink data of cloud service and the uplink data of Internet of Things service, and the "edge" in edge computing refers to any computing and network resources between the data source and the cloud computing center. In short, edge computing deploys servers to edge nodes near users, provides services to users at the edge of the network (such as wireless access points), avoids long-distance data transmission, and provides users with faster response.

[0054] The method, device, vehicle, electronic device and storage medium for monitoring driving state of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0055] Figure 1 A flowchart of a method for monitoring driving state provided by the embodiments of the present disclosure is shown.

[0056] As shown in Figure 1 , the method comprises the following steps:

[0057] Step 101: obtaining driving state data of a driver and vehicle driving state data;

[0058] As a refinement of step 101, the driving state data of the driver and the vehicle driving state data during the driving of the vehicle are obtained, the driving state data includes but is not limited to: facial expression, eye movement, head movement, eye closure frequency, etc. of the driver, and the vehicle driving state data includes but is not limited to: performance parameters and conditions of the vehicle, such as speed, fuel consumption, engine state, tire pressure, etc. The driving state data and the vehicle driving state data obtained are used to determine whether the vehicle is driving safely.

[0059] In some embodiments, the vehicle edge computing node collects and acquires the driving state data and the vehicle driving state data based on cameras or various preset sensors on the vehicle, wherein the vehicle edge computing node includes but is not limited to a processor device within the vehicle, and the various preset sensors include, for example, an acceleration sensor, a gyroscope, a pressure sensor, a camera, a GPS module, etc., which collect and acquire various driving state data and vehicle driving state data in real time during the operation of the vehicle. The vehicle driving state data includes vehicle speed, acceleration, steering angle, road conditions, traffic signals, pedestrian behavior, etc., as well as position information and trajectory data of the vehicle. After the raw data is collected by the sensor data, it is transmitted to the vehicle edge computing node for processing. Based on the received various driving state data and vehicle driving state data, the vehicle edge computing node performs real-time analysis and processing. Using advanced algorithms and models, the computing node can quickly identify the surrounding environment, assess the driving state, and predict possible driving conditions.

[0060] Step 102, fusion processing of the driving state data and the vehicle driving state data according to time sequence to obtain target fusion features;

[0061] As a refinement of the above step 102, the driving state data and the vehicle driving state data are fusion processed according to time sequence to obtain the target fusion features, that is, the information of multiple data sources is integrated according to time sequence to eliminate noise and error data, thereby improving the accuracy and quality of the data. And fusion of data from different data sources can obtain more rich information, making the data more valuable.

[0062] Step 103, analysis of the target fusion features based on a preset safe driving model algorithm, and determination of whether the driving state of the driver and the driving state of the vehicle meet the respective corresponding safe driving conditions according to the analysis result; the preset safe driving model algorithm is a safe driving model algorithm obtained from a cloud server.

[0063] As a refinement of the above step 103, the target fusion features are analyzed directly on the vehicle edge computing node, and it is determined whether the driving state of the driver and the driving state of the vehicle meet the respective corresponding safe driving conditions according to the analysis result, that is, the state analysis of whether the vehicle is safe to drive is performed on the edge device of the data source, thereby ensuring the transmission real-time of each data generated during the driving of the vehicle, and improving the output rate of the analysis result, and further realizing more timely monitoring of the driving state of the vehicle.

[0064] In some embodiments, the preset safe driving model algorithm obtained from the cloud server can be called during the analysis, and the target fusion features are analyzed based on the preset safe driving model algorithm.

[0065] Step 104, in the case that the driving state of the driver and / or the running state of the vehicle does not meet the respective corresponding safe running condition, determining that the vehicle is in unsafe running.

[0066] As a refinement of the above step 104, in the case that the driving state of the driver and / or the running state of the vehicle does not meet the respective corresponding safe running condition, determining that the vehicle is in unsafe running, that is, either the driving state of the driver and the running state of the vehicle does not meet the respective corresponding safe running condition, or both the driving state of the driver and the running state of the vehicle does not meet the respective corresponding safe running condition. By combining the driving state of the driver and the running state of the vehicle to determine the running state of the vehicle, the accuracy of the determination result can be ensured.

[0067] The method for monitoring the driving state provided by the present disclosure comprises the following steps: acquiring the driving state data of the driver and the running state data of the vehicle; fusing the driving state data and the running state data of the vehicle according to the time sequence to obtain target fusion features; analyzing the target fusion features based on a preset safe driving model algorithm, and determining whether the driving state of the driver and the running state of the vehicle meet the respective corresponding safe running condition according to the analysis result, wherein the preset safe driving model algorithm is a safe driving model algorithm obtained from a cloud server; in the case that the driving state of the driver and / or the running state of the vehicle does not meet the respective corresponding safe running condition, determining that the vehicle is in unsafe running. Compared with the related art, after the vehicle obtains the driving state data and the running state data of the vehicle, the driving state data and the running state data of the vehicle are analyzed based on the preset safe driving model algorithm obtained from the cloud server to determine whether the vehicle is in safe running. The driving state data and the running state data of the vehicle are directly obtained from the vehicle end, thereby avoiding the transmission time delay of uploading the driving state data and the running state data of the vehicle to the cloud server, and further realizing the timely monitoring of the running state of the vehicle.

[0068] As a refinement of the embodiments of the present disclosure, when performing step 103 of analyzing the target fusion feature based on the preset safe driving model algorithm, and determining whether the driving state of the driver and the driving state of the vehicle both meet the respective corresponding safe driving conditions according to the analysis result, the following implementation manners can also be used, but are not limited to, for example: analyzing the target fusion feature based on the preset safe driving model algorithm to obtain the analysis result, wherein the preset safe driving model algorithm is a safe driving model algorithm obtained from a cloud server according to a preset update period; and determining whether the driving state of the driver and the driving state of the vehicle both meet the respective corresponding safe driving conditions according to the analysis result. The preset update period includes but is not limited to a period of days, a period of weeks, and the specific period is not limited.

[0069] In some embodiments, the preset safe driving model algorithm includes but is not limited to a fatigue detection algorithm. The fatigue detection algorithm can use computer vision methods such as machine learning to analyze facial features, eye closure frequency, etc. of the driver, for example, to classify the driving state of the driver based on the target fusion feature using a support vector machine. The basic formula of the support vector machine is shown in formula (1):

[0070] (y = \textbf{w}^\top\phi(\textbf{x}) + b) Formula (1)

[0071] wherein (y) is an output label, (\textbf{w}) is a weight vector, (\textbf{x}) is an input feature vector, (\phi) is a function mapped to a high-dimensional space, and (b) is a bias term.

[0072] In some embodiments, the preset safe driving model algorithm includes but is not limited to a vehicle diagnosis algorithm. The vehicle diagnosis algorithm is used to analyze vehicle driving state data obtained from an on-board diagnostic system (OBD) in the target fusion feature to classify the driving state of the vehicle. The form of the vehicle driving state data includes but is not limited to engine fault codes. The vehicle diagnosis algorithm can be implemented based on a decision tree or a neural network. The decision tree implements the vehicle diagnosis algorithm as shown in formula (2):

[0073] (R_m:\text{if}\S_m(\textbf{x})\\text{then}\y=k(m)) Formula (2)

[0074] wherein (S_m(\textbf{x})) is a rule, and (k(m)) is a classification result. The decision tree is a collection of a series of rules used when making a classification decision.

[0075] In combination with the above embodiments, the embodiments provide exemplary descriptions of the above solutions. The target fusion feature is analyzed based on the fatigue detection algorithm to obtain a first analysis result corresponding to the driving state data. The first analysis result shows whether the driving state of the driver meets a first safe driving condition. The target fusion feature is analyzed based on the vehicle diagnosis algorithm to obtain a second analysis result corresponding to the vehicle driving state data. The second analysis result shows whether the driving state of the vehicle meets a second safe driving condition. The safe driving conditions include the first safe driving condition and the second safe driving condition. The analysis results include the first analysis result and the second analysis result. Whether the driving state of the driver and the driving state of the vehicle both meet the respective safe driving conditions is determined based on the first analysis result and the second analysis result. It should be understood that the fatigue detection algorithm and the vehicle diagnosis algorithm used in the foregoing are only exemplary. The embodiments do not limit the detection algorithm used to be the two algorithms described above. For example, the fatigue detection algorithm can be replaced by an attention monitoring algorithm.

[0076] In some embodiments, the preset safe driving model algorithm includes but is not limited to a probability prediction algorithm. The probability prediction algorithm estimates whether the vehicle is safely driven using Bayes' theorem, which is shown in the following formula (3):

[0077] (P(A|B) = \frac{P(B|A)P(A)}{P(B)}) Formula (3)

[0078] Formula (3) is used to update the safe driving state of the vehicle. The first analysis result and the second analysis result are used as inputs of the probability prediction algorithm, which can achieve estimation of whether the vehicle is safely driven.

[0079] The preset safe driving model algorithm is a safe driving model algorithm obtained from a cloud server according to a preset update period. That is, the preset safe driving model algorithm is a safe driving model algorithm obtained by the vehicle edge computing node from the cloud server. The latest preset safe driving model algorithm obtained will replace the originally cached preset safe driving model algorithm in the vehicle edge computing node. The preset safe driving model algorithm obtained is used to analyze the target fusion feature within the preset update period.

[0080] As a refinement of the above-mentioned embodiments, in the step of performing fusion processing on the driving state data and the vehicle running state data in time sequence to obtain the target fusion feature, the following implementation modes can be adopted, but are not limited to, for example: performing data preprocessing on the driving state data and the vehicle running state data respectively to obtain processed driving state data and processed vehicle running state data; performing feature extraction on the processed driving state data and the processed vehicle running state data respectively to obtain driving state features and vehicle running state features; and performing feature fusion on the driving state features and the vehicle running state features with the same timestamp based on a preset fusion algorithm to obtain the target fusion feature.

[0081] As a refinement of the above-mentioned embodiments, the driving state data and the vehicle running state data obtained by the vehicle edge computing node are fused based on a preset fusion algorithm. The purpose of feature fusion is to integrate data from multiple sensors in a unified framework to obtain more accurate and complete information. The vehicle edge computing node adopts the following basic steps for data feature fusion:

[0082] I. Preprocessing: The driving state data and the vehicle running state data obtained from various sensors (such as cameras, OBD) are cleaned, standardized, and synchronized.

[0083] II. Feature extraction: Perform feature extraction operations on sensor data, such as extracting facial features from driving state data collected by cameras using image processing algorithms, or extracting vehicle running features from vehicle running state data obtained from OBD.

[0084] III. Feature fusion: Apply a preset fusion algorithm to combine feature data from different data sources to create a comprehensive data view. This involves sensor data alignment and timestamp synchronization, i.e., performing feature fusion on the driving state features and the vehicle running state features with the same timestamp based on a preset fusion algorithm to obtain the target fusion feature.

[0085] The preset fusion algorithm includes but is not limited to using weighted calculation, which is shown in the following formula (4):

[0086] [x_{\text{fused}}=\sum_{i=1}^{n}w_ix_i] Formula (4)

[0087] Where (x_{\text{fused}}) is the fused estimate, (x_i) is the observation value of the (i)th sensor, and (w_i) is the corresponding weight, usually based on the signal-to-noise ratio index.

[0088] As a refinement of the above-mentioned embodiments, after determining that the vehicle is in unsafe driving, the method can further employ, but is not limited to, the following implementation manners, for example: determining whether the driving state of the driver and the driving state of the vehicle meet the respective corresponding safe driving conditions after a preset time period; if the driving state of the driver and / or the driving state of the vehicle does not meet the respective corresponding safe driving conditions, controlling the vehicle according to a preset braking strategy.

[0089] For the convenience of understanding the above-mentioned embodiments, the present embodiment provides an exemplary illustration that, after determining that the vehicle is in unsafe driving, the vehicle outputs prompt information to remind the driver to adjust the driving state or adjust the driving state of the vehicle within the preset time period, and then determines whether the driving state of the driver and the driving state of the vehicle meet the respective corresponding safe driving conditions after the preset time period is met. If the driving state of the driver and / or the driving state of the vehicle does not meet the respective corresponding safe driving conditions, the vehicle is controlled according to a preset braking strategy. The braking strategy includes, but is not limited to, controlling the vehicle to stop on the side of the road or to slow down.

[0090] As a refinement of the above-mentioned embodiments, when performing the determination of whether the driving state of the driver and the driving state of the vehicle meet the respective corresponding safe driving conditions according to the analysis result, the following implementation manners can be employed, but are not limited to, for example: determining whether the driving state of the driver meets a first preset safe driving condition according to the analysis result; and determining whether the driving state of the vehicle meets a second preset safe driving condition according to the analysis result, wherein the safe driving condition includes the first preset safe driving condition and the second preset safe driving condition. The processes involved in the foregoing embodiments have been exemplarily described in the foregoing, and will not be described here again for the sake of clarity and brevity.

[0091] In some embodiments, after analyzing the target fusion feature based on the preset safe driving model algorithm and determining whether the driving state of the driver and the driving state of the vehicle meet the respective corresponding safe driving conditions according to the analysis result, the method further includes: in the case where the driving state of the driver and the driving state of the vehicle meet the respective corresponding safe driving conditions, determining that the vehicle is in safe driving, and continuing to acquire the driving state data of the driver and the driving state data of the vehicle. That is, only in the scenario where the driving state of the driver and the driving state of the vehicle meet the corresponding safe driving conditions at the same time, it is determined that the vehicle is in safe driving.

[0092] For the convenience of understanding the above-mentioned embodiments, Figure 2 Another flowchart of a method for monitoring a driving state provided by the embodiments of the present disclosure is shown in FIG. 6. Figure 2The method comprises the following steps: a driver enters a vehicle interior, a sensor acquires a driving state of the driver and a vehicle driving state data; a vehicle sensor reports the acquired driver and vehicle data to a vehicle monitoring system of a vehicle edge computing node; the vehicle edge computing node regularly pulls a latest preset safe driving model from a cloud to cache the vehicle edge computing node. The vehicle monitoring system arranged at the vehicle edge computing node and comprising the preset safe driving model algorithm is used for quickly identifying the driving state of the driver and the vehicle driving state, and analyzing the real-time reported data based on the safe driving model to output a result of whether the current vehicle is safely driven; if the current state of the driver and the vehicle is safe driving, the monitoring is continued; if the current state of the driver or the vehicle is unsafe driving, the driver is prompted by voice to adjust the current state and perform safe operation; after the voice prompt, if the driver adjusts the state and performs the safe operation, the vehicle returns to the safe driving state, and the monitoring is continued; if the driver does not perform any operation and the vehicle is still in the unsafe driving state, the vehicle automatically makes a decision and performs safe operation, that is, the vehicle is slowed down or parked on the side (automatic driving function). It should be understood that the prompting mode is not limited to voice prompt, but also can be vibration seat prompt.

[0093] In some embodiments, when the vehicle has a serious fault, the vehicle monitoring system can automatically perform fault limitation, such as limiting the vehicle speed, or guiding the vehicle to a parking state in a safe manner, and notifying a background service center or an emergency contact service.

[0094] In order to make the above-mentioned embodiment more detailed, the embodiment illustrates the above-mentioned embodiment, for example: a driver drives at night, has continuously driven for 4 hours, the current speed is 90 km / h, and the road condition is relatively empty. In this scenario, the driver may face a safety risk due to fatigue. The following is a solution implemented according to the above-mentioned embodiment:

[0095] (I) Data acquisition:

[0096] Driver monitoring: a camera (30 fps, 1920x1080 resolution) captures the facial expression and eye movement of the driver. A face tracking algorithm is set up to detect the blinking frequency and the number of yawns in real time.

[0097] Vehicle state monitoring: through an OBD-II system, data such as vehicle speed (90 km / h), fuel consumption (8.5 L / 100 km), and tire pressure (front wheel 2.3 bar and rear wheel 2.1 bar) are captured. The sensor reports the captured data to the vehicle-mounted edge computing node once every 1 second.

[0098] (II) Local data processing:

[0099] The vehicle edge computing node (quad-core processor, 2.5 GHz per core, 8 GB RAM) analyzes the incoming data in real time.

[0100] The driver's state is analyzed by the algorithm to determine the blink rate (18 times per minute) and the number of yawns (2 times per 5 minutes), and the system defines this frequency as indicating driver fatigue.

[0101] (III) Edge computing data analysis:

[0102] In the preset safe driving model stored in the vehicle edge computing node, the criteria for determining fatigue driving is that the blink rate exceeds 15 times per minute and the yawn rate exceeds 1 time per 20 minutes.

[0103] The preset safe driving model quickly identifies the driver's fatigue state, and the vehicle still maintains a safe operating state.

[0104] (IV) Decision and response:

[0105] According to the analysis results, the system starts the seat vibration and issues a voice prompt through the vehicle audio (75 decibel volume): "Fatigue driving detected, please stop and rest."

[0106] If the driver does not respond to the prompt, the system will continue to repeat the audio prompt every 10 seconds, while the interface displays navigation to the nearest rest area.

[0107] If the vehicle maintains the current state for more than 5 minutes, the system will start the auxiliary driving function (such as automatically reducing the vehicle speed to 60 km / h) to ensure driving safety, and try to contact the emergency contact person through the vehicle communication system.

[0108] (V) Additional feedback and record:

[0109] After the driver adjusts the state, the system re-evaluates and continues to monitor.

[0110] All captured data and system responses will be recorded in the vehicle's log system for future analysis and potential legal matters. Through the above implementation, the edge computing-based vehicle monitoring system can timely alert the driver and prevent fatigue driving accidents, ensuring driving safety.

[0111] In summary, the embodiment can achieve the following effects:

[0112] 1. Running a preset safety driving model algorithm on a vehicle edge computing node to determine whether the vehicle is driving safely, omitting the process of uploading driving state data and vehicle driving state data to a cloud server for processing, thereby avoiding the transmission delay of uploading driving state data and vehicle driving state data to a cloud server, and achieving more timely monitoring of vehicle driving state.

[0113] 2. Edge computing can distribute data processing tasks to the edge of the network, i.e. the closest location to the data source, which helps to reduce the time required for data transmission to the central server, thereby achieving real-time monitoring of driver behavior and vehicle state.

[0114] 3. Edge computing allows pre-processing near the data source, sending only the required data or processing results to the cloud or central server, which can significantly reduce the use of communication bandwidth.

[0115] 4. Edge computing can reduce communication delay and quickly respond to various vehicle operating states and driver behavior monitoring, especially for autonomous or highly automated vehicles, where low latency is critical.

[0116] 5. By processing data through vehicle edge computing nodes, the monitoring system can continue to operate even in poor network connection or lost connection, ensuring continuous monitoring.

[0117] 6. Edge computing reduces the risk of data interception during transmission by processing sensitive data locally, helping to protect the privacy information of drivers and vehicles.

[0118] 7. Since data is processed locally, it is easier to manage data security and reduce security threats encountered by data in the cloud server.

[0119] 8. Edge computing can reduce unnecessary data transmission, thereby reducing energy consumption to some extent.

[0120] 9. By integrating biometric sensing, facial recognition and other technologies, the system can assess the driver's fatigue level, attention dispersion and other states.

[0121] 10. Based on local data processing of vehicles, faster response time than traditional monitoring systems can be achieved, thereby improving the ability to handle emergency situations.

[0122] 11. Because data processing does not rely on cloud services, data transmission can be reduced, saving bandwidth and reducing the risk of relying on remote servers.

[0123] 12. Reducing the possibility of problems with remote servers (such as network interruptions) improves the reliability and stability of the entire monitoring system.

[0124] 13. Edge computing enables personal data to be processed locally, reducing the risk of privacy leakage due to remote transmission.

[0125] 14. The system can monitor the driver's behavior and physiological state (such as pupil change, head posture, expression, heart rate, etc.) in almost real time, quickly identify and warn behaviors that may cause accident risks. Real-time monitoring of vehicle status, such as engine performance, tire condition, braking system, etc., can also timely discover problems and take preventive measures.

[0126] 15. Since data processing is done locally, the system eliminates the process of data transmission over the network, improving processing speed, which is particularly useful for scenarios that require fast decision-making and response. The optimization algorithm integrated in the monitoring system can analyze and process data faster, ensuring timely feedback.

[0127] 16. Reduces dependence on long-distance network connections, optimizing the system's adaptability to unstable network environments, especially in remote areas or environments with poor network coverage. Since edge computing can reduce dependence on cloud services, it can also reduce related data transmission costs and cloud service costs.

[0128] 17. Local processing of data reduces the risk of sensitive information being intercepted during transmission. More stringent security measures can be deployed to protect data and prevent unauthorized access.

[0129] 18. The vehicle monitoring system is less dependent on the stability of remote servers, so it can still operate independently when the server fails. The redundant design of edge computing nodes can further ensure that the system can maintain critical monitoring functions even in the event of partial hardware failure.

[0130] The above technical effects collectively improve the overall performance of the vehicle monitoring system, enhancing the safety of the vehicle and providing a higher level of protection and security for the driver and passengers.

[0131] Corresponding to the above-mentioned method for monitoring the driving state, the present application also provides a device for monitoring the driving state. Since the device embodiments of the present application correspond to the above-mentioned method embodiments, for details not disclosed in the device embodiments, please refer to the above-mentioned method embodiments, which will not be described in detail in the present application.

[0132] Figure 3 A structural diagram of a device for monitoring the driving state provided by the embodiments of the present disclosure is shown in Figure 3 , which includes:

[0133] The acquisition unit 21 is configured to acquire the driving state data of the driver and the vehicle driving state data.

[0134] The processing unit 22 is configured to perform fusion processing on the driving state data and the vehicle driving state data in a time sequence to obtain target fusion features;

[0135] The analysis unit 23 is configured to analyze the target fusion features based on a preset safe driving model algorithm, and determine whether the driving state of the driver and the driving state of the vehicle both meet the respective corresponding safe driving conditions according to an analysis result, the preset safe driving model algorithm being a safe driving model algorithm obtained from a cloud server;

[0136] The determination unit 24 is configured to determine that the vehicle is in unsafe driving in a case where the driving state of the driver and / or the driving state of the vehicle does not meet the respective corresponding safe driving conditions.

[0137] The device for monitoring driving state provided by the present disclosure obtains driving state data of a driver and vehicle driving state data; performs fusion processing on the driving state data and the vehicle driving state data in a time sequence to obtain target fusion features; analyzes the target fusion features based on a preset safe driving model algorithm, and determines whether the driving state of the driver and the driving state of the vehicle both meet the respective corresponding safe driving conditions according to an analysis result, the preset safe driving model algorithm being a safe driving model algorithm obtained from a cloud server; and determines that the vehicle is in unsafe driving in a case where the driving state of the driver and / or the driving state of the vehicle does not meet the respective corresponding safe driving conditions. Compared with related technologies, after the vehicle obtains the driving state data and the vehicle driving state data, the vehicle analyzes the driving state data and the vehicle driving state data based on the preset safe driving model algorithm obtained from the cloud server to determine whether the vehicle is in safe driving, directly obtains the driving state data and the vehicle driving state data from the vehicle side, thereby avoiding transmission time delay of uploading the driving state data and the vehicle driving state data to the cloud server, and further achieving timely monitoring of the driving state of the vehicle.

[0138] Figure 4 A structural schematic diagram of a device for monitoring driving state provided by an embodiment of the present disclosure is shown in FIG. 1. As shown in FIG. 1, the analysis unit 23 includes: Figure 4

[0139] The analysis module 231 is configured to analyze the target fusion features based on the preset safe driving model algorithm to obtain the analysis result, the preset safe driving model algorithm being a safe driving model algorithm obtained from the cloud server in a preset update period;

[0140] ​The determination module 232 is used to determine whether the driving state of the driver and the driving state of the vehicle meet the corresponding safe driving conditions according to the analysis results.

[0141] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 4 As shown, the processing unit 22 includes:

[0142] a processing module 221 for performing data preprocessing on the driving state data and the vehicle travel state data to obtain processed driving state data and processed vehicle travel state data;

[0143] An extraction module 222 is used to extract features from the processed driving state data and the processed vehicle travel state data to obtain driving state features and vehicle travel state features;

[0144] The fusion module 223 is configured to perform feature fusion on the driving state feature and the vehicle travel state feature with the same timestamp based on a preset fusion algorithm to obtain the target fusion feature.

[0145] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 4 As shown, the device includes:

[0146] a judgment unit 25, configured to re-judge whether the driving state of the driver and the driving state of the vehicle meet the corresponding safe driving conditions after a preset time period;

[0147] The control unit 26 is configured to control the vehicle according to a preset braking strategy when the driving state of the driver and / or the driving state of the vehicle do not meet the corresponding safe driving conditions.

[0148] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 4 As shown, the determination unit 24 is further used to analyze the target fusion features based on a preset safe driving model algorithm, and determine whether the driver's driving state and the vehicle's driving state meet their respective corresponding safe driving conditions according to the analysis results. If the driver's driving state and the vehicle's driving state meet their respective corresponding safe driving conditions, it is determined that the vehicle is driving safely and the driver's driving state data and the vehicle's driving state data are continued to be acquired.

[0149] It should be noted that the above explanation of the method embodiment is also applicable to the device of the embodiment of the present disclosure, and the principles are the same, which is no longer limited in the embodiment of the present disclosure.

[0150] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0151] Figure 5 A schematic block diagram of an example electronic device 300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0152] As shown in Figure 5 The device 300 includes a computing unit 301 that can perform various appropriate actions and processes in accordance with a computer program stored in a ROM (Read-Only Memory) 302 or a computer program loaded into a RAM (Random Access Memory) 303 from a storage unit 308. Various programs and data required for the operation of the device 300 can also be stored in the RAM 303. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. An I / O (Input / Output) interface 305 is also connected to the bus 304.

[0153] Various components in the device 300 are connected to the I / O interface 305, including an input unit 306, such as a keyboard, a mouse, etc., an output unit 307, such as various types of displays, speakers, etc., a storage unit 308, such as a magnetic disk, an optical disk, etc., and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the device 300 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0154] The computing unit 301 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 301 performs various methods and processes described above, such as the method of driving state monitoring. For example, in some embodiments, the method of driving state monitoring can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded onto the RAM 303 and executed by the computing unit 301, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the computing unit 301 can be configured to perform the aforementioned method of driving state monitoring by any other appropriate means, such as by means of firmware.

[0155] Various implementations of the systems and techniques described above herein can be realized in digital electronic circuitry, integrated circuitry, a Field Programmable Gate Array (FPGA), an Application-Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a System on Chip (SOC), a Complex Programmable Logic Device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0156] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0157] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage medium can include, without limitation, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory), or flash memory, fiber optics, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0158] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0159] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, and a blockchain network.

[0160] The computer system can include clients and servers. The clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server is one of communication and distribution, with the server receiving requests from the client and transmitting responses via the communication network. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system. The server can also be a server of a distributed system, or a server combined with a blockchain.

[0161] It should be noted that artificial intelligence is a discipline that studies enabling computers to simulate some thinking processes and intelligent behaviors of people (such as learning, reasoning, thinking, planning, etc.), which has both hardware and software technologies. Artificial intelligence hardware technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, etc.; artificial intelligence software technology mainly includes computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning, big data processing technology, knowledge graph technology, etc. several major directions.

[0162] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, each step described in the present disclosure can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions provided by the present disclosure can be achieved, which is not limited herein.

[0163] The above detailed description does not limit the scope of the disclosure. Various modifications, combinations, sub-combinations and alternatives can be made to the detailed description. Any modification, equivalent replacement and improvement etc. made within the spirit and principle of the disclosure shall be included in the scope of the disclosure.

Claims

1. A method of driving state monitoring, characterized by, The method comprises the following steps: obtaining driving state data of a driver and vehicle driving state data; fusing the driving state data and the vehicle driving state data in a time sequence to obtain target fusion features; analyzing the target fusion features based on a preset safe driving model algorithm, and determining whether the driving state of the driver and the driving state of the vehicle meet respective corresponding safe driving conditions according to an analysis result; the preset safe driving model algorithm is a safe driving model algorithm obtained from a cloud server; in the case that the driving state of the driver and / or the driving state of the vehicle does not meet the respective corresponding safe driving conditions, the vehicle is determined to be in unsafe driving.

2. The method of claim 1, wherein, The method comprises the following steps: obtaining driving state data of a driver and vehicle driving state data; fusing the driving state data and the vehicle driving state data in a time sequence to obtain target fusion features; 3. The method of claim 1, wherein, analyzing the target fusion features based on a preset safe driving model algorithm, and determining whether the driving state of the driver and the driving state of the vehicle meet respective corresponding safe driving conditions according to an analysis result; the preset safe driving model algorithm is a safe driving model algorithm obtained from a cloud server; in the case that the driving state of the driver and / or the driving state of the vehicle does not meet the respective corresponding safe driving conditions, the vehicle is determined to be in unsafe driving. The method comprises the following steps:

4. The method of claim 1, wherein, obtaining driving state data of a driver and vehicle driving state data; fusing the driving state data and the vehicle driving state data in a time sequence to obtain target fusion features; analyzing the target fusion features based on a preset safe driving model algorithm, and determining whether the driving state of the driver and the driving state of the vehicle meet respective corresponding safe driving conditions according to an analysis result; 5. The method according to any one of claims 1-4, characterized in that, the preset safe driving model algorithm is a safe driving model algorithm obtained from a cloud server; in the case that the driving state of the driver and / or the driving state of the vehicle does not meet the respective corresponding safe driving conditions, the vehicle is determined to be in unsafe driving.

6. An apparatus for monitoring a driving state, characterized by comprising: The method comprises the following steps: obtaining driving state data of a driver and vehicle driving state data; fusing the driving state data and the vehicle driving state data in a time sequence to obtain target fusion features; analyzing the target fusion features based on a preset safe driving model algorithm, and determining whether the driving state of the driver and the driving state of the vehicle meet respective corresponding safe driving conditions according to an analysis result; the preset safe driving model algorithm is a safe driving model algorithm obtained from a cloud server; in the case that the driving state of the driver and / or the driving state of the vehicle does not meet the respective corresponding safe driving conditions, the vehicle is determined to be in unsafe driving. The method comprises the following steps: obtaining driving state data of a driver and vehicle driving state data; fusing the driving state data and the vehicle driving state data in a time sequence to obtain target fusion features; analyzing the target fusion features based on a preset safe driving model algorithm, and determining whether the driving state of the driver and the driving state of the vehicle meet respective corresponding safe driving conditions according to an analysis result; the preset safe driving model algorithm is a safe driving model algorithm obtained from a cloud server; in the case that the driving state of the driver and / or the driving state of the vehicle does not meet the respective corresponding safe driving conditions, the vehicle is determined to be in unsafe driving. The method comprises the following steps: obtaining driving state data of a driver and vehicle driving state data; fusing the driving state data and the vehicle driving state data in a time sequence to obtain target fusion features; analyzing the target fusion features based on a preset safe driving model algorithm, and determining whether the driving state of the driver and the driving state of the vehicle meet respective corresponding safe driving conditions according to an analysis result; the preset safe driving model algorithm is a safe driving model algorithm obtained from a cloud server; in the case that the driving state of the driver and / or the driving state of the vehicle does not meet the respective corresponding safe driving conditions, the vehicle is determined to be in unsafe driving. The method comprises the following steps: obtaining driving state data of a driver and vehicle driving state data; fusing the driving state data and the vehicle driving state data in a time sequence to obtain target fusion features; analyzing the target fusion features based on a preset safe driving model algorithm, and determining whether the driving state of the driver and the driving state of the vehicle meet respective corresponding safe driving conditions according to an analysis result; the preset safe driving model algorithm is a safe driving model algorithm obtained from a cloud server; in the case that the driving state of the driver and / or the driving state of the vehicle does not meet the respective corresponding safe driving conditions, the vehicle is determined to be in unsafe driving. a processing unit, configured to perform fusion processing on the driving state data and the vehicle running state data in a time sequence to obtain target fusion features; an analysis unit, configured to analyze the target fusion features based on a preset safe driving model algorithm, and determine whether the driving state of the driver and the running state of the vehicle both conform to respective corresponding safe running conditions according to an analysis result, the preset safe driving model algorithm being a safe driving model algorithm obtained from a cloud server; a determination unit, configured to determine that the vehicle is in unsafe running in a case where the driving state of the driver and / or the running state of the vehicle does not conform to the respective corresponding safe running conditions.

7. A vehicle characterized by comprising: The device for monitoring the driving state according to claim 6.

8. An electronic device, comprising: comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-5.

9. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method according to any one of claims 1-5.

10. A computer program product, characterised in that, The computer program, when executed by a processor, implements the method according to any one of claims 1-5.

Images