Driving behavior monitoring method and device based on multi-modal data, vehicle-mounted edge computing equipment and medium

Abstract

The invention relates to a driving behavior monitoring method and device based on multi-modal data, a vehicle-mounted edge computing device and a medium, and belongs to the technical field of intelligent monitoring, and the method comprises the steps: obtaining the physiological information of a driver, the video information of the driver in a cab, the vehicle position and the vehicle speed; preliminarily determining whether a driving behavior is abnormal or not according to the physiological information and the video information; in response to the fact that the preliminary determination result is abnormal, whether the driving behavior is abnormal or not is rechecked according to the vehicle position and the vehicle speed; and in response to the re-checking result being abnormal, generating early warning information for respectively sending early warning signals to a control center and a driver. The method has the effect of improving the monitoring accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent monitoring, and in particular to a driving behavior monitoring method, apparatus, vehicle-mounted edge computing device and medium based on multimodal data. Background Art

[0002] Fatigue driving is one of the main factors leading to traffic accidents, so accurate assessment of driver fatigue is helpful to reduce the occurrence of traffic accidents. Especially for drivers of buses, shuttle buses, shuttle buses, commuter buses or other public transportation vehicles, fatigue monitoring is of greater significance to improving public safety.

[0003] At present, driver monitoring mainly relies on traditional on-board cameras for visual monitoring or single physiological monitoring equipment. The judgment method is single, and there are shortcomings such as low detection accuracy and limited usage scenarios. It is impossible to accurately judge whether the driver's actual driving behavior is abnormal. Summary of the invention

[0004] In order to improve monitoring accuracy, the present application provides a driving behavior monitoring method, apparatus, vehicle-mounted edge computing device and medium based on multimodal data.

[0005] In a first aspect, the present application provides a driving behavior monitoring method based on multimodal data, which adopts the following technical solution: Obtaining the driver's physiological information, the driver's video information in the cab, the vehicle's position and the vehicle's speed; preliminarily determining whether the driving behavior is abnormal based on the physiological information and the video information; In response to the preliminary determination result being abnormal, whether the driving behavior is abnormal is reviewed based on the vehicle position and the vehicle speed; in response to the review result being abnormal, warning information is generated for sending warning signals to a control center and a driver respectively.

[0006] By adopting the above technical solution, the driver's physiological information can reflect the driver's fatigue level and tension state, and the video information can record the driver's driving behavior in real time, and then comprehensively judge whether the driving behavior is abnormal through physiological signals and behavioral characteristics, and verify the accuracy of the abnormality again through the vehicle's driving position and speed. Therefore, physiological feedback, image recognition and positioning data are integrated to comprehensively monitor driving behavior with higher accuracy and stronger real-time performance.

[0007] Further, the preliminarily determining whether the driving behavior is abnormal based on the physiological information and the video information includes: Determining behavior features according to the video information, wherein the behavior features include emotional features and action features, including: Acquiring image information based on the video information; Performing face detection on the image information; If a human face is detected in the image information, extracting facial expression features based on the image information; Inputting the facial expression features into an emotion recognition classification model trained by a deep learning algorithm to determine the driver's emotional features; Inputting the image information into an action recognition model trained by a deep learning algorithm to determine the action characteristics of the driver; If no human face is detected in the image information, determining that the driver's behavior characteristic is leaving the driving seat; Inputting the physiological information and the behavioral characteristics into a driving behavior monitoring model trained by a deep learning algorithm to determine whether the driving behavior is abnormal; If abnormal, the result is initially determined to be abnormal; If there is no abnormality, the preliminary result is determined to be normal.

[0008] By adopting the above technical solution, after obtaining image information from the video, first determine whether the driver has left the seat based on whether there is a face, and then further use the image information to identify emotions and actions when the driver has not left the seat, so as to comprehensively judge the driver's behavioral characteristics, analyze the image from all aspects, monitor the driver's driving behavior visually, improve monitoring accuracy, and then use the driving behavior monitoring model trained by the deep learning algorithm to analyze physiological information and behavioral characteristics, so as to further accurately judge whether the driving behavior is abnormal.

[0009] Furthermore, before inputting the physiological information and the behavioral characteristics into the driving behavior monitoring model trained by the deep learning algorithm, the method further includes: Respectively obtaining timestamps corresponding to the physiological information and the behavioral characteristics; The physiological information and the behavioral characteristics corresponding to the same timestamp are synchronized to obtain a set of input data for simultaneous input into a driving behavior monitoring model.

[0010] By adopting the above technical solution, the on-board edge computing device synchronizes physiological information and behavioral characteristics through timestamps, so that it is more accurate when input into the driving behavior monitoring model for recognition.

[0011] Furthermore, the review of whether the driving behavior is abnormal based on the vehicle position and the vehicle speed includes: comparing the vehicle position with the driving route to determine whether the vehicle is located within a preset range of any station on the driving route; and verifying whether the driving behavior is abnormal based on the determination result and the change in the vehicle speed.

[0012] Further, if the determination result is that the vehicle is located within a preset range of any station on the driving route, verifying whether the driving behavior is abnormal based on the determination result and the change in the vehicle speed includes: If the vehicle enters the station and the vehicle speed slows down, and the driver's behavior characteristics belong to the feature set of driving into the station, then it is determined that the driving behavior is normal; If the vehicle leaves the station and the vehicle speed is accelerated, and the driver's behavior characteristics belong to the feature set of driving away from the station, it is determined that the driving behavior is normal.

[0013] If the determination result is that the vehicle is not within a preset range of any station on the driving route, verifying whether the driving behavior is abnormal based on the determination result and the change in the vehicle speed includes: Determining the driving state of the vehicle according to the vehicle position and vehicle speed corresponding to a plurality of consecutive moments; If the driving state of the vehicle is any one of sudden braking, sharp turning or unstable driving trajectory, the driving behavior is determined to be abnormal; and / or, if the driving state of the vehicle is stopped and the driver's physiological information is normal, the driving behavior is determined to be normal; and / or, if the driving state of the vehicle is stopped and the driver's physiological information is abnormal, the driving behavior is determined to be abnormal.

[0014] By adopting the above technical solution, by comparing the vehicle position with the preset range of stations on the driving route, it is possible to accurately determine whether the vehicle is in a specific driving situation (such as entering a station, leaving a station, normal driving, etc.), so as to more specifically analyze the driver's driving behavior, and combine the changes in vehicle speed and the driver's behavioral characteristics, such as decelerating when entering a station and the behavior is consistent with the driving characteristics of entering a station, accelerating when leaving a station and the behavior is consistent with the driving characteristics of leaving a station, it is possible to more accurately determine whether the driver's driving behavior is in compliance with the regulations.

[0015] Furthermore, the method further includes: synchronously displaying the physiological information, the video information, the vehicle position and the vehicle speed, and sending them to a control center.

[0016] By adopting the above technical solutions, the information can be synchronously displayed and sent to the control center in real time, which can significantly improve the real-time performance of data transmission, resource optimization, system scalability, data transmission reliability, and comprehensive monitoring and decision support capabilities. These technical effects together provide a strong guarantee for the intelligentization, safety, and efficiency of public transportation systems.

[0017] In the second aspect, the present application provides a bus driving behavior monitoring device based on multimodal data, which adopts the following technical solution: A data acquisition module, used to acquire the driver's physiological information, the driver's video information in the cab, the vehicle's position and the vehicle's speed; A preliminary determination module, used to preliminarily determine whether the driving behavior is abnormal based on the physiological information and the video information; A review module, used for reviewing whether the driving behavior is abnormal according to the vehicle position and the vehicle speed when the preliminary determination result of the preliminary determination module is abnormal; The early warning module is used to generate early warning information when the review result of the review module is abnormal, and is used to send early warning signals to the control center and the driver respectively.

[0018] By adopting the above technical solution, the physiological information of the driver obtained by the data acquisition module can reflect the driver's fatigue level and tension state, the video information can record the driver's driving behavior in real time, the preliminary determination module comprehensively judges whether the driving behavior is abnormal through physiological signals and behavioral characteristics, and in the event of an abnormality, the review module verifies whether the abnormality is accurate again through the vehicle's driving position and speed. Therefore, physiological feedback, image recognition and positioning data are integrated to comprehensively monitor driving behavior with higher accuracy and stronger real-time performance.

[0019] In a third aspect, the present application provides a vehicle-mounted edge computing device, which adopts the following technical solution: A vehicle-mounted edge computing device, comprising: at least one processor; Memory; At least one computer program, wherein the at least one computer program is stored in the memory and configured to be executed by the at least one processor, the at least one computer program being configured to: execute the method as described in any one of the first aspects.

[0020] By adopting the above technical solution, the processor executes the computer program in the memory to obtain the driver's physiological information, which can reflect the driver's fatigue level and tension state, obtain video information, record the driver's driving behavior in real time, and obtain the driver's behavioral characteristics, including facial expressions, body movements, etc., and comprehensively judge whether the driving behavior is abnormal through physiological signals and behavioral characteristics, and in case of abnormality, verify the accuracy of the abnormality again through the vehicle's driving position and speed. Therefore, physiological feedback, image recognition and positioning data are integrated to comprehensively monitor driving behavior with higher accuracy and stronger real-time performance.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium stores a computer program that can be loaded by a processor and executes the method as described in any one of the first aspects.

[0022] By adopting the above technical solution, the processor executes the computer program in the computer-readable storage medium, obtains the driver's physiological information, which can reflect the driver's fatigue level and tension state, obtains video information, can record the driver's driving behavior in real time, and obtains the driver's behavioral characteristics, including facial expressions, body movements, etc., and comprehensively judges whether the driving behavior is abnormal through physiological signals and behavioral characteristics, and verifies the accuracy of the abnormality again through the vehicle's driving position and speed when it is abnormal. Therefore, it integrates physiological feedback, image recognition and positioning data to comprehensively monitor driving behavior with higher accuracy and stronger real-time performance.

[0023] In summary, the present application includes at least one of the following beneficial technical effects: 1. Obtaining the driver's physiological information can reflect the driver's fatigue and nervous state. Obtaining video information can record the driver's driving behavior in real time and obtain the driver's behavioral characteristics, including facial expressions, body movements, etc., and comprehensively judge whether the driving behavior is abnormal through physiological signals and behavioral characteristics. In the event of an abnormality, the abnormality is verified again through the vehicle's driving position and speed. Therefore, it integrates physiological feedback, image recognition and positioning data to comprehensively monitor driving behavior with higher accuracy and stronger real-time performance; 2. Synchronize physiological information and behavioral characteristics through timestamps, so that they are more accurate when input into the driving behavior monitoring model for recognition; 3. By comparing the vehicle position with the preset range of stations on the driving route, it is possible to accurately determine whether the vehicle is in a specific driving situation, thereby analyzing the driver's driving behavior more specifically, and combining the changes in vehicle speed and the driver's behavioral characteristics to more accurately determine whether the driver's driving behavior complies with the regulations. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of a driving behavior monitoring system based on multimodal data in an embodiment of the present application.

[0025] Figure 2 It is a flow chart of a driving behavior monitoring method based on multimodal data in an embodiment of the present application.

[0026] Figure 3 It is a schematic diagram of points selected around the eye in an embodiment of the present application.

[0027] Figure 4 It is a schematic diagram of selecting points around the mouth in an embodiment of the present application.

[0028] Figure 5 It is a structural block diagram of a driving behavior monitoring device based on multimodal data in an embodiment of the present application.

[0029] Figure 6It is a structural block diagram of the vehicle-mounted edge computing device in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0032] The embodiment of the present application discloses a driving behavior monitoring method for multimodal data, a driving behavior monitoring system for multimodal data, and is executed by an on-board edge computing device, which can be applied to buses, shuttle buses, shuttle buses, commuter buses, or other types of vehicles.

[0033] Reference Figure 1 The system includes a vehicle-mounted terminal and a control center. The vehicle-mounted terminal includes a physiological monitoring device worn by the driver, such as a smart watch or a heart rate belt, and also includes a camera installed in front of the vehicle's driving position. The camera captures video information of the driver during driving. Furthermore, the vehicle-mounted terminal also includes a positioning device installed on the vehicle. The positioning device can be a GPS positioning device or a Beidou positioning device. An encoder is also installed on the vehicle's wheel axle.

[0034] Physiological monitoring equipment, cameras, positioning devices and encoders all communicate with the on-board edge computing equipment. The control center communicates with the on-board edge computing equipment and can obtain monitoring data and warning information sent by the on-board edge computing equipment. It has a display device that can intuitively display the console interface, display and process vehicle warning information in real time, and ensure that supervisors can quickly identify and respond to potential safety issues. The control center also includes a data center for storing monitoring data and monitoring results.

[0035] Reference Figure 2 , the driving behavior monitoring method based on multimodal data includes (step S101 to step S105): Step S101: Acquire the driver's physiological information, the driver's video information in the cab, the vehicle's position and the vehicle's speed.

[0036] Specifically, multimodal data includes physiological data, vehicle data, video data and other modal data. During driving, the driver wears a physiological monitoring device in real time, and the on-board edge computing device communicates with the physiological monitoring device to obtain the driver's physiological information. Physiological information includes heart rate, blood pressure, and skin conductance. The on-board edge computing device communicates with the camera to obtain video information; communicates with the positioning device to obtain the vehicle position; and calculates the vehicle speed through encoder data.

[0037] Step S102: Preliminarily determine whether the driving behavior is abnormal based on the physiological information and the video information, including (steps S1021 to S1022): Step S1021: determining behavior characteristics according to the video information, wherein the behavior characteristics include emotion characteristics and action characteristics.

[0038] Specifically, the vehicle-mounted edge computing device needs to analyze the driver's behavior characteristics based on the video information, including (steps Sa to Sg): Step Sa: Obtain image information based on video information.

[0039] Specifically, the vehicle-mounted edge computing device obtains corresponding image information every preset time period.

[0040] Step Sb: Perform face detection on the image information.

[0041] Specifically, the on-board edge computing device adjusts the brightness and contrast of the image information and performs image enhancement. The pre-processed image information is clearer, and then face detection is performed to extract facial features.

[0042] Step Sc: If a face is detected in the image information, execute steps S14 to S16.

[0043] Specifically, the on-board edge computing device inputs the image information into the face recognition model, and can determine whether there is a face in the image information.

[0044] Step Sd: extracting facial expression features based on image information.

[0045] Specifically, the vehicle-mounted edge computing device uses a face detection algorithm to detect the face of the image information, such as a Haar feature classifier or a convolutional neural network, and then determines the position of the face in the image. The detected face image is aligned by detecting the key points of the face using a similarity transformation, where the key points of the face can be eyes, nose, mouth, etc. Similarity transformations such as rotation, scaling, and translation are used to eliminate the effects of different postures and angles on expression recognition. Finally, the vehicle-mounted edge computing device applies a pre-trained deep convolutional neural network model to extract expression features on the aligned face image.

[0046] Step Se: Input the facial expression features into the emotion recognition classification model trained by the deep learning algorithm to determine the emotional features of the bus driver.

[0047] Specifically, the on-board edge computing device inputs the extracted expression features into the emotion recognition classification model, performs expression classification, and identifies specific emotion features.

[0048] Among them, deep learning algorithms can use deep learning networks, deep convolutional generative adversarial networks, long short-term memory networks, and deep belief networks, etc. Emotional characteristics include happiness, sadness, anger, fear, surprise, and neutrality.

[0049] Step Sf: Input the image information into the action recognition model trained by the deep learning algorithm to determine the driver's action characteristics.

[0050] Specifically, the deep learning algorithm for training the action recognition model may include a convolutional neural network, a posture estimation model, a recurrent neural network, or a long short-term memory network, etc.

[0051] The on-board edge computing device inputs the image information into the action recognition model to identify the driver's action characteristics, including standing, drinking water, making phone calls, eating, talking, and turning the head to look around.

[0052] Step Sg: If no face is detected in the image information, it is determined that the driver's behavior feature is leaving the driving seat.

[0053] Specifically, if there is no face in the image information, it means that the driver is not in the cab, and then it can be determined that the driver's behavior feature is leaving the driving seat.

[0054] Step S1022: Input the physiological information and behavioral characteristics into the driving behavior monitoring model trained by the deep learning algorithm to determine whether the driving behavior is abnormal; if abnormal, the preliminary determination result is abnormal; if normal, the preliminary determination result is normal.

[0055] Specifically, the driver's state is closely related to physiology and behavior.

[0056] The driver's abnormal state includes abnormalities caused by physiological factors, psychological factors and behavioral factors.

[0057] Among them, abnormal conditions caused by physiological factors include fatigue driving. When driving while fatigued, the blinking frequency increases, the blinking amplitude increases, the eye closure time becomes longer, PERCLOSE increases, the line of sight direction becomes dull, the line of sight axis deviates from the normal range, and the eye movement slows down; the heart rate variability decreases, reflecting the decreased tension and balance of cardiac nerve activity, the breathing rate decreases, and the breathing becomes smoother; the head position changes abnormally, etc.

[0058] Abnormal conditions caused by physiological factors also include health problems, such as chronic diseases, which can be reflected in heart rate.

[0059] For example, when a driver suffers from a sudden heart attack, heart rate variability or blood pressure change rate may show abnormalities, and behavior may show facial expressions of tension or pain.

[0060] Abnormal conditions caused by psychological factors include mood swings, such as quarrels with passengers or traffic jams at work, which are manifested in increased heart rate and blood pressure due to excitement while driving, and facial expressions of excitement or frustration when communicating with others.

[0061] Abnormal conditions caused by behavioral factors include illegal driving and distracted driving. Vehicles may exceed the speed limit, run red lights, drive in the wrong direction, etc. Drivers may be distracted by using mobile phones, talking to passengers, or observing roadside scenery while driving.

[0062] Therefore, when training the driving behavior monitoring model, this application obtains a large amount of psychological and behavioral information corresponding to the driver when driving abnormally, and uses it as sample data. Public data sets such as BDD100K, Cityscapes, etc. can also be used. These data sets contain rich driving scenes and annotation information.

[0063] Preprocess the sample data, including data cleaning, data labeling, and data enhancement.

[0064] Select appropriate deep learning models, such as convolutional neural networks (CNN), recurrent neural networks (RNN), long short-term memory networks (LSTM), and target detection algorithms such as YOLO, and use the processed data set to train the model. During the training process, it is necessary to continuously adjust the model parameters and hyperparameters to achieve the best performance. Use an independent test data set to evaluate the trained model. Evaluation indicators can include accuracy, recall rate, F1 score, etc. Then train the driving behavior monitoring model.

[0065] The on-board edge computing device inputs physiological information and behavioral characteristics into the driving behavior monitoring model trained by the deep learning algorithm, and can determine whether the driver's driving behavior is abnormal.

[0066] In order to accurately identify the driver's physiological and behavioral information, before inputting the physiological information and behavioral characteristics into the driving behavior monitoring model, the method further includes: The timestamps corresponding to the physiological information and the behavioral characteristics are obtained respectively; the physiological information and the behavioral characteristics corresponding to the same timestamp are synchronized to obtain a set of input data for simultaneous input into the driving behavior monitoring model.

[0067] In another possible implementation, the facial expression feature includes an eye feature. For example, when the driver is tired, he frequently closes his eyes within a preset time period. Then step Sd may include (steps S11 to S15): Step S11: Acquire a plurality of image information of a preset time period.

[0068] Step S12: Establish the same coordinate system in each image information, and select points p around the eyeball in sequence. i , 0≤i≤5, determine the coordinates of each point.

[0069] Specifically, PERCLOS (percentage of eyelid closure over the pupil over time) is used to describe the driver's fatigue state.

[0070] For example, refer to Figure 3 , select 6 points around the eyes in the image information, that is, select one point at the beginning and end of the eye, and select two points at the upper and lower edges of the eye.

[0071] Step S143: Calculate the PERCLOS of the eye according to the coordinates of each point, as shown in the formula: Among them, p i is the coordinate of point i, |p a -p b | is the Euclidean distance between point a and point b.

[0072] Step S14: If percloysEYE is lower than a first preset value, it is determined that the bus driver is in a closed-eye state in the current image information.

[0073] Step S15: If it is determined according to each image information that the number of times the eye is closed within the preset time period is higher than the threshold, the number of times the eye is closed within the preset time period is determined as an eye feature.

[0074] Specifically, if the number of times the driver closes his eyes within a preset time period is higher than a threshold, the eye feature is obtained as the number of times the driver closes his eyes within the preset time period, and after inputting it into the emotion recognition classification model, the emotional feature can be obtained as fatigue.

[0075] In another possible implementation, the facial expression feature also includes a mouth feature. When the driver is tired, he or she may relieve sleepiness by yawning. Therefore, step Sd may also include (steps S21 to S25): Step S21: Acquire a plurality of image information of a preset time period.

[0076] Step S22: Establish the same coordinate system in each image information, and select points p around the mouth in sequence i , 0≤i≤7, determine the coordinates of each point.

[0077] Specifically, using the same principle, PERCLOS is applied to monitor mouth features.

[0078] For example, refer to Figure 4 , select 8 points around the mouth in the image information, that is, select one point at each of the two corners of the mouth, and select three points at the upper and lower edges of the mouth.

[0079] Step S23: Calculate the PERCLOS of the mouth according to the coordinates of each point, as shown in the formula: Among them, p i is the coordinate of point i, |p a -p b | is the Euclidean distance between point a and point b.

[0080] Step S24: If percloysMouth is lower than a second preset value, it is determined that the driver is in a yawning state in the current image information.

[0081] Step S25: If it is determined that the number of yawning states within the preset time period determined by each image information is higher than the threshold, the number of yawning states within the preset time period is determined as a mouth feature.

[0082] Specifically, if the number of yawns of the driver within a preset time period is higher than a threshold, the mouth feature is obtained as the number of yawns within the preset time period, and then after inputting it into the emotion recognition classification model, it can be obtained that the emotion feature is fatigue.

[0083] If the preliminary determination result is abnormal, step S103 is executed: in response to the preliminary determination result being abnormal, whether the driving behavior is abnormal is reviewed based on the vehicle position and vehicle speed.

[0084] Specifically, when a public vehicle enters or exits a station, the driver may look around because he needs to observe the road conditions around the station. Such actions may cause the driving behavior monitoring model to identify it as abnormal driving. In order to check whether it is a misjudgment caused by entering or exiting the station, the judgment will continue to be made based on the vehicle position and speed.

[0085] Step S103 includes (step S1031 to step S1032): Step S1031: Compare the vehicle position with the driving route to determine whether the bus is within a preset range of any station on the driving route.

[0086] Specifically, the on-board edge computing device presets the driving route of each vehicle, and then determines whether the vehicle is traveling on the prescribed driving route based on the vehicle's position. Moreover, when the vehicle is within the preset range of any station on the driving route, it can be determined that the vehicle is about to enter the station, stop at the station, or is about to exit the station.

[0087] Step S1032: Verify whether the driving behavior is abnormal based on the determination result and the change in vehicle speed.

[0088] If the determination result is that the vehicle is located within the preset range of any station on the driving route, step S1032 includes (steps S31 to S32): Step S31: If the vehicle enters the station and the vehicle speed slows down, and the driver's behavior characteristics belong to the feature set of driving into the station, it is determined that the driving behavior is normal.

[0089] Specifically, the feature set of driving into a station includes watching, paying attention to pedestrians, talking to passengers, or shaking left and right. Therefore, when the vehicle enters the station and the speed slows down, and the behavior characteristics are any of the above, it is determined that the driver's driving behavior is safe.

[0090] Step S32: If the vehicle leaves the station and the vehicle speed is accelerated, and the driver's behavior characteristics belong to the feature set of driving away from the station, it is determined that the driving behavior is normal.

[0091] Specifically, the feature set of driving away from the station includes waiting, paying attention to pedestrians, looking left and right, etc. Therefore, the vehicle's position, speed and the driver's behavior characteristics are comprehensively considered to comprehensively judge that the driver's behavior is normal.

[0092] If the determination result is that the vehicle is not located within the preset range of any station on the driving route, step S1032 includes: determining the driving state of the vehicle according to the vehicle positions and vehicle speeds corresponding to multiple consecutive moments.

[0093] A: If the vehicle's driving state is any of sudden braking, sharp turning or unstable driving trajectory, it is determined that the driving behavior is abnormal.

[0094] Specifically, if the vehicle is not at the station or leaving the station, and the bus is driving unstably, when the driver's physiological information and behavioral characteristics are abnormal, it can be determined that the driving behavior is abnormal.

[0095] B: If the vehicle's driving state is stopped and the driver's physiological information is normal, it is determined that the driving behavior is normal.

[0096] Specifically, if any one of the physiological information and behavioral characteristics is abnormal, the driving behavior monitoring model may identify the driving behavior as abnormal. If the vehicle is not arriving at or leaving the station, but is stopped, the driver may be waiting for a red light, stuck in traffic, or stopped for repairs. At this time, the physiological signal is normal, and the driving behavior may be determined to be normal even if the behavioral characteristics are abnormal.

[0097] C: If the driving state of the vehicle is stopped and the driver's physiological information is abnormal, it is determined that the driving behavior is abnormal.

[0098] Specifically, if the driver's physiological information is abnormal and the driving state is stopped, it cannot be ruled out that the driver has an emergency condition such as illness.

[0099] Through the above analysis, step S103 can determine whether the review result is normal or abnormal.

[0100] If the review result is abnormal, step S104 is executed: in response to the review result being abnormal, warning information is generated for sending warning signals to the control center and the driver respectively.

[0101] Specifically, if the on-board edge computing device verifies that the driving behavior is abnormal through comprehensive analysis of the driver's physiological information and behavioral characteristics, as well as the vehicle's location and speed, it will immediately generate a warning message.

[0102] The warning information can be sent to the control center and the cab respectively, or directly to the driver's monitoring device, such as by sending information to a smart watch to make the smart watch vibrate.

[0103] The control center has an intuitive console interface for real-time display and processing of vehicle warning information, ensuring that supervisors can quickly identify and respond to potential safety issues. It is also a centralized information management system for comprehensive management of vehicles, routes, drivers and stations, improving operational efficiency and data transparency. It also includes a data center that can store and analyze historical data, presenting data in various forms such as curve charts, distribution radar charts and tables to provide data support for decision-making, and can provide incident tracing analysis of abnormal behavior and warnings.

[0104] For example, after receiving the warning information, the control center will display it in real time on the console interface, and at the same time, through communication with the vehicle terminal, it will contact the driver in time to assist in solving abnormal driving behavior, and automatically alarm when the driver's life is in danger. Afterwards, the supervisor can analyze and trace back through the data in the data center. The warning information can also be played in the cab as a sound or visual signal to remind the driver to drive properly in time and improve driving safety.

[0105] When an abnormal warning is issued, the on-board edge computing device immediately pushes the content to the connected clients, such as the control center and the cab terminal, without waiting for the client to request new data, ensuring that the latest warning data can be displayed and browsed when the physiological information and video information can be obtained while displaying the vehicle's location, thereby improving the monitoring response speed and user experience, and ensuring the timeliness and completeness of the message.

[0106] Step S105: If the preliminary determination result is normal or the review result is normal, the driving behavior is determined to be normal.

[0107] In another possible implementation, the method further includes: synchronously displaying the physiological information, video information, vehicle position and vehicle speed, and sending them to a control center.

[0108] Specifically, the vehicle-mounted edge computing device obtains physiological information, video information, vehicle position and vehicle speed respectively sent using the MQTT protocol. The MQTT (Message Queuing Telemetry Transport) protocol is a message publishing and subscription transmission protocol that is lightweight, simple, open and easy to implement, and has a wide range of uses. The vehicle-mounted edge computing device uses the MQTT protocol to obtain data from each monitoring device in real time, and synchronously displays physiological information, video information, vehicle position and vehicle speed, which can be displayed in the cab or sent synchronously to the control center, which can observe the vehicle driving status in real time.

[0109] In order to better implement the above method, the present application embodiment also provides a driving behavior monitoring device for multimodal data, referring to Figure 5 , the driving behavior monitoring device 200 based on multimodal data includes: The data acquisition module 201 is used to acquire the driver's physiological information, the driver's video information in the cab, the vehicle's position and the vehicle's speed; A preliminary determination module 202, for preliminarily determining whether the driving behavior is abnormal based on the physiological information and the video information; A review module 203 is used to review whether the driving behavior is abnormal according to the vehicle position and vehicle speed when the preliminary determination result of the preliminary determination module 202 is abnormal; The early warning module 204 is used to generate early warning information when the review result of the review module 203 is abnormal, and to send early warning signals to the control center and the driver respectively.

[0110] Furthermore, the preliminary determination module 202 is specifically used to: Determine behavior features based on video information, where the behavior features include emotional features and action features, including: Acquiring image information based on video information; Perform face detection on image information; If a face is detected in the image information, the facial expression features are extracted based on the image information; Input the facial expression features into the emotion recognition classification model trained by the deep learning algorithm to determine the driver's emotional characteristics; The image information is input into the action recognition model trained by the deep learning algorithm to determine the driver's action characteristics; If no human face is detected in the image information, the driver's behavior characteristic is determined to be leaving the driving seat; Input physiological information and behavioral characteristics into the driving behavior monitoring model trained by the deep learning algorithm to determine whether the driving behavior is abnormal; If abnormal, the result is initially determined to be abnormal; If there is no abnormality, the preliminary result is determined to be normal.

[0111] Furthermore, the driving behavior monitoring device 200 based on multimodal data further includes: A timestamp acquisition module, used to respectively acquire timestamps corresponding to physiological information and behavioral characteristics; The synchronization module is used to synchronize the physiological information and behavioral characteristics corresponding to the same timestamp to obtain a set of input data for simultaneous input into the driving behavior monitoring model.

[0112] Furthermore, the review module 203 is specifically used for: Compare the vehicle's location with the bus route to determine whether the vehicle is within the preset range of any station on the route; based on the determination result and changes in vehicle speed, verify whether the driving behavior is abnormal.

[0113] If the determination result is that the vehicle is within the preset range of any station on the driving route, the review module 203 verifies whether the driving behavior is abnormal based on the determination result and the change in vehicle speed, specifically for: If the vehicle enters the station and the vehicle speed slows down, and the driver's behavior characteristics belong to the feature set of driving into the station, then the driving behavior is determined to be normal; If the vehicle leaves the station and the vehicle speed increases, and the driver's behavior characteristics belong to the feature set of driving away from the station, the driving behavior is determined to be normal; If the determination result is that the vehicle is not within the preset range of any station on the driving route, the review module 203 verifies whether the driving behavior is abnormal based on the determination result and the change in the vehicle speed, specifically for: Determine the driving state of the vehicle according to the vehicle position and vehicle speed corresponding to a plurality of consecutive moments; If the vehicle's driving state is any of sudden braking, sharp turning or unstable driving trajectory, the driving behavior is determined to be abnormal; and / or, if the vehicle's driving state is stopped and the driver's physiological information is normal, the driving behavior is determined to be normal; and / or, if the vehicle's driving state is stopped and the driver's physiological information is abnormal, the driving behavior is determined to be abnormal.

[0114] Furthermore, the driving behavior monitoring device 200 based on multimodal data further includes: The data synchronization transmission module is used to synchronously display physiological information, video information, vehicle position and vehicle speed, and send them to the control center in real time.

[0115] The various variations and specific examples of the methods in the aforementioned embodiments are also applicable to the driving behavior monitoring device based on multimodal data in the present embodiment. Through the aforementioned detailed description of the driving behavior monitoring method based on multimodal data, those skilled in the art can clearly understand the implementation method of the driving behavior monitoring device based on multimodal data in the present embodiment. Therefore, for the sake of brevity of the specification, it will not be described in detail here.

[0116] In order to better implement the above method, the embodiment of the present application provides a vehicle-mounted edge computing device, referring to Figure 6 , the vehicle-mounted edge computing device 300 includes: a processor 301, a memory 303 and a display screen 305. Among them, the memory 303 and the display screen 305 are connected to the processor 301, such as through a bus 302. Optionally, the vehicle-mounted edge computing device 300 may also include a transceiver 304. It should be noted that in actual applications, the transceiver 304 is not limited to one, and the structure of the vehicle-mounted edge computing device 300 does not constitute a limitation on the embodiments of the present application.

[0117] Processor 301 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0118] The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 302 may be divided into an address bus, a data bus, a control bus, etc.

[0119] The memory 303 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0120] The memory 303 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the contents shown in the above method embodiment.

[0121] Figure 6 The vehicle-mounted edge computing device 300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0122] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, the driving behavior monitoring method based on multimodal data provided in the above embodiment is implemented. The processor executes the computer program in the computer-readable storage medium to obtain the driver's physiological information, which can reflect the driver's fatigue level and tension state, obtain video information, and can record the driver's driving behavior in real time, and obtain the driver's behavioral characteristics, including facial expressions, body movements, etc. The driving behavior model trained by the deep learning algorithm can comprehensively judge whether the driving behavior is abnormal through physiological signals and behavioral characteristics, and verify the accuracy of the abnormality again through the vehicle's driving position and speed when it is abnormal. Therefore, physiological feedback, image recognition and positioning data are integrated to comprehensively monitor driving behavior with higher accuracy and stronger real-time performance.

[0123] In this embodiment, the computer-readable storage medium may be a tangible device that holds and stores instructions used by the instruction execution device. The computer-readable storage medium may be, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof. Specifically, the computer-readable storage medium may be a portable computer disk, a hard disk, a USB flash drive, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, an optical disk, a magnetic disk, a mechanical encoding device, and any combination thereof.

[0124] The computer program in this embodiment includes program codes for executing all the aforementioned methods, and the program codes may include instructions corresponding to the steps of the methods provided in the above embodiments. The computer program can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The computer program can be executed entirely on a user's computer or as an independent software package.

[0125] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

[0126] In addition, it is to be understood that relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprises," "comprising," or any other variation thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or elements that are inherent to such process, method, article, or device.

Claims

1. A driving behavior monitoring method based on multimodal data, characterized in that: include: Obtaining the driver's physiological information, the driver's video information in the cab, the vehicle's position and the vehicle's speed; preliminarily determining whether the driving behavior is abnormal based on the physiological information and the video information; In response to the preliminary determination result being abnormal, reviewing whether the driving behavior is abnormal based on the vehicle position and the vehicle speed; In response to the review result being abnormal, warning information is generated for sending warning signals to the control center and the driver respectively.

2. The method according to claim 1, characterized in that The preliminarily determining whether the driving behavior is abnormal based on the physiological information and the video information includes: Determining behavior features according to the video information, wherein the behavior features include emotional features and action features, including: Acquiring image information based on the video information; Performing face detection on the image information; If a human face is detected in the image information, extracting facial expression features based on the image information; Inputting the facial expression features into an emotion recognition classification model trained by a deep learning algorithm to determine the driver's emotional features; Inputting the image information into an action recognition model trained by a deep learning algorithm to determine the action characteristics of the driver; If no human face is detected in the image information, determining that the driver's behavior characteristic is leaving the driving seat; Inputting the physiological information and the behavioral characteristics into a driving behavior monitoring model trained by a deep learning algorithm to determine whether the driving behavior is abnormal; If abnormal, the result is initially determined to be abnormal; If there is no abnormality, the preliminary result is determined to be normal.

3. The method according to claim 2, characterized in that Before inputting the physiological information and the behavioral characteristics into the driving behavior monitoring model trained by the deep learning algorithm, the method further includes: Respectively obtaining timestamps corresponding to the physiological information and the behavioral characteristics; The physiological information and the behavioral characteristics corresponding to the same timestamp are synchronized to obtain a set of input data for simultaneous input into a driving behavior monitoring model.

4. The method according to claim 1, characterized in that The step of reviewing whether the driving behavior is abnormal according to the vehicle position and the vehicle speed includes: Comparing the vehicle position with the driving route to determine whether the vehicle is within a preset range of any station on the driving route; Based on the determination result and the change in the vehicle speed, it is verified whether the driving behavior is abnormal.

5. The method according to claim 4, characterized in that If the determination result is that the vehicle is located within a preset range of any station on the driving route, verifying whether the driving behavior is abnormal based on the determination result and the change in the vehicle speed includes: If the vehicle enters the station and the vehicle speed slows down, and the driver's behavior characteristics belong to the feature set of driving into the station, then it is determined that the driving behavior is normal; If the vehicle leaves the station and the vehicle speed is accelerated, and the driver's behavior characteristics belong to the feature set of driving away from the station, it is determined that the driving behavior is normal.

6. The method according to claim 4, characterized in that If the determination result is that the vehicle is not within a preset range of any station on the driving route, verifying whether the driving behavior is abnormal based on the determination result and the change in the vehicle speed includes: Determining the driving state of the vehicle according to the vehicle position and vehicle speed corresponding to a plurality of consecutive moments; If the driving state of the vehicle is any one of sudden braking, sharp turning or unstable driving trajectory, the driving behavior is determined to be abnormal; and / or, if the driving state of the vehicle is stopped and the driver's physiological information is normal, the driving behavior is determined to be normal; and / or, if the driving state of the vehicle is stopped and the driver's physiological information is abnormal, the driving behavior is determined to be abnormal.

7. The method according to claim 1, characterized in that The method further comprises: The physiological information, the video information, the vehicle position and the vehicle speed are synchronously displayed and sent to a control center.

8. A bus driving behavior monitoring device based on multimodal data, characterized in that: include: A data acquisition module, used to acquire the driver's physiological information, the driver's video information in the cab, the vehicle's position and the vehicle's speed; A preliminary determination module, used to preliminarily determine whether the driving behavior is abnormal based on the physiological information and the video information; A review module, used for reviewing whether the driving behavior is abnormal according to the vehicle position and the vehicle speed when the preliminary determination result of the preliminary determination module is abnormal; The early warning module is used to generate early warning information when the review result of the review module is abnormal, and is used to send early warning signals to the control center and the driver respectively.

9. A vehicle-mounted edge computing device, characterized in that: include: at least one processor; Memory; At least one computer program, wherein the at least one computer program is stored in the memory and configured to be executed by the at least one processor, the at least one computer program being configured to: execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 7.

Images