A dangerous driving control method and system based on multi-feature fusion

Abstract

The present invention discloses a dangerous driving control method and system based on multi-feature fusion, comprising the following steps: obtaining driver state features and vehicle control features, and calculating the driving behavior riskiness based on a weight coefficient; inputting the driving behavior riskiness into a first neural network to determine whether the driver's current state involves dangerous driving, and issuing an alarm when dangerous driving occurs; after the alarm meets set conditions, sending vehicle information to a cloud control system, which plans a parking route for the current vehicle based on the vehicle information; switching vehicle control to the cloud control system, which controls the vehicle to park at a designated location based on the parking route; and releasing vehicle control from the cloud control system after safety is detected. The present invention not only provides auxiliary reminders but also actively intervenes in dangerous driving behavior, thereby fundamentally eliminating dangerous driving and possessing greater practicality and safety.

Description

Technical Field

[0001] The present invention relates to the field of vehicle safety control technology, and more specifically, to a dangerous driving control method and system based on multi-feature fusion. Background Art

[0002] Reckless driving is a major contributing factor to traffic accidents. This can occur when drivers are emotionally unstable or angry, and when they are extremely excited. Repeatedly engaging in reckless driving can create the potential for collisions with other vehicles, leading to traffic accidents. Reckless driving is a significant contributing factor to traffic accidents, with countless accidents caused by fatigue driving occurring annually. There are several factors that contribute to reckless driving: 1) anger; 2) extreme excitement; 3) competitiveness; and 4) fatigue.

[0003] Currently, existing dangerous driving detection methods rely on a single cue, typically detecting dangerous driving through signals from the driver or vehicle control. However, this approach suffers from low robustness and reliability, and a high rate of false positives. Alternatively, existing dangerous driving warning methods mostly alert drivers through voice prompts and other methods, failing to proactively prevent or intervene in dangerous driving.

[0004] The current dangerous driving warning system can only remind the vehicle, but cannot control the vehicle, and cannot fundamentally eliminate hidden dangers. Summary of the Invention

[0005] In order to solve the problem that the existing technology cannot actively intervene in dangerous driving behavior and cannot fundamentally eliminate dangerous driving by only providing auxiliary reminders when facing a driver who engages in dangerous driving behavior, the present invention provides a dangerous driving control method and system based on multi-feature fusion, which can not only provide auxiliary reminders, but also actively intervene in dangerous driving behavior, thereby fundamentally eliminating dangerous driving.

[0006] In order to achieve the above-mentioned purpose of the present invention, the technical solutions adopted are as follows:

[0007] A dangerous driving control method based on multi-feature fusion, the method comprising the following steps:

[0008] Obtain driver status characteristics and vehicle control characteristics, and calculate the driving behavior risk level based on the weight coefficient;

[0009] Inputting the driving behavior risk level into the first neural network to determine whether the driver is currently driving dangerously, and issuing an alarm if dangerous driving is present;

[0010] When the alarm prompt meets the set conditions, the vehicle information is sent to the cloud control system, and the cloud control system plans the parking route of the current vehicle based on the vehicle information;

[0011] Switching the vehicle control to the cloud control system, and having the cloud control system control the vehicle to park at a designated location according to the parking route;

[0012] After safety is verified, the cloud control system will release the vehicle control rights.

[0013] Preferably, the driver status characteristics are analyzed and judged by collecting the driver's facial image and voice information.

[0014] Furthermore, the driver's facial image is collected and analyzed to determine the driver's status characteristics, as follows:

[0015] The collected facial image of the driver is input into the second neural network to identify the driver's eyes and mouth, and then the third neural network is used to determine the opening and closing status of the mouth and eyes;

[0016] A driver is considered awake when their eyes are open and their mouth is closed;

[0017] A driver is considered drowsy when his eyes and mouth are closed;

[0018] When the driver's eyes and mouth are both open, the driver is considered to be in an agitated state.

[0019] Furthermore, the driver's state characteristics are analyzed and judged based on the collected voice information of the driver, as follows:

[0020] The collected driver's voice information is input into the convolutional neural network to extract the spectral features and voiceprint features in the speech signal. Then, the violent speech features are extracted through feature fusion. The classification neural network LSTM is used to identify whether violent speech is present in the voice and judge whether the driver's emotion is angry or normal.

[0021] Furthermore, the state feature time series of the entire driver's facial image or the state feature time series of the sound information is input into the long-short-term memory model; the cell state of the memory unit in the long-short-term memory model continuously updates and records the state feature time series of the facial image or sound information until it is output to the fusion network after completion, and is used to calculate the risk level of the driving behavior by adding a weight coefficient.

[0022] Preferably, the vehicle handling characteristics include sudden acceleration, sudden deceleration, steering wheel rotation speed, and vehicle speed.

[0023] Furthermore, several vehicle-side data information such as steering wheel rotation angle time series, accelerator pedal force time series, brake pedal force time series, and wheel speed time series are detected by vehicle-side sensors;

[0024] The vehicle-side data information is first preprocessed and then input into a deep convolutional neural network to extract several vehicle control feature time series, including sudden acceleration, sudden deceleration, steering wheel rotation speed, and vehicle speed.

[0025] Then, the vehicle control feature time series is input into the long short-term memory model to extract the feature information in the vehicle control feature time series;

[0026] The extracted feature information is input into the fusion network to calculate the risk level of driving behavior by adding weight coefficients.

[0027] Preferably, the weight coefficient is calculated based on the driver's previous driving behavior, and the vehicle control characteristics corresponding to the driver under different state characteristics are obtained through driving behavior analysis to calculate the weight coefficient under dangerous driving operation.

[0028] Prioritize, determine whether the driver is currently in dangerous driving state. Specifically, set a number threshold. When the driver's dangerous driving behavior reaches the set number threshold, it is determined that dangerous driving exists.

[0029] Preferably, the alarm prompt meets the set conditions, specifically, the alarm prompt exceeds a preset time threshold, or the alarm prompt exceeds a preset alarm number threshold.

[0030] Preferably, before switching the vehicle control right to the cloud control system, the cloud control system first obtains the owner's authorization for the cloud control system to control the vehicle and signs a safe driving agreement.

[0031] Preferably, while sending vehicle information to the cloud control system, the vehicle information is also broadcast to other surrounding vehicles; the vehicle information includes the vehicle's location information and ID information.

[0032] Preferably, the vehicle information is sent to a cloud control system, and the cloud control system also locks the vehicle position based on the vehicle information.

[0033] Preferably, a motion trajectory image of the vehicle traveling on the road is also obtained, and information on the vehicle's arbitrary lane changes and braking in the absence of other interfering vehicles is extracted based on the motion trajectory image; and information on competitive maneuvering with other vehicles when other vehicles are present is extracted to further determine whether there is dangerous driving; the vehicle information includes the vehicle's location information and ID information.

[0034] Preferably, the driving behavior risk is divided into different levels, and corresponding management and intervention strategies are implemented according to different levels.

[0035] A dangerous driving control system includes a vehicle, a base station, and a cloud control system;

[0036] Wherein, the vehicle executes the dangerous driving control method based on multi-feature fusion as described above;

[0037] The vehicle sends vehicle information to a cloud control system through a base station; the cloud control system plans a parking route for the current vehicle based on the vehicle information, and controls the vehicle to park at a designated location based on the parking route.

[0038] Preferably, it also includes a roadside camera for obtaining the motion trajectory image of the vehicle traveling on the road, and transmitting the motion trajectory image to the vehicle through the base station; the vehicle extracts the vehicle's arbitrary lane change and braking information in the absence of other interfering vehicles based on the motion trajectory image; and extracts the competitive control information between the vehicle and other vehicles when other vehicles are present, to further determine whether there is dangerous driving.

[0039] A dangerous driving control system includes a vehicle, a base station, an edge cloud computing center, and a cloud control system;

[0040] The vehicle collects the driver's facial image and voice information, as well as sudden acceleration, sudden deceleration, steering wheel rotation speed, and vehicle speed information through vehicle-side sensors;

[0041] The collected information is sent to the edge cloud computing center through the vehicle-side communication unit in the vehicle via the base station, and the edge cloud computing center executes the dangerous driving control method based on multi-feature fusion as described above;

[0042] When there is dangerous driving, the edge cloud computing center sends the vehicle information to the cloud control system; the cloud control system plans the parking route of the current vehicle based on the vehicle information, and controls the vehicle to park at a designated location according to the parking route.

[0043] Preferably, it also includes a road-side camera for obtaining the motion trajectory image of the vehicle traveling on the road, and transmitting the motion trajectory image to the edge cloud computing center through the base station; the edge cloud computing center extracts the vehicle's arbitrary lane change and braking information in the absence of other interfering vehicles based on the motion trajectory image; and extracts the competitive control information between the vehicle and other vehicles when other vehicles are present, to further determine whether there is dangerous driving.

[0044] The beneficial effects of the present invention are as follows:

[0045] Compared with the existing dangerous driving perception method that uses a single clue, the present invention comprehensively considers the driver's state characteristics and vehicle control characteristics, and determines whether the driver has dangerous driving behavior through multi-feature fusion. If so, an alarm prompts the driver to correct the driving behavior.

[0046] Compared to existing dangerous driving warning systems, which can only warn vehicles but cannot control them, and cannot fundamentally eliminate hidden dangers, the present invention can proactively intervene in vehicles that are driving dangerously. Specifically, if the driver still engages in dangerous driving after the alarm is triggered, the vehicle's control is transferred to the cloud control system. The cloud control system plans the current vehicle's parking route based on vehicle information and controls the vehicle to park at a designated location, allowing the vehicle to park at a suitable location. Only after confirming safety will the cloud control system's control of the vehicle be released. Therefore, compared to traditional methods, the present invention is more practical and safer. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of a dangerous driving control method based on multi-feature fusion described in the present invention.

[0048] Figure 2 This is a schematic diagram of the principle of analyzing the driver's facial image to determine the driver's status characteristics.

[0049] Figure 3 This is a schematic diagram of the principle of analyzing the driver's voice information to determine the driver's status characteristics.

[0050] Figure 4 It is a schematic diagram of dangerous driving analysis.

[0051] Figure 5 The invention relates to a flow chart of a method for implementing dangerous driving control.

[0052] Figure 6 This is a schematic diagram of a method for controlling dangerous driving on the vehicle side.

[0053] Figure 7 It is a scenario graph for implementing a dangerous driving control method.

[0054] Figure 8 This is another schematic diagram of dangerous driving analysis.

[0055] Figure 9 It is another scenario diagram for implementing dangerous driving control methods.

[0056] Figure 10 This is a schematic diagram of another method for implementing dangerous driving control on the vehicle side.

[0057] Figure 11 It is a flowchart of another method for implementing dangerous driving control.

[0058] In the figure, 110-vehicle, 120-base station, 130-cloud control system, 140-safe location, 150-edge cloud computing center, 160-roadside camera, 170-traffic police. DETAILED DESCRIPTION

[0059] The following describes the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art will readily appreciate the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the various details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are intended only to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0060] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0061] like Figure 1 As shown, a dangerous driving control method based on multi-feature fusion is provided, wherein the method comprises the following steps:

[0062] Obtain driver status characteristics and vehicle control characteristics, and calculate the driving behavior risk level based on the weight coefficient;

[0063] Inputting the driving behavior risk level into the first neural network to determine whether the driver is currently driving dangerously, and issuing an alarm if dangerous driving is present;

[0064] When the alarm prompt meets the set conditions, the vehicle information is sent to the cloud control system, and the cloud control system plans the parking route of the current vehicle based on the vehicle information;

[0065] Switching the vehicle control to the cloud control system, and having the cloud control system control the vehicle to park at a designated location according to the parking route;

[0066] After safety is verified, the cloud control system will release the vehicle control rights.

[0067] Compared with the existing dangerous driving perception method that uses a single clue, the present invention comprehensively considers the driver's state characteristics and vehicle control characteristics, and determines whether the driver has dangerous driving behavior through multi-feature fusion. If so, an alarm prompts the driver to correct the driving behavior.

[0068] Compared to existing dangerous driving warning systems, which can only warn vehicles but cannot control them, and cannot fundamentally eliminate hidden dangers, the present invention can proactively intervene in vehicles that are driving dangerously. Specifically, if the driver still engages in dangerous driving after the alarm is triggered, the vehicle's control is transferred to the cloud control system. The cloud control system plans the current vehicle's parking route based on vehicle information and controls the vehicle to park at a designated location, allowing the vehicle to park at a suitable location. Only after confirming safety will the cloud control system's control of the vehicle be released. Therefore, compared to traditional methods, the present invention is more practical and safer.

[0069] In this embodiment, if Figure 2 As shown, the driver status characteristics are analyzed and judged by collecting the driver's facial image and voice information, as follows:

[0070] The driver's state characteristics are determined by analyzing the collected driver's facial image as follows:

[0071] First, the vehicle-side smart camera collects the driver's image information, which is then calibrated by the camera. In the first stage, the second neural network detects the driver's eyes and mouth. In the second stage, the collected facial image of the driver is input into the third neural network to identify the driver's eyes and mouth. The third neural network then determines whether the mouth and eyes are open or closed.

[0072] A driver is considered awake when their eyes are open and their mouth is closed;

[0073] A driver is considered drowsy when his eyes and mouth are closed;

[0074] When the driver's eyes and mouth are both open, the driver is considered to be in an agitated state.

[0075] In this embodiment, the second neural network includes four models: Yolov5s, Yolov5m, Yolov5l, and Yolov5x. This embodiment adopts the YOLOV5 neural network, and the third neural network adopts the VGG network.

[0076] In this embodiment, Figure 3 As shown, the driver's state characteristics are analyzed and judged based on the collected driver's voice information, as follows:

[0077] The collected driver's voice information is input into a convolutional neural network to extract the spectral features and voiceprint features in the speech signal. The violent speech features are then extracted by fusing the spectral features and voiceprint features. The classification neural network LSTM is used to identify whether violent speech is present in the voice and to judge whether the driver's emotion is angry or normal.

[0078] In this implementation, drivers are required to record their voiceprint information before driving, ensuring that the subsequent voice recognition system can confirm that the collected voice information is indeed the driver's. After the driver's voice information is collected through the on-board microphone, the voice signal is preprocessed to obtain the sound spectrum and voiceprint graph. A deep convolutional neural network is used to extract the general features and voiceprint features of the voice signal. A feature fusion network is then used to identify the driver's voice signal and extract violent language from it. A classification neural network is then used to determine whether the driver is in a normal or angry state.

[0079] like Figure 4 As shown, in this embodiment, several vehicle-side data information including steering wheel rotation angle time series, accelerator pedal force time series, brake pedal force time series, and wheel speed time series are detected by vehicle-side sensors;

[0080] The vehicle-side data is preprocessed and then fed into a deep convolutional neural network to extract time series of vehicle control characteristics, including sudden acceleration, sudden deceleration, steering wheel speed, and vehicle speed. This preprocessing aligns and normalizes the vehicle-side data, initially filtering and eliminating useless information.

[0081] Then, the vehicle control feature time series is input into the long short-term memory model to extract the feature information in the vehicle control feature time series;

[0082] The extracted feature information is input into the fusion network to calculate the risk level of driving behavior by adding weight coefficients.

[0083] The driver's facial image is detected by a camera, and the driver's voice information is detected by a microphone. These are input into the corresponding deep convolutional neural network. The entire time series of the driver's facial image state features or the time series of the voice information state features are input into the long-short-term memory model. The cell state of the memory unit in the long-short-term memory model continuously updates and records the time series of the facial image or voice information state features until the state features are output to the fusion network, which then adds weight coefficients to calculate the driving behavior risk.

[0084] The deep convolutional neural network (CNN) is very effective at processing vehicle-side information in single-frame time series. However, for continuous time series data of vehicles, where there are temporal relationships and motion information between frames, the long short-term memory (LSTM) model is introduced. The gate structure is added to form a more complex cell unit. The gate structure enables the LTSM structure to forget and store contextual information, and can successfully extract feature information from vehicle time series.

[0085] In this embodiment, the vehicle control characteristics include sudden acceleration, sudden deceleration, steering wheel rotation speed, and vehicle speed. Specifically, vehicle control characteristics are collected through built-in sensors, such as a steering wheel torque detection sensor, brake and throttle force detection sensors, wheel speed sensors, and a combined inertial navigation GPS / IMU. These sensors collect information such as steering wheel torque, throttle and brake force, and wheel speed.

[0086] This embodiment collects multiple signals from vehicle-side sensors, including the driver's facial image, voice information, sudden acceleration and deceleration, steering wheel speed, and vehicle speed. By integrating these multiple features, it determines whether the driver is engaging in dangerous driving behavior. If so, the vehicle-side reminder system warns the driver to correct the behavior. If the driver continues to engage in dangerous driving after multiple reminders, vehicle control is transferred to the cloud-based control system, which then controls the vehicle and stops it at an appropriate location. The cloud-based control system then releases control of the vehicle, determining whether the driver's emotions are stable through an intelligent voice system and facial recognition system, or waiting for traffic police to intervene.

[0087] This embodiment determines whether the driver is currently engaging in dangerous driving. Specifically, a threshold number of dangerous driving behaviors is set. When the driver's dangerous driving behavior reaches the set threshold number of dangerous driving behaviors, the driver is deemed to be engaging in dangerous driving. Multiple detection results are used to determine whether the driver is indeed engaging in dangerous driving behavior, thereby improving the accuracy of the detection system and reducing the false alarm rate.

[0088] In this embodiment, the fusion network adds weighting coefficients to vehicle control characteristics and driver state characteristics. These weighting coefficients are derived from the driver's past driving behavior. Through driving behavior analysis, the corresponding vehicle control characteristics for different driver states are determined, and the weighting coefficients for dangerous driving maneuvers are calculated. After the weighting coefficients are added to the fused vehicle control characteristics and driver state characteristics, they are fed back into a deep convolutional neural network (CNN) to determine whether the driver's current state is normal or dangerous. A threshold number of dangerous driving behaviors is set. When the number of dangerous driving behaviors reaches the threshold, it indicates that the driver is indeed engaging in dangerous driving behavior, thereby reducing the neural network's false positive rate.

[0089] In this embodiment, the alarm prompt meets the set conditions, specifically, the alarm prompt exceeds a preset time threshold, or the alarm prompt exceeds a preset alarm number threshold.

[0090] In this embodiment, before switching the vehicle control right to the cloud control system, the cloud control system first obtains the owner's authorization for the cloud control system to control the vehicle and signs a safe driving agreement.

[0091] The cloud control system needs to obtain the owner's authorization to control the vehicle. When the owner activates the dangerous driving behavior safety detection function, the owner needs to authorize the cloud control system to automatically control the vehicle. When the driver engages in dangerous driving behavior, a safe driving agreement needs to be signed during the authorization process to ensure the legality of the cloud control system's control of the vehicle and provide a legal basis for automatic vehicle control.

[0092] In this embodiment, while sending vehicle information to the cloud control system, the vehicle information is also broadcast to other vehicles in the vicinity. The vehicle information includes the vehicle's location information and ID information. This allows other vehicles in the vicinity to avoid accidents.

[0093] In this embodiment, the vehicle information is sent to the cloud control system, and the cloud control system also locks the vehicle position based on the vehicle information and obtains the motion trajectory image of the vehicle on the road. Based on the motion trajectory image, the vehicle extracts the information of arbitrary lane changes and braking when there are no other interfering vehicles; it also extracts the competitive control information between the vehicle and other vehicles when other vehicles are present, and further determines whether there is dangerous driving. The vehicle information includes the vehicle's position information and ID information.

[0094] like Figure 5 As shown, on-board sensors collect multiple signals, including the driver's facial image, voice information, steering wheel movements, brake and accelerator pressure, and wheel speed. By integrating these multiple features, the system determines whether the driver is engaging in dangerous driving behavior. If so, the on-board warning system warns the driver to correct the behavior. If the on-board warning system fails to alert the driver to correct the behavior, indicating that the dangerous driving behavior has not been resolved, the vehicle's dangerous driving behavior is reported to the cloud control system via V2X communication technology, and the vehicle's information is broadcast to surrounding vehicles. The cloud control system then selects the nearest parking spot based on the vehicle's current location, plans the optimal route, and controls the vehicle to park in that area. After the vehicle is parked at a safe parking spot through the cloud control system, the vehicle communicates with the driver through the on-board intelligent voice, and uses the voice emotion analysis function to determine whether the driver's current emotional state is stable, and detect whether the driver's angry or excited expression has disappeared; when the driver's emotions are stable and his expression is normal, the cloud control system receives the driver's normal emotions and releases vehicle control through V2X communication technology; if there are traffic police or management departments near the vehicle, dangerous driving vehicles can also be handled manually. After the manual processing is completed, the management personnel send the processing results to the cloud control system, and the cloud control system releases automatic control of the vehicle, and the driver drives the vehicle.

[0095] In an embodiment, the driving behavior risk is divided into different levels, and corresponding management and intervention strategies are executed according to the different levels.

[0096] Driving behavior is classified into three levels of dangerousness: low, medium, and high. When a driver is in a low-level, controllable dangerous driving behavior, a dangerous warning warning will be issued through the cloud control system first. The dangerous driving warning information will be transmitted to the cloud control system through the vehicle network communication unit, and a reminder will be given through the vehicle-side voice broadcast. If the driver's dangerous driving behavior persists after the in-vehicle reminder system has issued five warnings, the dangerous driving behavior of the vehicle will be reported to the cloud control system, and the cloud control system will issue a violation penalty notice. The violation penalty notice will be transmitted to the vehicle-side system through the vehicle network communication unit, and the vehicle-side voice system will remind the driver that he has violated the traffic rules and broadcast the penalty notice. If the driver still engages in dangerous driving behavior after the violation penalty notice has been broadcast three times, the cloud control system platform will notify nearby traffic police 170 to rush to deal with the dangerous driving vehicle and warn the driver through voice broadcast that traffic police 170 is on their way to deal with the dangerous driving operation. When dangerous driving behavior is detected at the intermediate level, the cloud control system directly uploads the dangerous driving vehicle's information to the cloud control system, which then issues a penalty notice and warns the driver via voice broadcast. This can more quickly alert the driver to the dangerous operation and reduce the consequences of dangerous driving. If the dangerous operation still occurs, the cloud control system will notify the nearby traffic police 170 to rush to control the dangerous driving vehicle and warn the driver via voice broadcast. When the dangerous operation is detected at the advanced level and there is a risk of a major safety accident, the cloud control system directly reports the dangerous driving vehicle information, immediately notifies the nearby traffic police 170 to rush to control the dangerous driving vehicle, and warns the driver via voice broadcast that the traffic police 170 is coming.

[0097] The dangerous driving behavior risk level analysis method divides the risk level according to the different state values ​​of various extracted dangerous driving characteristics, such as the degree and frequency of changes in the driver's facial expressions, and the volume of the driver's voice. In addition, the vehicle side can use the above different numerical ranges to fuse and judge the different levels of dangerous driving according to the speed of change of the vehicle steering wheel angle, the acceleration, etc.

[0098] In a specific embodiment, Figure 6 As shown, a dangerous driving control system includes a vehicle, a base station, and a cloud control system;

[0099] Wherein, the vehicle executes the dangerous driving control method based on multi-feature fusion as described above;

[0100] The vehicle sends vehicle information to a cloud control system through a base station; the cloud control system plans a parking route for the current vehicle based on the vehicle information, and controls the vehicle to park at a designated location based on the parking route.

[0101] In this embodiment, a roadside camera is also included for acquiring a motion trajectory image of the vehicle traveling on the road, and transmitting the motion trajectory image to the vehicle via a base station; the vehicle extracts information on the vehicle's arbitrary lane change and braking in the absence of other interfering vehicles based on the motion trajectory image; and extracts information on competitive maneuvering with other vehicles when other vehicles are present, to further determine whether there is dangerous driving.

[0102] In this embodiment, the vehicle-side onboard computing unit is used to implement the dangerous driving control method based on multi-feature fusion as described above. Its specific application scenarios are as follows: Figure 6 As shown, the vehicle is equipped with a camera, microphone, steering wheel torque sensor, brake and throttle force sensor, wheel speed sensor, combined inertial navigation GPS / IMU, onboard computing unit, onboard communication unit, and vehicle control unit. The vehicle's intelligent camera recognizes the driver's facial expressions, the microphone captures the driver's voice, and the built-in sensors collect vehicle maneuverability, such as steering wheel torque, throttle and brake force, and wheel speed. This collected driver expression information is input into a deep neural network to determine whether the driver is angry, excited, or tired. The vehicle's maneuvers are then used to detect and measure dangerous driving. Finally, the system integrates this multi-faceted sensor information to determine if the vehicle is engaging in dangerous driving. If dangerous driving behavior is detected, the driver's behavior is retested. Multiple test results confirm the driver's actual behavior, thereby improving the detection system's accuracy and reducing false alarm rates. If dangerous driving behavior is detected, the vehicle control system restricts the driver's actions and receives control signals from the cloud control system via the base station and the vehicle-side communication unit, maneuvering the vehicle to a nearby safe parking space. When the vehicle stops moving, the intelligent voice assistant on the vehicle side will ask the driver about his mood and detect changes in the driver's expression to determine whether the driver's mood has returned to normal. If the driver's mood has returned to normal, the vehicle control will be released. Or if there is a traffic manager nearby, after the traffic police 170 has handled the driver's dangerous behavior and reported it to the cloud control system, a command to release the vehicle control will be sent to the vehicle side.

[0103] In this embodiment, as long as the computing power load on the vehicle side can reach a certain level, the calculations involved in the dangerous driving method described in this invention can be deployed on the vehicle side for calculation. This approach can save costs, but it has certain requirements for the computing power load on the vehicle side.

[0104] like Figure 8As shown, this embodiment uses multi-feature fusion to determine whether a driver has engaged in dangerous driving behavior. The fusion network calculates the driving behavior risk by adding weights to the driver's state features and vehicle control features. Finally, it integrates the vehicle feature information captured by the roadside camera and uses a neural network to determine whether the driver has engaged in dangerous driving behavior. In the first approach, when the vehicle can normally collect data about dangerously driving vehicles, the edge cloud computing center first uses the driver information and vehicle driving information collected by the vehicle to extract dangerous driving behavior features from the driver's facial image and voice information using a deep neural network. Furthermore, vehicle control features are analyzed based on the collected vehicle driving information. When dangerous driving behavior is detected, the vehicle information is reported to the cloud control system. The cloud control system locks the vehicle's position based on the vehicle's positioning information and ID information, and obtains camera image information on the road where the vehicle is traveling. Using the roadside camera video data, the onboard computing unit analyzes whether the suspected dangerous vehicle has engaged in dangerous driving behavior. The multi-feature fusion system analyzes multiple sensor information to ultimately determine whether the vehicle is engaged in dangerous driving. The second type is that when the dangerous driving data collection system on the vehicle side is destroyed or malfunctions, and the dangerous driving detection data uploaded by the vehicle side is lost, the road side determines the location of the vehicle through the basic data information uploaded by the vehicle, such as positioning information and ID information, and extracts the motion trajectory image of whether the vehicle has dangerous driving through the video data of the road-side camera. In this way, when the dangerous driving perception data is lost, dangerous driving vehicles on the road can also be detected.

[0105] In a specific embodiment, Figure 9 As shown, a dangerous driving control system includes a vehicle, a base station, an edge cloud computing center, and a cloud control system;

[0106] The vehicle collects the driver's facial image and voice information, as well as sudden acceleration, sudden deceleration, steering wheel rotation speed, and vehicle speed information through vehicle-side sensors;

[0107] The collected information is sent to the edge cloud computing center through the vehicle-side communication unit in the vehicle via the base station, and the edge cloud computing center executes the dangerous driving control method based on multi-feature fusion as described above;

[0108] When there is dangerous driving, the edge cloud computing center sends the vehicle information to the cloud control system; the cloud control system plans the parking route of the current vehicle based on the vehicle information, and controls the vehicle to park at a designated location according to the parking route.

[0109] This embodiment implements the calculations involved in the present invention through an edge cloud computing center, effectively reducing the computing load on the vehicle side and improving computing efficiency.

[0110] This embodiment also includes a roadside camera for acquiring motion trajectory images of vehicles traveling on the road, and transmitting the motion trajectory images to an edge cloud computing center via a base station; the edge cloud computing center extracts information on the vehicle's arbitrary lane changes and braking in the absence of other interfering vehicles based on the motion trajectory images; and extracts information on competitive maneuvers with other vehicles when other vehicles are present, to further determine whether there is any dangerous driving.

[0111] like Figure 9 As shown, on-board sensors collect driver data and vehicle driving data, and the detection data is transmitted to the edge cloud computing center via 120 base stations. When the edge cloud computing center detects dangerous driving behavior from the driver using a deep neural convolutional neural network based on the driver's facial image and vehicle control information collected by the on-board sensors, if the vehicle is suspected of dangerous driving behavior, the edge computing cloud platform obtains real-time motion video data from the vehicle based on the vehicle's positioning information, adds data about the vehicle's surrounding environment, and uses a fusion network to determine whether the driver is actually engaging in dangerous driving behavior. If dangerous driving behavior occurs and the dangerous driving operation has not been resolved after multiple reminders, the basic information of the dangerously driving vehicle is uploaded to the cloud control system for dangerous driving control operations. If the dangerous operation still occurs after the cloud control system issues a violation processing notice, the vehicle is notified to switch the vehicle controller to the cloud control system. The cloud control system controls the vehicle to park at a designated location according to the parking route and simultaneously notifies nearby traffic police through the cloud control system to rush to control the dangerous driver.

[0112] Figure 10This is a diagram of the vehicle-side system used in the application scenario of this embodiment. The vehicle-side dangerous driving detection system includes: a camera, a microphone, a steering wheel torque sensor, brake and throttle force sensors, a wheel speed sensor, a combined inertial navigation GPS / IMU, an onboard computing unit, and an onboard communication unit. The vehicle-side intelligent camera recognizes the driver's facial expressions, the microphone captures the driver's voice, and the built-in sensors collect vehicle maneuverability, such as steering wheel torque, throttle and brake force, and wheel speed. The captured driver's facial image information is uploaded to the edge cloud computing center and input into a deep neural network to determine whether the driver is angry or excited. Dangerous driving is then detected based on the vehicle's maneuvers. By integrating information from multiple sensors, dangerous driving behavior is identified. If dangerous driving behavior is detected, the vehicle's location is determined based on its positioning information and the vehicle-side ID. The roadside camera detects the vehicle's surroundings. Multi-source data fusion is used to determine whether the driver is indeed engaging in dangerous driving behavior, thereby improving the accuracy of the detection system and reducing false alarm rates. When dangerous driving behavior is detected, the cloud control system issues a warning or notifies nearby traffic police to address the situation. In addition, when the driver information or vehicle driving information collected by the vehicle side is lost, the video data provided by the road-side camera can be used to extract the characteristics of dangerous driving behavior using a deep convolutional neural network.

[0113] In this embodiment, if Figure 11 As shown, this embodiment collects multiple signals from the vehicle-side sensors, including the driver's facial image, sound information, steering wheel control, brake and accelerator force, wheel speed, etc., and uses the vehicle-side to upload the collected driver's facial expression information and vehicle-side driving information to the edge cloud computing center for calculation and processing through the vehicle-side communication unit, and uses multi-feature fusion to determine whether the driver has dangerous driving behavior. If so, according to the different types and severity of dangerous driving behavior, different warning information is sent to the vehicle-side through the vehicle-side communication unit to warn the driver to correct the driving behavior. If the driver still engages in dangerous driving after multiple reminders, the dangerous driving vehicle information is reported to the cloud control system, and the cloud control system sends a dangerous driving penalty notice to the vehicle-side, and reminds the penalty content through voice broadcast. If the driver still engages in dangerous driving after receiving the violation penalty notice, the cloud control system notifies the attached traffic police 170 to rush to control the dangerous driver.

[0114] This embodiment uses the edge cloud computing center for computing and processing, which is beneficial to reducing the computing load on the vehicle side and improving computing efficiency.

[0115] Obviously, the above embodiments of the present invention are merely examples for the purpose of illustrating the present invention, and are not intended to limit the embodiments of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A dangerous driving control method based on multi-feature fusion, characterized by: The method comprises the following steps: Obtain driver status characteristics and vehicle control characteristics, and calculate the driving behavior risk level based on the weight coefficient; Inputting the driving behavior risk level into the first neural network to determine whether the driver is currently driving dangerously, and issuing an alarm if dangerous driving is present; The method further includes: acquiring a motion trajectory image of the vehicle traveling on the road, extracting information about the vehicle's arbitrary lane changes and braking when no other interfering vehicles are present based on the motion trajectory image; and extracting information about competitive maneuvers with other vehicles when other vehicles are present, to further determine whether dangerous driving exists; the vehicle information includes the vehicle's location information and ID information; When the alarm prompt meets the set conditions, the vehicle information is sent to the cloud control system, and the cloud control system plans the parking route of the current vehicle based on the vehicle information; Switching the vehicle control to the cloud control system, and having the cloud control system control the vehicle to park at a designated location according to the parking route; After safety is verified, the cloud control system will release the vehicle control rights.

2. The dangerous driving control method based on multi-feature fusion according to claim 1 is characterized by: The driver status characteristics are analyzed and judged by collecting the driver's facial image and voice information.

3. The dangerous driving control method based on multi-feature fusion according to claim 2 is characterized by: The driver's facial image is collected and analyzed to determine the driver's status characteristics, as follows: The collected facial image of the driver is input into the second neural network to identify the driver's eyes and mouth, and then the third neural network is used to determine the opening and closing status of the mouth and eyes; A driver is considered awake when their eyes are open and their mouth is closed; A driver is considered drowsy when his eyes and mouth are closed; When the driver's eyes and mouth are both open, the driver is considered to be in an agitated state.

4. The dangerous driving control method based on multi-feature fusion according to claim 2 is characterized by: The driver's status characteristics are analyzed and judged based on the collected voice information of the driver, as follows: The collected driver's voice information is input into the convolutional neural network to extract the spectral features and voiceprint features in the speech signal. Then, the violent speech features are extracted through feature fusion. The classification neural network LSTM is used to identify whether violent speech is present in the voice and judge whether the driver's emotion is angry or normal.

5. The dangerous driving control method based on multi-feature fusion according to any one of claims 3 or 4, characterized in that: The state feature time series of the entire driver's facial image or the state feature time series of the sound information is input into the long-short-term memory model; the cell state of the memory unit in the long-short-term memory model continuously updates and records the state feature time series of the facial image or sound information until it is output to the fusion network after completion, and is used to calculate the driving behavior risk by adding a weight coefficient.

6. The dangerous driving control method based on multi-feature fusion according to claim 1 is characterized by: The vehicle handling characteristics include sudden acceleration, sudden deceleration, steering wheel rotation speed, and vehicle speed.

7. The dangerous driving control method based on multi-feature fusion according to claim 6 is characterized by: Through vehicle-side sensor detection, several vehicle-side data information such as steering wheel rotation angle time series, accelerator pedal force time series, brake pedal force time series, and wheel speed time series are obtained; The vehicle-side data information is first preprocessed and then input into a deep convolutional neural network to extract several vehicle control feature time series, including sudden acceleration, sudden deceleration, steering wheel rotation speed, and vehicle speed. Then, the vehicle control feature time series is input into the long short-term memory model to extract the feature information in the vehicle control feature time series; The extracted feature information is input into the fusion network to calculate the risk level of driving behavior by adding weight coefficients.

8. The dangerous driving control method based on multi-feature fusion according to claim 1 is characterized by: The weight coefficient is calculated based on the driver's previous driving behavior, the vehicle control characteristics corresponding to the driver under different state characteristics are obtained through driving behavior analysis, and the weight coefficient under dangerous driving operation is calculated.

9. The dangerous driving control method based on multi-feature fusion according to claim 1 is characterized by: Determine whether the driver is currently driving dangerously. Specifically, set a number threshold. When the driver's dangerous driving behavior reaches the set number threshold, it is determined that dangerous driving is occurring.

10. The dangerous driving control method based on multi-feature fusion according to claim 1 is characterized by: The alarm prompt meets the set conditions, specifically, the alarm prompt exceeds a preset time threshold, or the alarm prompt exceeds a preset alarm number threshold.

11. The dangerous driving control method based on multi-feature fusion according to claim 1 is characterized by: Before switching the vehicle control right to the cloud control system, the cloud control system first obtains the owner's authorization for the cloud control system to control the vehicle and signs a safe driving agreement.

12. The dangerous driving control method based on multi-feature fusion according to claim 1, characterized in that: While sending vehicle information to the cloud control system, the vehicle information is also broadcast to other surrounding vehicles; the vehicle information includes the vehicle's location information and ID information.

13. The dangerous driving control method based on multi-feature fusion according to claim 1, characterized in that: The vehicle information is sent to a cloud control system, and the cloud control system also locks the vehicle position based on the vehicle information.

14. The dangerous driving control method based on multi-feature fusion according to claim 1, characterized in that: The dangerousness of driving behavior is divided into different levels, and corresponding control and intervention strategies are implemented according to different levels.

15. A dangerous driving control system, characterized by: Including vehicles, base stations, and cloud control systems; Wherein, the vehicle executes the dangerous driving control method based on multi-feature fusion according to any one of claims 1 to 14; The vehicle sends vehicle information to a cloud control system via a base station; the cloud control system plans a parking route for the current vehicle based on the vehicle information, and controls the vehicle to park at a designated location based on the parking route; The system also includes a roadside camera for acquiring a motion trajectory image of the vehicle as it travels on the road, and transmitting the motion trajectory image to the vehicle via a base station; the vehicle extracts information on the vehicle's arbitrary lane changes and braking in the absence of other interfering vehicles based on the motion trajectory image; and extracts information on competitive maneuvers with other vehicles when other vehicles are present, to further determine whether there is any dangerous driving.

16. A dangerous driving control system, characterized by: Including vehicles, base stations, edge cloud computing centers, and cloud control systems; The vehicle collects the driver's facial image and voice information, as well as sudden acceleration, sudden deceleration, steering wheel rotation speed, and vehicle speed information through vehicle-side sensors; The collected information is sent to an edge cloud computing center via a base station through a vehicle-side communication unit in the vehicle, and the edge cloud computing center executes the dangerous driving control method based on multi-feature fusion according to any one of claims 1 to 14; When there is dangerous driving, the edge cloud computing center sends the vehicle information to the cloud control system; the cloud control system plans the parking route of the current vehicle based on the vehicle information, and controls the vehicle to park at a designated location according to the parking route; The system also includes a roadside camera for capturing motion trajectory images of vehicles traveling on the road, and transmitting the motion trajectory images to an edge cloud computing center via a base station; the edge cloud computing center extracts information about the vehicle changing lanes and braking in the absence of other interfering vehicles based on the motion trajectory images; It also extracts information on competitive maneuvers with other vehicles when other vehicles are present to further determine whether there is dangerous driving.

Images