Multi-information fusion vehicle driving state monitoring method and system

Abstract

The invention belongs to the technical field of vehicle driving state monitoring, and relates to a multi-information fusion vehicle driving state monitoring method and system, and the method comprises the following steps: detecting and recording the occupied state of the hands and mouths of a driver based on an image recognition technology; air components in the vehicle are detected and recorded through air detection equipment in the vehicle; the real-time driving state of the vehicle is monitored, the multivariate information is integrated, and the driving state of the vehicle is analyzed according to the driving state analysis model; a driver is warned according to the analysis result, and the vehicle is automatically regulated and controlled based on the response of the driver. The behavior state of the driver and the vehicle driving state can be monitored in real time, potential driving risks can be found in time, driving suggestions are provided for the driver, and the vehicle driving safety is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle driving state monitoring, and in particular relates to a vehicle driving state monitoring method and system with multi-information fusion. Background Art

[0002] A driver's bad driving habits will greatly affect the vehicle's driving condition, and thus affect road traffic safety; smoking while driving is a relatively common bad driving habit. Studies have shown that smoking will distract the driver's attention, affect driving vision, and make it difficult to correctly perceive the situation outside the car.

[0003] Many vehicles are currently equipped with a DMS driver monitoring system, which can monitor in real time whether the driver is present and his current status. DMS can provide alerts to the driver and initiate intervention.

[0004] Traditional DMS driver monitoring systems mainly use target detection algorithms to identify drivers' smoking behavior. Existing target detection methods generally achieve target detection by manually designing features and classifying the extracted features through classifiers. Most current research focuses on improving recognition accuracy and recognition speed, and there are deficiencies in the comprehensive performance of vehicle driving status monitoring.

[0005] To address this problem, the inventor proposes a vehicle driving status monitoring method and system based on multi-information fusion, which comprehensively detects the driver's behavior and analyzes the vehicle driving status through multi-information fusion. Summary of the invention

[0006] The purpose of the present invention is to provide a vehicle driving state monitoring method and system with multi-information fusion to solve the problems raised in the above background technology.

[0007] In order to achieve the purpose of the present invention, the present invention will be implemented by adopting the following technical solutions.

[0008] A vehicle driving state monitoring method based on multi-information fusion comprises the following steps: Step S100: Detecting and recording the driver's hand and mouth occupied state based on image recognition technology; Step S200: Detect and record the air composition in the vehicle through the air detection equipment in the vehicle; Step S300: monitoring the real-time driving state of the vehicle, and analyzing the driving state of the vehicle based on the detection results of steps S100 and S200; Step S400: Based on the vehicle driving status analyzed in step S300, the driver is warned and the vehicle is automatically controlled.

[0009] As a preferred solution of the present invention, the implementation process of step S100 includes the following steps: Step S101: obtaining an image to be detected of the current driver of the vehicle, wherein the image to be detected includes the driver's hands and mouth; performing target detection on the image to be detected to obtain target areas corresponding to the driver's hands and mouth in the image to be detected; Step S102: Determining the hand and mouth occupied state of the current driver of the vehicle based on the target area, including the following steps: Step S102-1: cropping a target area from the image to be detected, normalizing the image data of the target area, inputting the processed image into the target detection model, and marking a detection frame whose subject is a hand as a left hand detection frame, a right hand detection frame, or a hand detection frame, and marking a detection frame whose subject is a mouth as a mouth detection frame according to the detection frame output by the target detection model and the subject information in the detection frame. The marking method of the left hand detection frame, the right hand detection frame or the hand detection frame is as follows: a marking tool is selected to draw a boundary box for the hand area, and when only one of the detection frames output by the model has a subject that is a hand, it is marked as a hand detection frame; when two of the detection frames output by the model have subjects that are hands, with the image to be detected as a reference, the detection frame on the left is marked as a left hand detection frame, and the detection frame on the right is marked as a right hand detection frame; Step S102-2: When the left hand detection frame, the right hand detection frame, or the hand in the hand detection frame holds any object, the hand in the detection frame is considered to be in an occupied state; the arbitrary object includes cigarettes, beverage bottles, mobile phones, and food; Step S102 - 3 : When the mouth in the mouth detection frame contacts any object, it is considered that the mouth in the detection frame is in an occupied state.

[0010] As a preferred solution of the present invention, the implementation process of step S200 includes: The air detection equipment is installed at the instrument panel and the seat headrest of the vehicle respectively; the air detection equipment detects the air composition in the vehicle by connecting to the on-board computer system, and can accurately reflect the air condition in the vehicle; the nicotine concentrations at the two positions in the vehicle are monitored and recorded in real time to generate an air monitoring data set, denoted as G(t), and G(t)={s(t),k(t)|t∈[1,T]}, where s(t) represents the nicotine concentration at the instrument panel detected at the t-th time point, k(t) represents the nicotine concentration at the seat headrest detected at the t-th time point, and T represents the total number of time points monitored and recorded; It should be noted that nicotine exists in the air for a long time. Even after smoking stops for a period of time, nicotine may still remain in the air. At the same time, there may be passengers smoking in the car. Simply detecting nicotine cannot determine the occurrence of smoking behavior. The present invention monitors the nicotine concentration at two locations in the car by arranging air detection equipment at two locations. The driver's behavior state can be comprehensively judged in combination with the driver's image recognition result, thereby improving the accuracy of the judgment result.

[0011] As a preferred solution of the present invention, the implementation process of step S300 includes the following steps: Step S301: Based on the Internet of Vehicles technology, the real-time driving state of the vehicle is monitored, the driving speed, deflection angle and driving route of the vehicle are obtained, and the distance of objects around the vehicle is detected by radar; the relevant data of each vehicle state element monitored in real time are coordinated to generate a vehicle driving state set, which is recorded as D(t), and D(t)={f j (t)|j∈[1,z]}, where f j (t) represents the characteristic data value of the jth vehicle state element currently monitored in real time, and z represents the number of types of vehicle state elements currently monitored in real time; It should be noted that the characteristic data values ​​include the driving speed of the vehicle, the steering wheel deflection angle, the difficulty index of the driving route and the congestion degree of the vehicle's location; Step S302: judging whether the driver is smoking based on the driver's hand and mouth occupied state and the air monitoring data set, including the following steps: Step S302-1: when the nicotine concentration in the air monitoring data set is not zero, extract the hand-mouth occupation state in the time period corresponding to the time when the nicotine concentration in the air monitoring data set is not zero; obtain the nicotine concentration at the instrument panel position and the nicotine concentration at the seat headrest position detected at each time point in the time period; Step S302-2: segmenting the hand-mouth occupied state in the time period in chronological order, and marking the segmented time period as only hand occupied, only mouth occupied, and both hand and mouth occupied; During the time period when only the hands are occupied, when the nicotine concentration at the instrument panel and the nicotine concentration at the seat headrest exceed the preset thresholds, and the nicotine concentration at the instrument panel is greater than the nicotine concentration at the seat headrest, the driver is determined to be in a smoking state; For the time period when only the mouth is occupied, when the nicotine concentration at the instrument panel and the nicotine concentration at the seat headrest exceed the preset threshold, and the nicotine concentration at the instrument panel is less than or equal to the nicotine concentration at the seat headrest, the driver is determined to be in a smoking state; For a time period when the hands and mouth are occupied at the same time, when the nicotine concentration at the instrument panel and the nicotine concentration at the seat headrest exceed the preset threshold, and the nicotine concentration at either position exceeds the average nicotine concentration at the two positions corresponding to the previous time point of the time period, the driver is determined to be in a smoking state; Step S303: Analyze the vehicle driving state set according to the driving state analysis model, and calculate the risk value of the current vehicle driving by the following formula: ; Wherein, DF(T) represents the vehicle driving risk value at the Tth time point; p(T) represents the driver state index at the Tth time point. When it is determined that the driver at the Tth time point is in a smoking state, p(T)=1; when it is determined that the driver at the Tth time point is not in a smoking state, p(T)=0.5; represents the weight of the j-th vehicle state element; H(T) represents the hand occupancy index at the T-th time point. When the detection box output by the target detection model is the hand detection box, H(T)=0.5; when the detection box output by the target detection model is the left hand detection box and the right hand detection box, H(T)=1; λ represents the weight of the hand occupancy index.

[0012] As a preferred solution of the present invention, the implementation process of step S400 includes the following steps: Step S401: Preset a vehicle driving state risk threshold d. According to the vehicle driving state analyzed in step S300, if the current vehicle driving risk value DF(T)≥d, it is determined that the current vehicle driving state has a risk and the driver is warned. If the current vehicle driving risk value DF(T)<d, it is determined that the current vehicle driving state has no risk and monitoring continues. Step S402: If it is determined that there is a risk in the current vehicle driving state, a vehicle driving state set is obtained, and the vehicle is automatically controlled according to each vehicle state element in the vehicle driving state set.

[0013] It should be noted that after the driver is warned, if the driver's behavior status and vehicle driving status have not changed, or the risk value after the change increases, the vehicle will be forcibly controlled through the Internet of Vehicles technology, and the risk value of the vehicle driving status will be reduced by reducing the driving speed, etc.

[0014] A vehicle driving state monitoring system with multi-information fusion, comprising: a monitoring data processing module, a multi-information fusion module, a vehicle driving state analysis module and a vehicle management module; The monitoring data processing module is used to monitor the real-time driving status of the vehicle and detect the occupied state of the driver's hands and mouth and the air composition in the vehicle; The multivariate information fusion module is used to determine whether the driver is in a smoking state according to the output of the monitoring data processing module; The vehicle driving state analysis module is used to analyze the vehicle driving state set according to the driving state analysis model; The vehicle management module is used to manage the vehicle driving state according to the output of the vehicle driving state analysis module, and to give corresponding prompts to the driver through the vehicle's voice settings and visual screen.

[0015] As a feasible embodiment of the present application, when the vehicle driving status exceeds the preset normal range, an alarm is issued in time, and the driver is reminded through voice or screen to open windows for ventilation, reduce speed, pay attention to surroundings, etc.

[0016] Further, the monitoring data processing module includes an image monitoring unit, an air monitoring unit and a vehicle driving monitoring unit; The image monitoring unit is used to obtain the image to be detected of the current driver of the vehicle, perform target detection on the image to be detected, and obtain the target area corresponding to the driver's hands and mouth in the image to be detected; based on the target area, determine the occupied state of the hands and mouth of the current driver of the vehicle; The air monitoring unit is used to install air detection equipment at the instrument panel and seat headrest of the vehicle. The air detection equipment detects the air composition in the vehicle by connecting to the on-board computer system, and can accurately reflect the air condition in the vehicle; monitor and record the nicotine concentration at two locations in the vehicle in real time; The vehicle driving monitoring unit monitors the real-time driving status of the vehicle based on the Internet of Vehicles technology, obtains the vehicle's driving speed, deflection angle and driving route, detects the distance of objects around the vehicle through radar, and coordinates the relevant data of each vehicle status element monitored in real time.

[0017] Furthermore, the vehicle driving state analysis module includes a model building unit and a state determination unit; The model building unit calculates the risk value of the current vehicle driving by the following formula: ; Wherein, DF(T) represents the vehicle driving risk value at the Tth time point; p(T) represents the driver state index at the Tth time point. When the driver at the Tth time point is judged to be in a smoking state, p(T) = 1; when the driver at the Tth time point is judged to be not in a smoking state, p(T) = 0.5; f j (t) represents the characteristic data value of the jth vehicle state element currently monitored in real time, represents the weight of the j-th vehicle state element; H(T) represents the hand occupancy index at the T-th time point. When the detection frame output by the target detection model is the hand detection frame, H(T)=0.5; when the detection frame output by the target detection model is the left hand detection frame and the right hand detection frame, H(T)=1; λ represents the weight of the hand occupancy index; The state determination unit is used to preset a vehicle driving state risk threshold d. If the current vehicle driving risk value DF(T) ≥ d, it is determined that the current vehicle driving state is at risk. If the current vehicle driving risk value DF(T) < d, it is determined that the current vehicle driving state is not at risk and monitoring continues.

[0018] Compared with the prior art, the beneficial effects achieved by the present invention are: The present invention monitors the real-time driving status of the vehicle through a monitoring data processing module, and detects the occupied status of the driver's hands and mouth and the air composition in the vehicle; integrates the output data of the monitoring data processing module through a multi-information fusion module to comprehensively judge whether the driver is in a smoking state; reduces the accuracy requirement for a single hardware, and can accurately judge the driver's behavior status through monitoring information of multiple elements.

[0019] The present invention analyzes the vehicle driving state according to the driving state analysis model through the vehicle driving state analysis module, which can timely discover potential driving risks and avoid driving accidents caused by the driver's behavior; the vehicle management module manages the vehicle driving state according to the output of the vehicle driving state analysis module, and provides personalized warnings and control measures to help the driver identify bad driving habits and improve the driver's driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of a vehicle driving state monitoring method of multi-information fusion according to the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention will be clearly and completely described below in combination with the embodiments of the present invention and the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] See also Figure 1 In this embodiment: a vehicle driving state monitoring method based on multi-information fusion is provided, comprising the following steps: Step S100: Detecting and recording the driver's hand and mouth occupied state based on image recognition technology; Step S101: obtaining an image to be detected of the current driver of the vehicle, wherein the image to be detected includes the driver's hands and mouth; performing target detection on the image to be detected to obtain target areas corresponding to the driver's hands and mouth in the image to be detected; As a feasible embodiment of the present application, the image to be detected of the driver to be detected can be obtained by collecting an image of the person in the driver's seat of the vehicle through a vehicle-mounted camera, obtaining images of the person corresponding to a number of time nodes, and setting the images of the person in sequence according to the time sequence as the images to be detected; Step S102: Determining the hand and mouth occupied state of the current driver of the vehicle based on the target area, including the following steps: Step S102-1: cropping a target area from the image to be detected, normalizing the image data of the target area, inputting the processed image into the target detection model, and marking a detection frame whose subject is a hand as a left hand detection frame, a right hand detection frame, or a hand detection frame, and marking a detection frame whose subject is a mouth as a mouth detection frame according to the detection frame output by the target detection model and the subject information in the detection frame. The marking method of the left hand detection frame, the right hand detection frame or the hand detection frame is as follows: a marking tool is selected to draw a boundary box for the hand area, and when only one of the detection frames output by the model has a subject that is a hand, it is marked as a hand detection frame; when two of the detection frames output by the model have subjects that are hands, with the image to be detected as a reference, the detection frame on the left is marked as a left hand detection frame, and the detection frame on the right is marked as a right hand detection frame; Step S102-2: When the left hand detection frame, the right hand detection frame, or the hand in the hand detection frame holds any object, the hand in the detection frame is considered to be in an occupied state; the arbitrary object includes cigarettes, beverage bottles, mobile phones, and food; Step S102 - 3 : When the mouth in the mouth detection frame contacts any object, it is considered that the mouth in the detection frame is in an occupied state.

[0023] It should be noted that according to the above method, there is no need to strictly require the accuracy of the target detection algorithm, and the amount of training required for recognition is also relatively low. Since the present invention determines the driver's behavior status through comprehensive information, there is no need to invest more time and resource costs in image acquisition and processing.

[0024] Step S200: Detect and record the air composition in the vehicle through the air detection equipment in the vehicle; The air detection equipment is installed at the instrument panel and the seat headrest of the vehicle respectively; the air detection equipment detects the air composition in the vehicle by connecting to the on-board computer system, and can accurately reflect the air condition in the vehicle; the nicotine concentrations at the two positions in the vehicle are monitored and recorded in real time to generate an air monitoring data set, denoted as G(t), and G(t)={s(t),k(t)|t∈[1,T]}, where s(t) represents the nicotine concentration at the instrument panel detected at the t-th time point, k(t) represents the nicotine concentration at the seat headrest detected at the t-th time point, and T represents the total number of time points monitored and recorded; As a feasible embodiment of the present application, the air detection device may be an electrochemical sensor or a MOS sensor.

[0025] It should be noted that nicotine exists in the air for a long time. Even after smoking stops for a period of time, nicotine may still remain in the air. At the same time, there may be passengers smoking in the car. Simply detecting nicotine cannot determine the occurrence of smoking behavior. The present invention monitors the nicotine concentration at two locations in the car by arranging air detection equipment at two locations. The driver's behavior state can be comprehensively judged in combination with the driver's image recognition result, thereby improving the accuracy of the judgment result.

[0026] Step S300: monitoring the real-time driving state of the vehicle, and analyzing the driving state of the vehicle based on the detection results of steps S100 and S200; Step S301: Based on the Internet of Vehicles technology, the real-time driving state of the vehicle is monitored, the driving speed, deflection angle and driving route of the vehicle are obtained, and the distance of objects around the vehicle is detected by radar; the relevant data of each vehicle state element monitored in real time are coordinated to generate a vehicle driving state set, which is recorded as D(t), and D(t)={f j (t)|j∈[1,4]}, where f j (t) represents the characteristic data value of the jth vehicle state element currently monitored in real time; It should be noted that the characteristic data values ​​include the driving speed of the vehicle, the steering wheel deflection angle, the difficulty index of the driving route and the congestion degree of the vehicle's location; As a feasible embodiment of the present application, the difficulty index of the driving route may be: ; in, Indicates the difficulty index of the driving route corresponding to the t-th time point, represents the driving route category corresponding to the t-th time point, The difficulty coefficient of the driving route category corresponding to the t-th time point is preset in the database by relevant personnel; As a feasible embodiment of the present application, the congestion degree of the location where the vehicle is located may be: ; in, represents the congestion degree of the vehicle at the t-th time point, represents the distance between the vth object and the vehicle at the tth time point. The crowding coefficient corresponding to the distance between the vth object and the vehicle around the vehicle is preset in the database by relevant personnel; Step S302: judging whether the driver is smoking based on the driver's hand and mouth occupied state and the air monitoring data set, including the following steps: Step S302-1: when the nicotine concentration in the air monitoring data set is not zero, extract the hand-mouth occupation state in the time period corresponding to the time when the nicotine concentration in the air monitoring data set is not zero; obtain the nicotine concentration at the instrument panel position and the nicotine concentration at the seat headrest position detected at each time point in the time period; Step S302-2: segmenting the hand-mouth occupied state in the time period in chronological order, and marking the segmented time period as only hand occupied, only mouth occupied, and both hand and mouth occupied; During the time period when only the hands are occupied, when the nicotine concentration at the instrument panel and the nicotine concentration at the seat headrest exceed the preset thresholds, and the nicotine concentration at the instrument panel is greater than the nicotine concentration at the seat headrest, the driver is determined to be in a smoking state; For the time period when only the mouth is occupied, when the nicotine concentration at the instrument panel and the nicotine concentration at the seat headrest exceed the preset threshold, and the nicotine concentration at the instrument panel is less than or equal to the nicotine concentration at the seat headrest, the driver is determined to be in a smoking state; For a time period when the hands and mouth are occupied at the same time, when the nicotine concentration at the instrument panel and the nicotine concentration at the seat headrest exceed the preset threshold, and the nicotine concentration at either position exceeds the average nicotine concentration at the two positions corresponding to the previous time point of the time period, the driver is determined to be in a smoking state; Step S303: Analyze the vehicle driving state set according to the driving state analysis model, and calculate the risk value of the current vehicle driving by the following formula: ; Wherein, DF(T) represents the vehicle driving risk value at the Tth time point; p(T) represents the driver state index at the Tth time point. When it is determined that the driver at the Tth time point is in a smoking state, p(T)=1; when it is determined that the driver at the Tth time point is not in a smoking state, p(T)=0.5; represents the weight of the j-th vehicle state element; H(T) represents the hand occupancy index at the T-th time point. When the detection box output by the target detection model is the hand detection box, H(T)=0.5; when the detection box output by the target detection model is the left hand detection box and the right hand detection box, H(T)=1; λ represents the weight of the hand occupancy index.

[0027] Step S400: According to the vehicle driving state analyzed in step S300, the driver is warned and the vehicle is automatically controlled; Step S401: Preset a vehicle driving state risk threshold d. According to the vehicle driving state analyzed in step S300, if the current vehicle driving risk value DF(T)≥d, it is determined that the current vehicle driving state has a risk and the driver is warned. If the current vehicle driving risk value DF(T)<d, it is determined that the current vehicle driving state has no risk and monitoring continues. As a feasible embodiment of the present application, different levels of warnings are set according to different risk values, and sound warnings can be given by playing voice, or by indicator lights and vibration warnings; Step S402: If it is determined that there is a risk in the current vehicle driving state, a vehicle driving state set is obtained, and the vehicle is automatically controlled according to each vehicle state element in the vehicle driving state set.

[0028] In one embodiment, after the driver is warned, if the driver's behavior state and the vehicle driving state have not changed, or the risk value after the change increases, the vehicle is forcibly controlled through the Internet of Vehicles technology, and the risk value of the vehicle driving state is reduced by reducing the driving speed, etc.

[0029] The technical solution of the present invention is described in detail above in conjunction with the embodiments / drawings, but the present invention is not limited to the above technical solution. For ordinary technicians in this technical field, after knowing the contents recorded in the present invention, they can make several equivalent changes and substitutions without departing from the principle of the present invention. These equivalent changes and substitutions should also be regarded as belonging to the protection scope of the present invention.

Claims

1. A vehicle driving state monitoring method based on multi-information fusion, characterized in that: The following steps are involved: Step S100: Detecting and recording the driver's hand and mouth occupied state based on image recognition technology; Step S200: Detect and record the air composition in the vehicle through the air detection equipment in the vehicle; Step S300: monitoring the real-time driving state of the vehicle, and analyzing the driving state of the vehicle based on the detection results of steps S100 and S200; Step S400: Based on the vehicle driving status analyzed in step S300, the driver is warned and the vehicle is automatically controlled.

2. The vehicle driving state monitoring method based on multi-information fusion according to claim 1 is characterized in that: The implementation process of step S100 includes the following steps: Step S101: obtaining an image to be detected of the current driver of the vehicle, wherein the image to be detected includes the driver's hands and mouth; performing target detection on the image to be detected to obtain target areas corresponding to the driver's hands and mouth in the image to be detected; Step S102: Determining the hand and mouth occupied state of the current driver of the vehicle based on the target area, including the following steps: Step S102-1: cropping a target area from the image to be detected, normalizing the image data of the target area, inputting the processed image into the target detection model, and marking a detection frame whose subject is a hand as a left hand detection frame, a right hand detection frame, or a hand detection frame, and marking a detection frame whose subject is a mouth as a mouth detection frame according to the detection frame output by the target detection model and the subject information in the detection frame. Step S102-2: when the left hand detection frame, the right hand detection frame, or the hand in the hand detection frame holds any object, the hand in the detection frame is considered to be in an occupied state; Step S102 - 3 : When the mouth in the mouth detection frame contacts any object, it is considered that the mouth in the detection frame is in an occupied state.

3. The vehicle driving state monitoring method based on multi-information fusion according to claim 1 is characterized in that: The air detection equipment is installed at the instrument panel and seat headrest of the vehicle respectively; the air detection equipment detects the air composition in the vehicle by connecting to the on-board computer system, and can accurately reflect the air condition in the vehicle; the nicotine concentrations at the two positions in the vehicle are monitored and recorded in real time to generate an air monitoring data set, denoted as G(t), and G(t)={s(t),k(t)|t∈[1,T]}, wherein s(t) represents the nicotine concentration at the instrument panel detected at the t-th time point, k(t) represents the nicotine concentration at the seat headrest detected at the t-th time point, and T represents the total number of time points monitored and recorded.

4. The vehicle driving state monitoring method based on multi-information fusion according to claim 1 is characterized in that: The implementation process of step S300 includes the following steps: Step S301: Based on the Internet of Vehicles technology, the real-time driving state of the vehicle is monitored, and the distance of objects around the vehicle is detected by radar; the relevant data of each vehicle state element monitored in real time are coordinated to generate a vehicle driving state set, which is recorded as D(t), and D(t)={f j (t)|j∈[1,z]}, where f j (t) represents the characteristic data value of the jth vehicle state element currently monitored in real time, and z represents the number of types of vehicle state elements currently monitored in real time; Step S302: judging whether the driver is smoking based on the driver's hand and mouth occupied state and the air monitoring data set, including the following steps: Step S302-1: when the nicotine concentration in the air monitoring data set is not zero, extract the hand-mouth occupation state in the time period corresponding to the time when the nicotine concentration in the air monitoring data set is not zero; obtain the nicotine concentration at the instrument panel position and the nicotine concentration at the seat headrest position detected at each time point in the time period; Step S302-2: segmenting the hand-mouth occupied state in the time period in chronological order, and marking the segmented time period as only hand occupied, only mouth occupied, and both hand and mouth occupied; During the time period when only the hands are occupied, when the nicotine concentration at the instrument panel and the nicotine concentration at the seat headrest exceed the preset thresholds, and the nicotine concentration at the instrument panel is greater than the nicotine concentration at the seat headrest, the driver is determined to be in a smoking state; For the time period when only the mouth is occupied, when the nicotine concentration at the instrument panel and the nicotine concentration at the seat headrest exceed the preset threshold, and the nicotine concentration at the instrument panel is less than or equal to the nicotine concentration at the seat headrest, the driver is determined to be in a smoking state; For a time period when the hands and mouth are occupied at the same time, when the nicotine concentration at the instrument panel and the nicotine concentration at the seat headrest exceed the preset threshold, and the nicotine concentration at either position exceeds the average nicotine concentration at the two positions corresponding to the previous time point of the time period, the driver is determined to be in a smoking state; Step S303: Analyze the vehicle driving state set according to the driving state analysis model, and calculate the risk value of the current vehicle driving by the following formula: ； Wherein, DF(T) represents the vehicle driving risk value at the Tth time point; p(T) represents the driver state index at the Tth time point. When it is determined that the driver at the Tth time point is in a smoking state, p(T)=1; when it is determined that the driver at the Tth time point is not in a smoking state, p(T)=0.5; represents the weight of the j-th vehicle state element; H(T) represents the hand occupancy index at the T-th time point. When the detection box output by the target detection model is the hand detection box, H(T)=0.5; when the detection box output by the target detection model is the left hand detection box and the right hand detection box, H(T)=1; λ represents the weight of the hand occupancy index.

5. The vehicle driving state monitoring method based on multi-information fusion according to claim 4 is characterized in that: The implementation process of step S400 includes the following steps: Step S401: Preset a vehicle driving state risk threshold d. According to the vehicle driving state analyzed in step S300, if the current vehicle driving risk value DF(T)≥d, it is determined that the current vehicle driving state has a risk and the driver is warned. If the current vehicle driving risk value DF(T)<d, it is determined that the current vehicle driving state has no risk and monitoring continues. Step S402: If it is determined that there is a risk in the current vehicle driving state, a vehicle driving state set is obtained, and the vehicle is automatically controlled according to each vehicle state element in the vehicle driving state set.

6. A vehicle driving state monitoring system using multi-information fusion, which is implemented by a vehicle driving state monitoring method using multi-information fusion according to any one of claims 1 to 5, characterized in that: include: Monitoring data processing module, multi-information fusion module, vehicle driving status analysis module and vehicle management module; The monitoring data processing module is used to monitor the real-time driving status of the vehicle and detect the occupied state of the driver's hands and mouth and the air composition in the vehicle; The multivariate information fusion module is used to determine whether the driver is in a smoking state according to the output of the monitoring data processing module; The vehicle driving state analysis module is used to analyze the vehicle driving state according to the driving state analysis model; The vehicle management module is used to manage the vehicle driving state according to the output of the vehicle driving state analysis module, and to give corresponding prompts to the driver through the vehicle's voice settings and visual screen.

7. The vehicle driving state monitoring system of multi-information fusion according to claim 6 is characterized by: The monitoring data processing module includes an image monitoring unit, an air monitoring unit and a vehicle driving monitoring unit; The image monitoring unit is used to obtain the image to be detected of the current driver of the vehicle, perform target detection on the image to be detected, and obtain the target area corresponding to the driver's hands and mouth in the image to be detected; Determine the hand and mouth occupation status of the current driver of the vehicle based on the target area; The air monitoring unit is used to install air detection equipment at the instrument panel and seat headrest of the vehicle. The air detection equipment detects the air composition in the vehicle by connecting to the on-board computer system, and can accurately reflect the air condition in the vehicle; monitor and record the nicotine concentration at two locations in the vehicle in real time; The vehicle driving monitoring unit monitors the real-time driving status of the vehicle based on the Internet of Vehicles technology, obtains the vehicle's driving speed, deflection angle and driving route, detects the distance of objects around the vehicle through radar, and coordinates the relevant data of each vehicle status element monitored in real time.

8. The vehicle driving state monitoring system of multi-information fusion according to claim 6 is characterized by: The vehicle driving state analysis module includes a model building unit and a state determination unit; The model building unit calculates the risk value of the current vehicle driving by the following formula: ； Wherein, DF(T) represents the vehicle driving risk value at the Tth time point; p(T) represents the driver state index at the Tth time point. When the driver at the Tth time point is judged to be in a smoking state, p(T) = 1; when the driver at the Tth time point is judged to be not in a smoking state, p(T) = 0.5; f j (t) represents the characteristic data value of the jth vehicle state element currently monitored in real time, represents the weight of the j-th vehicle state element; H(T) represents the hand occupancy index at the T-th time point. When the detection frame output by the target detection model is the hand detection frame, H(T)=0.5; when the detection frame output by the target detection model is the left hand detection frame and the right hand detection frame, H(T)=1; λ represents the weight of the hand occupancy index; The state determination unit is used to preset a vehicle driving state risk threshold d. If the current vehicle driving risk value DF(T) ≥ d, it is determined that the current vehicle driving state is at risk. If the current vehicle driving risk value DF(T) < d, it is determined that the current vehicle driving state is not at risk and monitoring continues.

Images