A driver analysis state comprehensive evaluation method based on multi-source information fusion

Abstract

The application relates to the field of early warning evaluation, and discloses a driver analysis state comprehensive evaluation method based on multi-source information fusion; the method comprises the following steps: S01, identifying a first evaluation interval, a second evaluation interval and an analysis early warning interval; S02, extracting a vehicle speed feature; S03, identifying a face feature; S04, evaluating a driving state of a driver; and S05, driving a danger alarm unit to perform a sound-light combined alarm operation; the application can provide multi-source information from two parallel dimensions of a vehicle speed and a driver face to evaluate whether the driver is in a fatigue driving state, avoids the limitation of single-dimension evaluation information in evaluating the fatigue state of the driver, can dynamically and orderly screen sufficient face images from massive frame images, avoids the huge burden brought by the massive frame images in subsequent feature recognition models for face feature recognition, and further improves the analysis and evaluation accuracy of the real driving state of the driver.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of early warning evaluation, more particularly, to a driver analysis state comprehensive evaluation method based on multi-source information fusion. BACKGROUND

[0002] With the gradual increase of highway construction projects, construction safety management is also facing great challenges. By analyzing and evaluating the fatigue driving state of the vehicle driver passing through the construction area and giving an alarm, the safety level of the construction area can be effectively improved, and the incidence of safety accidents can be reduced, thereby protecting the life safety of construction personnel. Therefore, it is necessary to combine artificial intelligence technology and Internet of Things technology to analyze and evaluate the real-time driving state of the driver.

[0003] The patent application with publication number CN118486130A discloses a hierarchical monitoring and early warning method based on driver suitability, including the following steps: S1: during vehicle driving, determine whether the vehicle driving state is abnormal, and generate vehicle driving abnormal signal and vehicle driving normal signal respectively; S2: determine the interaction degree of the driver after receiving the abnormal signal and the vehicle operation; S3: determine the frequency of the driver in the historical driving behavior data, and evaluate the effectiveness of the driving behavior data; S4: evaluate the accuracy of the monitoring result of the monitoring system; S5: divide the monitoring result of the monitoring system into accuracy monitoring result, possible accuracy monitoring result, and inaccuracy monitoring result; S6: further analyze and predict the accuracy of the monitoring result of the monitoring system.

[0004] The existing driver state evaluation method usually relies on single-dimensional information, which cannot analyze and evaluate the information after fusion of multiple different dimensions, so that the information dimension involved in the analysis and evaluation is single and has limitations. At the same time, after collecting a large amount of video frame images, the model directly identifies the features of the large amount and high density of frame images, which brings a huge burden to the identification and calculation operation of the model, reduces the rate of feature identification of the model, and thus leads to low accuracy of the real driving state analysis and evaluation result of the driver in the evaluation interval.

[0005] In view of this, the present application provides a driver analysis state comprehensive evaluation method based on multi-source information fusion to solve the above problems. SUMMARY

[0006] In order to overcome the above-mentioned defects of the prior art, in order to achieve the above-mentioned purpose, the present application provides the following technical scheme: a driver analysis state comprehensive evaluation method based on multi-source information fusion, applied to a control terminal, comprising:

[0007] S01: record the point where the target vehicle passes for the first time in the construction area as an evaluation point, take the evaluation point as a reference point, identify a first evaluation interval and a second evaluation interval for collecting multi-source information, and an analysis and early warning interval for performing sound and light combined alarm;

[0008] S02: build a vehicle speed monitoring unit in the first evaluation interval, collect vehicle speed data of the target vehicle in the first evaluation interval through the vehicle speed monitoring unit, and extract vehicle speed features from the vehicle speed data;

[0009] S03: build a face monitoring unit in the second evaluation interval, collect the face image of the driver in the second evaluation interval based on the dynamic interval criterion through the face monitoring unit, and import the face image into the feature recognition model to identify the face features of the driver;

[0010] S04: analyze and evaluate the vehicle speed features and the face features, evaluate the driving state of the driver at the local position in the first evaluation interval and the second evaluation interval, and determine whether to issue a danger warning information;

[0011] S05: build a danger alarm unit in the analysis and early warning interval, control the terminal to drive the danger alarm unit to perform sound and light combined alarm operation based on the danger warning information.

[0012] Further, the identification method of the multi-source evaluation interval is:

[0013] In the construction lane, mark a point at a position in the direction of the incoming vehicle and a calibrated safety distance from the reference point, which is recorded as the interval endpoint;

[0014] Mark the points at positions in the direction of the incoming vehicle and a preset first monitoring length and a preset second monitoring length from the interval endpoint, respectively, which are recorded as the first starting point and the second starting point;

[0015] Record the area between the first starting point and the interval endpoint as the first evaluation interval, and record the area between the second starting point and the interval endpoint as the second evaluation interval.

[0016] Further, the vehicle speed monitoring unit includes a quick-release truss and a radar sensor; the vehicle speed features include a local attenuation value and a terminal speed value.

[0017] Further, the extraction method of the local attenuation value is:

[0018] A1: take the first starting point as the entry point, collect the vehicle speed of the target vehicle passing through the entry point through the radar sensor, and record it as the entry speed value;

[0019] A2: convert the entry speed value into vehicle second speed, and generate a monitoring distance value after expanding the vehicle second speed by a corresponding second speed multiple;

[0020] A3: After the entry point, mark a monitoring point in the driving direction of the vehicle and at a distance of one monitoring distance value from the entry point;

[0021] A4: Take the monitoring point as the entry point, and repeat the steps of A1-A3 until the marked monitoring point is no longer in the first evaluation interval, and then stop. After collecting the entry points and all monitoring points, B+1 monitoring points are obtained;

[0022] A5: Arrange the entry speed values of the B+1 monitoring points in order, and obtain B local attenuation values by subtracting the entry speed value of the previous monitoring point from the entry speed value of the next monitoring point.

[0023] Further, the face detection unit comprises a quick-release truss, a zoom camera and a humidity sensor; the dynamic interval criterion is that the time span between adjacent two face images is inversely proportional to the dynamic adjustment coefficient.

[0024] Further, the face image acquisition method is:

[0025] B1: Acquire the video file of the target vehicle in the second evaluation interval through the zoom camera, and generate C continuous frame images by video stream decompression of the video file;

[0026] B2: Detect the vehicle speed at the corresponding time of the C frame images one by one through the radar sensor, and obtain C interval speed values;

[0027] B3: Record the time period covered by the C frame images as the interval period, and count the number of motor vehicles entering the second evaluation interval in the interval period, which is recorded as the interval vehicle value;

[0028] B4: Acquire the maximum environmental humidity in the interval period in the second evaluation interval through the humidity sensor, which is recorded as the interval humidity value;

[0029] B5: After eliminating the dimensions of the C interval speed values, interval vehicle values and interval humidity values respectively, C speed values, vehicle values and humidity values are obtained, and the C speed values, vehicle values and humidity values are added after being respectively assigned different weight coefficients;

[0030] B6: Compare the dynamic adjustment coefficients of the C frame images with the standard time length respectively, and calculate C interval time lengths;

[0031] B7: Take the time of the frame image located in the first position as the starting time, and from the starting time, record the time corresponding to the interval time length after the starting time as the target time, and record the frame image corresponding to the target time as the target image;

[0032] B8: taking the target time as a starting time, repeatedly performing the steps of B7 until the target time marked is no longer located in the interval period, and stopping, and after all target images and the frame image in the first position are collected, obtaining D face images.

[0033] Further, the facial features include an eyelid opening value and a gaze downward value;

[0034] The D face images collected are imported into a feature recognition model to recognize D eyelid opening values and D gaze downward values corresponding to the D face images.

[0035] Further, the driving state includes a fatigue state and a safety state;

[0036] The determination method of the driving state of the first evaluation interval is:

[0037] In the order of time, the positive and negative relationships of the B local attenuation values are recognized in sequence;

[0038] When the local attenuation value is negative or 0, the driving state of the first evaluation interval is determined as the fatigue state;

[0039] When the local attenuation value is positive, the size relationship between the terminal speed value and the calibrated safety speed value is compared;

[0040] If the terminal speed value is greater than or equal to the calibrated safety speed value, the driving state of the first evaluation interval is determined as the fatigue state;

[0041] If the terminal speed value is less than the calibrated safety speed value, the driving state of the first evaluation interval is determined as the safety state.

[0042] Further, the determination method of the driving state of the second evaluation interval is:

[0043] The interval period is divided into E sub-periods according to a preset evaluation length, and after the face images in which the target time is in the same sub-period are collected, E image sets are obtained;

[0044] The eye opening value less than the calibrated opening value is recorded as an abnormal opening value, and the number of abnormal opening values in the E image sets is respectively counted, and after the number of abnormal opening values in the E image sets is compared with the number of eye opening values, E closing proportion values are obtained;

[0045] The gaze downward value greater than the calibrated downward value is recorded as an abnormal downward value, and the number of abnormal downward values in the E image sets is respectively counted, and after the number of abnormal downward values in the E image sets is compared with the number of gaze downward values, E downward proportion values are obtained;

[0046] When the closing proportion value is greater than or equal to a preset first proportion value, or the downward moving proportion value is greater than or equal to a preset second proportion value, the driving state of the second evaluation interval is determined as a fatigue state;

[0047] When the closing proportion value is less than the preset first proportion value, and the downward moving proportion value is less than the preset second proportion value, the driving state of the second evaluation interval is determined as a safe state;

[0048] The determination method of whether to issue a danger alarm information is:

[0049] When the fatigue state occurs, it is determined to issue the danger alarm information; and when the fatigue state does not occur, it is determined not to issue the danger alarm information.

[0050] Further, the danger alarm unit comprises a sound warning device and a light intensity warning device;

[0051] When the sound and light combination alarm operation is performed, the control terminal sends alarm information to the sound warning device and the light intensity warning device, the sound warning device continuously broadcasts warning voice outward, and the light intensity warning device continuously flashes warning light outward.

[0052] The technical effect of the driver analysis state comprehensive evaluation method based on multi-source information fusion is:

[0053] (1) The driver analysis state comprehensive evaluation method based on multi-source information fusion can provide multi-source information from the vehicle speed and the driver's face in two parallel dimensions when evaluating whether the driver is in a fatigue driving state, avoids the limitations of single-dimensional evaluation information in evaluating the fatigue state of the driver, and also utilizes the technical means of independent collection and fusion evaluation of the vehicle speed feature and the face feature. On the basis of ensuring the independent and orderly collection of different dimensional features, the effect of multi-source information fusion evaluation of the driving state is realized, and the accuracy of the evaluation result of the driving state of the driver is effectively improved.

[0054] (2) The driver analysis state comprehensive evaluation method based on multi-source information fusion can dynamically and orderly screen sufficient face images from a large number of frame images by converting the video file into frame images and formulating the dynamic interval length of the face image under the limitation of the dynamic interval criterion, so that the face image can dynamically represent the face of the driver in the second evaluation interval throughout the whole process, thereby avoiding the huge burden brought by a large number of frame images in the subsequent feature recognition model for face feature recognition, speeding up the recognition speed of the feature recognition model for face feature, and further improving the analysis and evaluation accuracy of the real driving state of the driver. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 A flowchart of a driver analysis state comprehensive evaluation method based on multi-source information fusion is provided for the first embodiment of the present application.

[0056] Figure 2 A structural schematic diagram of the first evaluation interval, the second evaluation interval and the analysis early warning interval provided for the first embodiment of the present application;

[0057] Figure 3 A module schematic diagram of a driver analysis state comprehensive evaluation system based on multi-source information fusion provided for the second embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0059] Embodiment one: please refer to Figures 1-2 As shown in the figure, the driver analysis state comprehensive evaluation method based on multi-source information fusion described in the embodiment is applied to a control terminal, and includes:

[0060] S01: Mark an evaluation point position of a construction area in a construction lane, take the evaluation point position as a reference point, identify a multi-source evaluation interval for collecting multi-source information, the multi-source evaluation interval includes a first evaluation interval and a second evaluation interval, and an analysis early warning interval for performing sound-light combined alarm;

[0061] The construction lane refers to the position of a specific lane where the construction area of the expressway is located. The construction lane can be one lane or multiple lanes.

[0062] The evaluation point position refers to a reference point position of the construction area for collecting relevant data when a driver analyzes and evaluates a real-time driving state on the expressway, and serves as a position basis for subsequent judgment of whether the driver has fatigue driving.

[0063] In the embodiment, in order to find the position of the construction area as early as possible and improve the personal safety of the construction personnel and the driver in the construction area, the point position where the vehicle passes for the first time in the construction area is recorded as the evaluation point position. The evaluation point position at this time is the point position closest to the direction of the incoming vehicle.

[0064] The multi-source evaluation interval refers to an interval for collecting multi-source information for evaluating whether the driver has a drowsy driving phenomenon or the like during driving the vehicle, that is, to provide independent and accurate collection position limitation for information of different sources at different positions of the driver.

[0065] In the embodiment, the multi-source information is used to represent the relevant data information of the vehicle speed dimension and the driver face dimension.

[0066] The source evaluation interval includes a first evaluation interval and a second evaluation interval; the first evaluation interval is used to collect information related to the vehicle speed dimension, and the second evaluation interval is used to collect information related to the driver's face dimension.

[0067] The identification method of the multi-source evaluation interval is:

[0068] In the construction lane, a point is marked at a position in the direction of the oncoming vehicle and a calibrated safe distance from the reference point, which is the interval endpoint; the calibrated safe distance refers to the minimum safe distance that ensures the safe state between the vehicle and the construction area, that is, sufficient buffer distance can be left in front of the construction area to ensure that the vehicle has enough time and position to perform steering and other operations in the area; Specifically, the calibrated safe distance is formulated according to the actual highway construction safety specification, for example, the calibrated safe distance is 300 meters;

[0069] Points are marked at positions in the direction of the oncoming vehicle and a preset first monitoring length and a preset second monitoring length from the interval endpoint, respectively, which are the first starting point and the second starting point;

[0070] The area between the first starting point and the interval endpoint in the construction lane is recorded as the first evaluation interval, and the area between the second starting point and the interval endpoint is recorded as the second evaluation interval.

[0071] It should be noted that the preset first monitoring length and the preset second monitoring length are representations of the interval length range required for collecting vehicle speed dimension related information and driver face related information, respectively, to ensure that the first evaluation interval and the second evaluation interval can accurately and comprehensively collect vehicle speed dimension related information and driver face related information; Specifically, the preset first monitoring length is less than the preset second monitoring length; for example, the preset first monitoring length is 600 meters, and the preset second monitoring length is 2000 meters;

[0072] The analysis warning interval refers to the interval position corresponding to the dangerous alarm operation when the real-time driving state of the vehicle driver is analyzed and evaluated in the presence of dangerous situations, which can provide position limitation for the installation and deployment of subsequent dangerous alarm devices;

[0073] In this embodiment, the analysis warning interval is used to install and deploy dangerous alarm devices, and the dangerous alarm devices provide warning to vehicle drivers by issuing sound and light combination alarms, therefore, the range of the analysis warning interval needs to cover the first evaluation interval and the second evaluation interval, and needs to extend to the location of the construction interval;

[0074] Specifically, the analysis early warning interval is a guardrail region parallel to the region between the second starting point in the construction lane and the evaluation point.

[0075] As shown in Figure 2 By the above identification method, the first evaluation interval, the second evaluation interval and the analysis early warning interval can be accurately identified in the construction lane.

[0076] S02: A vehicle speed monitoring unit is built in the first evaluation interval, vehicle speed data of the target vehicle in the first evaluation interval is collected by the vehicle speed monitoring unit, and vehicle speed features are extracted from the vehicle speed data;

[0077] The target vehicle refers to the vehicle corresponding to the driver state analysis and evaluation when entering the multi-source evaluation interval, and the vehicle speed monitoring unit refers to the overall structure of the related equipment installed on the highway to collect vehicle speed data of the target vehicle in the first evaluation interval;

[0078] Specifically, the vehicle speed monitoring unit includes a quick-release truss and a radar sensor; wherein the quick-release truss is a mounting bracket for installing and fixing the radar sensor, the quick-release truss is built at the position of the interval endpoint, the radar sensor is fixed on the quick-release truss, the ranging range of the radar sensor is adjusted to face the construction lane, and the radar sensor completely covers the first evaluation interval.

[0079] The vehicle speed data refers to the vehicle speed data of the target vehicle at different times in the first evaluation interval, which can comprehensively represent the vehicle speed change of the target vehicle in the first evaluation interval;

[0080] Specifically, the vehicle speed data includes but is not limited to position data, time data, distance data, etc.; by comprehensive collection of vehicle speed data, real-time position, vehicle speed time, distance between vehicle and construction area, etc. of the target vehicle in the first evaluation interval can be represented, in this embodiment, the vehicle speed data is collected by the radar sensor.

[0081] After collecting the vehicle speed data, the vehicle speed data needs to be analyzed, and the vehicle speed features that can represent the vehicle speed of the target vehicle at different times in the first evaluation interval are extracted from the vehicle speed data, which are directly used as the basis for the change of the driving speed of the target vehicle in the first evaluation interval;

[0082] Specifically, the vehicle speed features include local attenuation value and terminal speed value.

[0083] The local attenuation value refers to the amplitude of the vehicle speed attenuation and reduction of the target vehicle after passing through a certain distance in the first evaluation interval, which can represent the reduction and change of the vehicle speed of the target vehicle in the first evaluation interval, thereby indirectly indicating the real-time driving condition of the driver;

[0084] The extraction method of the local attenuation value is:

[0085] A1: taking the first starting point as the entry point, collecting the vehicle speed of the target vehicle passing through the entry point by the radar sensor, and recording the entry speed value;

[0086] A2: converting the entry speed value into vehicle second speed, and generating a monitoring distance value after expanding the vehicle second speed by a corresponding second speed multiple; the second speed multiple refers to the corresponding time for expanding the vehicle second speed, and serves as the maximum time span for monitoring the real-time speed of the target vehicle; the second speed multiple is not unique, and is adjusted according to the size of the vehicle second speed; for example, when the vehicle second speed is 25-30 meters per second, the second speed multiple is 2.5 seconds;

[0087] A3: marking a monitoring point after the entry point along the driving direction of the vehicle and at a position with a monitoring distance value from the entry point;

[0088] A4: taking the monitoring point as the entry point, and repeating the steps of A1-A3 until the monitoring point is no longer located in the first evaluation interval; after collecting the entry points and all monitoring points, B+1 monitoring points are obtained;

[0089] A5: arranging the entry speed values of the B+1 monitoring points in sequence, and obtaining B local attenuation values by subtracting the entry speed value of the previous monitoring point from the entry speed value of the next monitoring point.

[0090] The terminal speed value refers to the real-time speed of the target vehicle when it exits the first evaluation interval, which directly represents the final speed of the target vehicle in the first evaluation interval, and indirectly indicates the real-time driving state of the driver;

[0091] In this embodiment, the terminal speed value is obtained by collecting the vehicle speed of the target vehicle passing through the interval endpoint by the radar sensor.

[0092] It should be noted that the collected local attenuation value and terminal speed value can be directly used as the data basis for judging whether the driver has fatigue driving in the first evaluation interval, and can also be analyzed comprehensively with the relevant data collected in the subsequent second evaluation interval.

[0093] S03: building a face monitoring unit in the second evaluation interval, collecting the face image of the driver in the second evaluation interval by the face monitoring unit based on the dynamic interval criterion, and inputting the face image into the feature recognition model to recognize the face features of the driver;

[0094] The face detection unit refers to the overall structure of the related equipment installed on the expressway for collecting the face data of the driver in the second evaluation interval.

[0095] The face detection unit comprises a quick-release truss, a zoom camera and a humidity sensor; wherein the zoom camera and the humidity sensor are both fixed on the quick-release truss, the shooting angle of the zoom camera is adjusted to face the construction lane, and the zoom camera completely covers the second evaluation interval.

[0096] After the face detection unit is built, the face detection unit can be used to capture the face features of the driver of the target vehicle in the second evaluation interval in real time, and the face features are analyzed and compared to determine whether the driver of the target vehicle has a fatigue driving phenomenon in the second evaluation interval.

[0097] The face image is a variation image of the facial expression features of the driver of the target vehicle at different times in the second evaluation interval, which can comprehensively represent the face of the driver in the second evaluation interval.

[0098] Since the face image is captured in real time by the zoom camera, the zoom camera directly collects the video file of the driver's face at this time. In order to obtain a face image that can be directly used for identification, the face image that can represent the driver's facial expression at different frame times needs to be extracted from the video file.

[0099] When extracting the face image from the video file, it needs to be done under the restriction of the dynamic interval criterion to ensure that the number of extracted face images meets the identification requirements, and the time corresponding to the face image remains in a dynamic dispersed state.

[0100] Specifically, the dynamic interval criterion is that the time span between two adjacent face images is inversely proportional to the dynamic adjustment coefficient.

[0101] The face image acquisition method is:

[0102] B1: Collect the video file of the target vehicle in the second evaluation interval by the zoom camera, and decompress the video stream to generate C continuous frame images;

[0103] B2: Detect the vehicle speed at the time corresponding to the C frame images one by one by the radar sensor to obtain C interval speed values;

[0104] B3: Record the time period covered by the C frame images as an interval period, and count the number of motor vehicles entering the second evaluation interval in the interval period, which is recorded as an interval vehicle value;

[0105] B4: Collect the maximum environmental humidity in the interval period in the second evaluation interval by the humidity sensor, which is recorded as an interval humidity value;

[0106] B5: After removing the dimensions of C interval speed values, interval vehicle values and interval humidity values respectively, C speed values, vehicle values and humidity values are obtained, and after assigning different weight coefficients to the C speed values, vehicle values and humidity values respectively, the C dynamic adjustment coefficients are calculated by adding them up;

[0107] The calculation formula of the dynamic adjustment coefficient is:

[0108] ;

[0109] In the formula, is the dynamic adjustment coefficient, is the speed value, is the vehicle value, is the humidity value, , , are the weight coefficients of the speed value, the vehicle value and the humidity value respectively, and ;

[0110] B6: The dynamic adjustment coefficients of the C frame images are compared with the standard duration respectively, and C interval durations are calculated; the standard duration refers to the interval duration between two target images under the condition of no error;

[0111] The calculation formula of the interval duration is:

[0112] ;

[0113] In the formula, is the interval duration, is the standard duration;

[0114] B7: The time of the frame image located in the first position is taken as the starting time, and from the starting time, the time after the interval duration corresponding to the starting time is recorded as the target time, and the frame image corresponding to the target time is recorded as the target image;

[0115] B8: The target time is taken as the starting time, and the steps of B7 are repeatedly executed until the target time marked is no longer located in the interval period, and all target images and frame images located in the first position are summarized to obtain D face images.

[0116] After the face images are collected, the face images need to be imported into the feature recognition model for deep learning, and the face features capable of directly representing the facial expressions of the driver at different times in the second evaluation interval are extracted from the face images;

[0117] Specifically, the facial features include an eyelid opening value and a gaze downward value; the eyelid opening value refers to a vertical distance between the upper eyelid and the lower eyelid of the driver in the facial image, that is, the eyelid opening value can represent the degree of opening and closing of the eyes of the driver; the smaller the eyelid opening value, the greater the probability that the driver closes his eyes, and the greater the probability that the driver drives while feeling tired; the gaze downward value refers to the downward deviation of the line of sight of the eyes of the driver from the horizontal plane, that is, the gaze downward value can represent the situation of the driver lowering his head and dozing off; the greater the gaze downward value, the greater the probability that the driver dozes off, and the greater the probability that the driver drives while feeling tired.

[0118] The feature recognition model is an artificial intelligence model based on the YOLOv5 model and combining deep learning technology to perform deep learning and recognition on the eyelid opening value and the gaze downward value in the facial image. When the feature recognition model is used, the model is not directly obtained, but is obtained through training and optimization based on a large number of existing facial images and facial features.

[0119] Specifically, in training the feature recognition model, a plurality of sets of facial images and facial features corresponding to the facial images are pre-acquired, a set of facial images is labeled as a set of feature vectors, a plurality of sets of feature vectors are obtained, and the facial features are converted into labels corresponding to the feature vectors.

[0120] One feature vector corresponds to one label, forming a set of training data, a plurality of sets of training data form a training set, and the labeled training data are divided into a training set and a test set; 70% of the training data are used as the training set, and 30% of the training data are used as the test set.

[0121] The feature vectors are used as the input of the YOLOv5 model, and the facial features corresponding to the facial images are used as the output of the YOLOv5 model; the training set is used to train the YOLOv5 model, and the test set is used to test the YOLOv5 model; a preset error threshold is set; when the average of the prediction errors of all training data in the test set is less than the preset error threshold, a feature recognition model for recognizing facial features from facial images is obtained.

[0122] In this embodiment, the D facial images collected are input into the feature recognition model, and D eyelid opening values and D gaze downward values corresponding to the D facial images are recognized.

[0123] It should be noted that the eyelid opening values and the gaze downward values collected above can be directly used as a data basis for determining whether the driver drives while feeling tired in the second evaluation interval, or can be analyzed comprehensively with the relevant data collected in the subsequent first evaluation interval.

[0124] S04: analyzing and evaluating the vehicle speed feature and the face feature, evaluating the driving state of the driver in the first evaluation interval and the second evaluation interval, and determining whether to issue a dangerous warning information;

[0125] After obtaining the local attenuation value, the terminal speed value, the eyelid opening value and the downward gaze value of the driver in the first evaluation interval and the second evaluation interval, comprehensive analysis and evaluation can be performed on the local attenuation value, the terminal speed value, the eyelid opening value and the downward gaze value, so as to analyze the real-time driving condition of the driver in the first evaluation interval and the second evaluation interval, and further accurately evaluate whether the driver has fatigue driving phenomenon.

[0126] When performing comprehensive analysis on the local attenuation value, the terminal speed value, the eyelid opening value and the downward gaze value, since these features are not for the same dimension and interval, independent analysis and evaluation need to be performed on the vehicle speed feature and the face feature in the first evaluation interval and the second evaluation interval, and the final analysis and evaluation result is represented by the driving state;

[0127] Specifically, the driving state includes a fatigue state and a safety state; wherein the fatigue state means that the driver has fatigue and drowsiness phenomenon in the local position, and the safety state means that the driver does not have fatigue and drowsiness phenomenon in the local position.

[0128] The determination method of the driving state in the first evaluation interval is as follows:

[0129] According to the chronological order, the B local attenuation values are arranged in sequence, and the positive and negative relationships of the B local attenuation values are identified in sequence;

[0130] When the local attenuation value is negative or 0, it means that the driver does not decelerate the target vehicle when the target vehicle passes through the monitoring point, at this time the driver may have fatigue driving phenomenon, and the driving state of the monitoring point in the first evaluation interval is determined as the fatigue state;

[0131] When the local attenuation value is positive, it means that the driver decelerates the target vehicle when the target vehicle passes through the monitoring point, at this time the size relationship between the terminal speed value and the calibrated safety speed value is compared;

[0132] If the terminal speed value is greater than or equal to the calibrated safety speed value, it means that the vehicle speed of the driver driving the target vehicle when driving out of the first evaluation interval is too high, at this time the driver may have fatigue driving phenomenon, and the driving state of the first evaluation interval is determined as the fatigue state; the calibrated safety speed value means the maximum speed of the target vehicle when driving out of the first evaluation interval without collision risk to the construction personnel of the construction area, and is one of the bases for judging whether the driver has fatigue driving phenomenon;

[0133] If the terminal velocity value is less than the calibrated safe speed value, it indicates that the vehicle speed of the target vehicle driven by the driver when driving out of the first evaluation interval is not high, and the driver does not have the phenomenon of fatigue driving, and the driving state of the first evaluation interval is determined as a safe state.

[0134] It should be noted that the analysis and evaluation operations of the driving states in the first evaluation interval and the second evaluation interval are parallel and independent, and there is no mutual interference phenomenon between them.

[0135] The determination method of the driving state of the second evaluation interval is:

[0136] In the order of time, the D face images are arranged in sequence, the interval period is divided into E sub-periods according to a preset evaluation time length, and the face images at the target time in the same sub-period are collected to obtain E image sets; the preset evaluation time length is used as a basis for limiting the time length of the sub-period, that is, the face images at different times can be classified and collected into a set, thereby facilitating the subsequent effective integration of the face images with a long time span at the same time length;

[0137] The eye opening value less than the calibrated opening value is recorded as an abnormal opening value, and the number of abnormal opening values in the E image sets is counted respectively; the calibrated opening value refers to the minimum value of the eye opening value that is not identified as closed eyes; the calibrated opening value is obtained by averaging the minimum values of a large number of eye opening values that are not identified as abnormal opening values in the history;

[0138] After comparing the number of abnormal opening values with the number of eye opening values in the E image sets, E closure proportion values are obtained;

[0139] When the closure proportion value is greater than or equal to a preset first proportion value, the eyes of the driver have a higher frequency of closing operation at this time, and the driver has the phenomenon of fatigue driving in the sub-period, and the driving state of the sub-period in the second evaluation interval is determined as a fatigue state; the preset first proportion value refers to the minimum value of the closure proportion value when fatigue driving is determined;

[0140] The gaze downward movement value greater than the calibrated downward movement value is recorded as an abnormal downward movement value, and the number of abnormal downward movement values in the E image sets is counted respectively; the calibrated downward movement value refers to the minimum value of the gaze downward movement value that is not identified as closed eyes; the calibrated downward movement value is obtained by averaging the minimum values of a large number of gaze downward movement values that are not identified as abnormal downward movement values in the history;

[0141] After comparing the number of abnormal downward movement values with the number of gaze downward movement values in the E image sets, E downward movement proportion values are obtained;

[0142] When the downward proportion value is greater than or equal to the preset second proportion value, the driver's head appears a higher frequency of lowering operation, and the driver appears a fatigue driving phenomenon in the sub-period, and the driving state of the sub-period in the second evaluation interval is determined as a fatigue state; the preset second proportion value refers to the minimum value of the downward proportion value when the fatigue driving is determined;

[0143] When the closing proportion value is less than the preset first proportion value, and the downward proportion value is less than the preset second proportion value, the driver does not appear a fatigue driving phenomenon in the sub-period, and the driving state of the sub-period in the second evaluation interval is determined as a safe state.

[0144] The danger warning information is used to represent the alarm result corresponding to the real-time driving state of the driver in the first evaluation interval and the second evaluation interval, and serves as a basis for subsequent alarm and prompt to the construction personnel and the driver in the construction area;

[0145] Specifically, the determination method of whether to issue the danger warning information is:

[0146] When the fatigue state appears in the local position in the first evaluation interval or the second evaluation interval, the construction personnel and the driver need to be pre-warned and prompted, and it is determined to issue the danger warning information;

[0147] When the fatigue state does not appear in the local position in the first evaluation interval or the second evaluation interval, the construction personnel and the driver do not need to be pre-warned and prompted, and it is determined not to issue the danger warning information.

[0148] S05: A danger warning unit is constructed in the analysis warning interval, and the danger warning unit performs a sound and light combined alarm operation under the control of the control terminal based on the danger warning information;

[0149] The danger warning unit refers to the overall structure of the related equipment installed on the highway to issue a danger warning prompt information in the analysis warning interval;

[0150] The danger warning unit includes a sound warning device and a light intensity warning device; wherein the number of the sound warning device and the light intensity warning device is multiple, the sound warning device and the light intensity warning device are arranged on the guardrails on both sides of the highway at the same position as the analysis warning interval, and are powered by a solar photovoltaic panel, and maintain wireless communication with the control terminal.

[0151] After the danger warning unit is constructed, the danger warning unit can perform a sound and light combined alarm operation under the control of the control terminal, thereby playing a sound and light combined pre-warning role for the driver and the construction personnel, reminding the driver to keep awake, and prompting the construction personnel to pay attention to avoid, thereby avoiding the phenomenon of vehicle and construction personnel collision accidents, and improving the safety of the construction personnel in the construction area.

[0152] Specifically, in the execution of the sound and light combined alarm operation, the control terminal sends alarm information to the sound alarm and the light intensity alarm, at this time the sound alarm continuously broadcasts the alarm voice to the outside, and the light intensity alarm continuously flashes the warning light to the outside.

[0153] It should be noted that after the sound and light combined alarm is performed by the danger alarm unit, the relevant warning information of the construction area can also be sent to the navigation software on the mobile device of the construction personnel and the mobile device of the driver through the Internet of Things technology, so that the construction personnel can quickly understand the possible fatigue driving situation of the driver through the terminal device and make corresponding danger avoidance preparations and measures, and also remind the driver of the position and distance of the construction area in front of the road, reminding the driver to maintain a safe driving state.

[0154] Embodiment two: please refer to Figure 3 As shown, the part not described in detail in this embodiment is described in embodiment one, and a driver analysis state comprehensive evaluation system based on multi-source information fusion is provided, which is applied to a control terminal, and is used to realize a driver analysis state comprehensive evaluation method based on multi-source information fusion. The method comprises a multi-dimensional interval division module, a vehicle speed feature extraction module, a face feature extraction module, a driving state analysis module and a sound and light combined alarm module, wherein each module is connected through wired or wireless network mode;

[0155] The multi-dimensional interval division module is used to record the point where the target vehicle passes for the first time in the construction area as an evaluation point, take the evaluation point as the reference point, identify the first evaluation interval and the second evaluation interval for collecting multi-source information, and the analysis warning interval for performing the sound and light combined alarm;

[0156] The vehicle speed feature extraction module is used to build a vehicle speed monitoring unit in the first evaluation interval, collect vehicle speed data of the target vehicle in the first evaluation interval through the vehicle speed monitoring unit, and extract vehicle speed features from the vehicle speed data;

[0157] The face feature extraction module is used to build a face monitoring unit in the second evaluation interval, collect the face image of the driver in the second evaluation interval based on the dynamic interval criterion through the face monitoring unit, and input the face image into the feature recognition model to identify the face features of the driver;

[0158] The driving state analysis module is used to analyze and evaluate the vehicle speed features and the face features, evaluate the driving state of the driver at the local position in the first evaluation interval and the second evaluation interval, and determine whether to issue a danger warning information;

[0159] The sound-light combined alarm module is used for setting up a dangerous alarm unit in the analysis early warning interval, and controlling the terminal to drive the dangerous alarm unit to perform sound-light combined alarm operation based on the dangerous alarm information.

[0160] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A driving state comprehensive evaluation method based on multi-source information fusion, applied to a control terminal, characterized in that, The method comprises the following steps: S01: record the point where the target vehicle passes for the first time in the construction area as an evaluation point, take the evaluation point as a reference point, identify a first evaluation interval and a second evaluation interval for collecting multi-source information, and an analysis and early warning interval for performing sound and light combined alarm; S02: build a vehicle speed monitoring unit in the first evaluation interval, collect vehicle speed data of the target vehicle in the first evaluation interval through the vehicle speed monitoring unit, and extract vehicle speed features from the vehicle speed data; S03: build a face monitoring unit in the second evaluation interval, collect face images of the driver in the second evaluation interval through the face monitoring unit based on a dynamic interval criterion, and input the face images into a feature recognition model to identify the face features of the driver; The face detection unit comprises a quick-release truss, a zoom camera and a humidity sensor, and the dynamic interval criterion is that the time span between adjacent two face images is inversely proportional to the dynamic adjustment coefficient; The face image collection method is as follows: B1: collect a video file of the target vehicle in the second evaluation interval through the zoom camera, perform video stream decompression on the video file, and generate C continuous frame images; B2: detect the vehicle speed at the corresponding time of the C frame images one by one through the radar sensor to obtain C interval speed values; B3: record the time period covered by the C frame images as an interval time period, and count the number of motor vehicles entering the second evaluation interval in the interval time period, which is recorded as an interval vehicle value; B4: collect the maximum environmental humidity of the second evaluation interval in the interval time period through the humidity sensor, which is recorded as an interval humidity value; B5: after eliminating the dimensions of the C interval speed values, the interval vehicle value and the interval humidity value, respectively, C speed values, vehicle values and humidity values are obtained, and the C speed values, vehicle values and humidity values are added after being respectively assigned different weight coefficients, and C dynamic adjustment coefficients are calculated; B6: compare the dynamic adjustment coefficients of the C frame images with the standard time length respectively to calculate C interval time lengths; B7: take the time of the frame image located in the first position as a starting time, and take the time after the interval time length corresponding to the starting time as a target time, and take the frame image corresponding to the target time as a target image; B8: take the target time as the starting time, repeat the step B7 until the marked target time is no longer located in the interval time period, and then stop, and all the target images and the frame image located in the first position are collected to obtain D face images; S04: analyze and evaluate the vehicle speed features and the face features to evaluate the driving state of the driver at the local position in the first evaluation interval and the second evaluation interval, and determine whether to issue a danger warning information; S05: build a danger alarm unit in the analysis and early warning interval, and control the terminal to drive the danger alarm unit to perform sound and light combined alarm operation based on the danger warning information. 2.The driver analysis state comprehensive evaluation method based on multi-source information fusion according to claim 1, characterized in that, The identification method of the multi-source evaluation interval is as follows: In the construction lane, mark a point at a position which is one calibrated safety distance away from the reference point in the direction of the incoming vehicle, and record the point as an interval end point; Mark a point at a position in the direction of the incoming vehicle and a preset first monitoring length and a preset second monitoring length away from the end of the interval, respectively, and mark it as the first starting point and the second starting point; Mark the area between the first starting point and the end of the interval as the first evaluation interval, and mark the area between the second starting point and the end of the interval as the second evaluation interval.

3. The driver analysis state comprehensive evaluation method based on multi-source information fusion according to claim 2, characterized in that, The vehicle speed monitoring unit includes a quick-release truss and a radar sensor; the vehicle speed feature includes a local attenuation value and a terminal speed value.

4. The driver analysis state comprehensive evaluation method based on multi-source information fusion according to claim 3, characterized in that, The extraction method of the local attenuation value is: A1: Take the first starting point as the entry point, collect the vehicle speed of the target vehicle passing through the entry point by the radar sensor, and mark it as the entry speed value; A2: Convert the entry speed value to vehicle second speed, and then expand the vehicle second speed by a corresponding second speed multiple to generate a monitoring distance value; A3: Mark a monitoring point after the entry point in the direction of the vehicle and a distance of the monitoring distance value away from the entry point; A4: Take the monitoring point as the entry point, and repeat steps A1-A3 until the monitoring point is no longer in the first evaluation interval, then stop; collect the entry points and all monitoring points to obtain B+1 monitoring points; A5: Arrange the entry speed values of the B+1 monitoring points in order, and then obtain B local attenuation values by subtracting the entry speed value of the previous monitoring point from the entry speed value of the next monitoring point.

5. The driver analysis state comprehensive evaluation method based on multi-source information fusion according to claim 4, characterized in that, The facial features include an eyelid opening value and a gaze downward value; Import the collected D facial images into the feature recognition model to identify D eyelid opening values and D gaze downward values corresponding to the D facial images.

6. The driver analysis state comprehensive evaluation method based on multi-source information fusion according to claim 5, characterized in that, The driving state includes a fatigue state and a safety state; The determination method of the driving state of the first evaluation interval is: Identify the positive and negative relationships of the B local attenuation values in order according to the chronological order; When the local attenuation value is negative or 0, the driving state of the first evaluation interval is determined as the fatigue state; When the local attenuation value is positive, compare the size relationship between the terminal speed value and the calibrated safety speed value; If the terminal speed value is greater than or equal to the calibrated safety speed value, the driving state of the first evaluation interval is determined as the fatigue state; If the terminal speed value is less than the calibrated safety speed value, the driving state of the first evaluation interval is determined as the safety state.

7. The driver analysis state comprehensive evaluation method based on multi-source information fusion according to claim 6, characterized in that, The determination method of the driving state of the second evaluation interval is: Divide the interval period into E sub-periods according to a preset evaluation time, and then obtain E image sets by collecting the facial images in the same sub-period at the target time; Mark the eye opening values less than the calibrated opening value as abnormal opening values, and then respectively count the number of abnormal opening values in the E image sets; compare the number of abnormal opening values in the E image sets with the number of eye opening values to obtain E closing proportion values; Mark the gaze downward values greater than the calibrated downward value as abnormal downward values, and then respectively count the number of abnormal downward values in the E image sets; compare the number of abnormal downward values in the E image sets with the number of gaze downward values to obtain E downward proportion values; When the closing proportion value is greater than or equal to a preset first proportion value, or the downward moving proportion value is greater than or equal to a preset second proportion value, the driving state of the second evaluation interval is determined as a fatigue state; When the closing proportion value is less than the preset first proportion value, and the downward moving proportion value is less than the preset second proportion value, the driving state of the second evaluation interval is determined as a safe state; The determination method of whether to issue a danger alarm information is: When the fatigue state occurs, it is determined to issue the danger alarm information; when the fatigue state does not occur, it is determined not to issue the danger alarm information.

8. The driver analysis state comprehensive evaluation method based on multi-source information fusion according to claim 7, characterized in that, The danger alarm unit includes a sound warning device and a light intensity warning device; When the sound and light combination alarm operation is performed, the control terminal sends alarm information to the sound warning device and the light intensity warning device, the sound warning device continuously broadcasts warning voice outward, and the light intensity warning device continuously flashes warning light outward.

Images