Driver abnormality monitoring method, device and storage medium

Abstract

The present application relates to a driver abnormality monitoring method, device and storage medium, the method comprising obtaining N different driving monitoring parameters of the driver, where N≥2; when at least one type of the driving monitoring parameter is within the corresponding parameter abnormality range, executing a first warning instruction based on the abnormal driving monitoring parameter; otherwise, obtaining a comprehensive evaluation value based on the N different driving monitoring parameters, and executing a second warning instruction when the comprehensive evaluation value is within the comprehensive abnormality range; the method of the present application is used to improve the problem of low accuracy of driver health abnormality monitoring in the prior art.

Description

Technical Field

[0001] The present application relates to the field of automobile technology, and in particular to a method, device, and storage medium for monitoring driver abnormality. Background Art

[0002] The driver's physical condition directly affects road transport safety. Healthy drivers can effectively avoid fatigue driving and reduce the probability of traffic accidents. When a driver's health condition deteriorates, driving behavior will be affected, such as decreased vision, hearing, and reaction ability.

[0003] The driver monitoring system can detect the driver's abnormal state in time and remind the driver to intervene through voice and text messages to avoid traffic accidents caused by physical discomfort. The current driver monitoring system can detect and issue warnings based on a certain physiological characteristic of the driver, but the warning results of a single parameter are not comprehensive and have poor accuracy. Summary of the Invention

[0004] Based on this, a driver abnormality monitoring method, device and storage medium are provided to improve the problem of low accuracy of driver abnormality monitoring in the prior art.

[0005] In one aspect, a method for monitoring driver abnormality is provided, the method comprising:

[0006] Obtain N different driving monitoring parameters of the driver, where N ≥ 2;

[0007] When at least one type of the driving monitoring parameter is within a corresponding parameter abnormal range, executing a first warning instruction based on the abnormal driving monitoring parameter;

[0008] Otherwise, a comprehensive evaluation value is obtained based on the N different driving monitoring parameters, and when the comprehensive evaluation value is within a comprehensive abnormal range, a second early warning instruction is executed.

[0009] In one embodiment, obtaining N different driving monitoring parameters of the driver includes:

[0010] Acquiring the driver's motion amplitude parameter as one of the driving monitoring parameters includes:

[0011] Acquire a first target image set;

[0012] Obtaining a first target image and a second target image based on the first target image set, and determining a first contour region corresponding to a first monitoring target in the first target image and a second contour region of the first monitoring target in the second target image, respectively, wherein the first monitoring target includes at least a partial torso region of the driver;

[0013] Obtaining an overlapping area, a first edge area, and a second edge area according to the first contour area and the second contour area, wherein the first edge area is the area of ​​a non-overlapping area in the first contour area, and the second edge area is the area of ​​a non-overlapping area in the second contour area;

[0014] The displacement change rate of the first monitoring target is obtained according to the overlapping area, the first edge area, and the second edge area, and the motion amplitude parameter is determined according to the displacement change rate.

[0015] In one embodiment, obtaining the displacement change rate of the first monitoring target according to the overlapping area, the first edge area, and the second edge area includes calculating the displacement change rate using the following mathematical expression:

[0016]

[0017] Wherein, ΔS is the displacement change rate, S A is the first edge area, S B is the second edge area, S AB is the overlapping area.

[0018] In one embodiment, after acquiring the first target image and the second target image according to the first target image set, the method further includes:

[0019] determining coordinates of a positioning point of a second monitoring target according to the target image, including obtaining a first positioning point according to the first target image and obtaining a second positioning point according to the second target image, wherein the second monitoring target includes a hand area of ​​the driver;

[0020] The offset rate of the second monitoring target is determined according to the coordinates of the first positioning point and the second positioning point, so as to determine the motion amplitude parameter according to the offset rate of the second monitoring target and the displacement change rate of the first monitoring target.

[0021] In one embodiment, determining the coordinates of the positioning point of the second monitoring target according to the target image includes:

[0022] determining a closed contour curve of a second monitoring target according to the target image;

[0023] Obtaining a binarization matrix according to the target image, wherein the values ​​of the points in the binarization matrix located within the closed contour curve are set to 1, and the values ​​of the points located outside the closed contour curve are set to 0;

[0024] The coordinates of the positioning point (x c ,y c ):

[0025]

[0026]

[0027] in:

[0028]

[0029]

[0030]

[0031] Wherein, V(i, j) is the value at point (i, j) in the binarized matrix.

[0032] In one embodiment, obtaining N different driving monitoring parameters of the driver includes:

[0033] Obtaining a facial color level change rate as one of the driving monitoring parameters includes:

[0034] Acquire a second target image set, wherein each image in the second target image set includes at least the driver's face, and acquire a third target image and a fourth target image based on the second target image set;

[0035] According to the third target image and the fourth target image, the change in the number of pixels of the same color level in the target area is counted to obtain a color level change rate, wherein the target area includes a facial area, and the color level change rate includes the facial color level change rate corresponding to the facial area.

[0036] In one embodiment, the target area further includes a reference area, and the step of obtaining a color scale change rate by counting the change in the number of pixels of the same color scale in the target area in the third target image and the fourth target image further includes:

[0037] Obtaining a reference color scale change rate corresponding to the reference area by performing statistics based on a change in the number of pixels of the same color scale in the reference area in the third target image and the fourth target image;

[0038] Whether the facial color scale change rate is valid is determined according to the numerical value of the reference color scale change rate, so as to output a valid facial color scale change rate.

[0039] In one embodiment, obtaining a color scale change rate based on a statistical change in the number of pixels of the same color scale in the target area in the third target image and the fourth target image includes obtaining the color scale change rate based on the following mathematical expression:

[0040]

[0041] Wherein, f is the rate of change of the color scale of the target area, S is the total number of pixels in the target area, z′(m) is the number of pixels of the mth color scale in the fourth target image, z(m) is the number of pixels of the mth color scale in the third target image, where 0≤m≤255, △t is the sampling time interval between the third target image and the fourth target image, where the color scale of the pixel points is divided into 0-255.

[0042] In another aspect, a driver abnormality monitoring device is provided, the device comprising:

[0043] An acquisition module, used to acquire N different driving monitoring parameters of the driver, where N ≥ 2;

[0044] an evaluation calculation module, configured to obtain a comprehensive evaluation value based on the N different driving monitoring parameters;

[0045] A judgment module, configured to judge whether the driving monitoring parameter is within a parameter abnormal range or whether the comprehensive evaluation value is within a comprehensive abnormal range;

[0046] The early warning execution module is used to execute a first early warning instruction based on the abnormal driving monitoring parameter when at least one type of driving monitoring parameter is within the corresponding parameter abnormal range; otherwise, when the comprehensive evaluation value is within the comprehensive abnormal range, execute a second early warning instruction.

[0047] A computer-readable storage medium is also provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method are implemented.

[0048] The above-mentioned driver abnormality monitoring method, device and storage medium obtain multiple driving monitoring parameters of the driver. When a single parameter is within the parameter abnormal range, the warning instruction is directly executed based on the parameter; when all parameters are within the normal range, a comprehensive evaluation is performed to obtain a comprehensive evaluation value, and a warning is issued based on the comprehensive evaluation value; the driver abnormality monitoring method provided in this application realizes both single parameter judgment and comprehensive judgment process, thereby improving the accuracy of abnormality monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 1 is a flow chart of a method for monitoring driver abnormality in one embodiment;

[0050] Figure 2 Schematic diagram of a flow chart of the steps for obtaining motion amplitude parameters in one embodiment;

[0051] Figure 3 is a schematic diagram of the displacement of a first monitoring target in a target image in one embodiment;

[0052] Figure 4is a schematic diagram of the displacement of the second monitoring target in the target image in one embodiment;

[0053] Figure 5 Schematic diagram of a flow chart of a step for obtaining a facial color scale change rate in one embodiment;

[0054] Figure 6 1 is a flow chart of a step of calculating a color scale change rate in one embodiment;

[0055] Figure 7 A schematic diagram of histogram statistics in an embodiment;

[0056] Figure 8 1 is a flow chart of a method for monitoring driver abnormality in another embodiment;

[0057] Figure 9 is a structural block diagram of a driver abnormality monitoring device in one embodiment;

[0058] Figure 10 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0060] Drivers are an important link in transportation and are also the focus and difficulty of traffic safety management. Their health status not only directly affects traffic safety, but also is related to the safety of people's lives and property.

[0061] For driver monitoring, sensors are mostly used to monitor a characteristic physiological parameter of the driver. However, when the driver exhibits abnormal behavior, the monitoring results obtained from a single physiological parameter may not be accurate.

[0062] The present application provides a driver abnormality monitoring method, which can improve the above-mentioned problem of inaccurate monitoring results.

[0063] In one embodiment, Figure 1 As shown, the driver abnormality monitoring method includes the following steps:

[0064] Step 101: Obtain N different driving monitoring parameters of the driver, where N≥2.

[0065] For example, the driving monitoring parameters are parameters associated with the driver's physiological or emotional state, including but not limited to the driver's breathing rate, breathing depth, body temperature, heart rate, emotional parameters, fatigue index, etc. The present application may also include the driver's motion amplitude parameter, the facial color change rate indicating changes in the driver's facial skin color, etc.

[0066] Step 102, executing corresponding warning instructions based on driving monitoring parameters, includes the following process:

[0067] Step 1021: When at least one type of driving monitoring parameter is within a corresponding parameter abnormal range, a first warning instruction is executed based on the abnormal driving monitoring parameter.

[0068] It can be understood that each type of driving monitoring parameter of the driver corresponds to a normal parameter range and an abnormal parameter range. Taking respiratory frequency as an example, by using sample data of multiple drivers, the respiratory frequency that the driver usually has when he is in normal health status can be calibrated as the normal parameter range, and the parameter range outside the normal range is the abnormal parameter range (for example, when the respiratory frequency exceeds or falls below a certain threshold, it can be considered that the respiratory frequency is in the abnormal parameter range).

[0069] The parameter abnormal range of each type of driving monitoring parameter is obtained through calibration, and each type of driving monitoring parameter is independently judged. When a certain type of driving monitoring parameter is abnormal, the first warning instruction is directly executed.

[0070] The above-mentioned independent judgment process of multiple driving monitoring parameters can effectively monitor the abnormality of a certain physiological characteristic caused by sudden illness, etc., and avoid ignoring the abnormality of a single characteristic when the overall characteristic monitoring results are normal.

[0071] The above-mentioned single parameter monitoring results can only effectively monitor some abnormal situations. When all driving monitoring parameters are within their corresponding normal parameter ranges, there may be a situation where the driver's overall situation is not ideal. For example, there may be a situation where multiple driving monitoring parameters are at high levels but are all normal. This state may still bring driving risks. At this time, execute step 1022.

[0072] Step 1022: Obtain a comprehensive evaluation value based on the N different driving monitoring parameters, and execute a second warning instruction when the comprehensive evaluation value is within a comprehensive abnormal range.

[0073] It can be understood that the comprehensive evaluation value obtained based on the N different driving monitoring parameters can be a weighted calculation, which is obtained by assigning different weights to different types of driving monitoring parameters and weighting them. Furthermore, a comprehensive evaluation value can be obtained by using a neural network model with multiple driving monitoring parameters as input. The neural network model can be trained by inputting multiple batches of labeled sample data. During the training process, the weights or biases of each neural node in the neural network model are adjusted to obtain the optimal output value.

[0074] For example, the first warning instruction and the second warning instruction may be entirely or partially identical. For example, both warning instructions may include the step of executing a health abnormality reporting step.

[0075] The above-mentioned driver abnormality monitoring method obtains multiple driving monitoring parameters, combines single parameter judgment and multi-parameter comprehensive judgment to achieve more comprehensive monitoring and obtain more accurate monitoring results.

[0076] In one embodiment, the action amplitude parameter is used as one of the driving monitoring parameters, and the step of obtaining the driver's action amplitude parameter is as follows: Figure 2 Shown, including:

[0077] Step 201: Acquire a first target image set.

[0078] The first target image set is illustratively a continuous frame image obtained by a camera set at the driver's seat, the image sampling interval is fixed, and the image needs to include the driver.

[0079] Step 202: Acquire a first target image and a second target image based on the first target image set. For example, determine two consecutive frames of images as the first target image and the second target image respectively, obtain the contour of at least part of the driver's torso area through color difference boundary and image edge detection algorithms, and compare and reinforce them through dynamic contours and static contours.

[0080] The driver's torso area is used as a reference for motion monitoring and as a first monitoring target. The first monitoring target can be a part of the torso area, such as the thorax area, the shoulder area, or the entire torso except the limbs, head and neck.

[0081] like Figure 3 As shown, the closed contour curve of the first monitoring target in the first target image (image at time T1) can be obtained through the edge detection algorithm, and the area in the closed contour curve is determined as the first contour area A. The same method is used to determine the second contour area B of the first monitoring target in the second target image (image at time T2).

[0082] Step 203: Obtain an overlapping area, a first edge area, and a second edge area according to the first contour area and the second contour area.

[0083] For example, the first target image set adopts a unified coordinate space, and image recognition obtains the contour curve and its dynamic change range. The first monitoring target in the two sampling images (at time T1 and time T2) may be different in position and area, and the contour area can be obtained by counting the pixels in the image contour.

[0084] like Figure 3 In the figure, the overlapping area S AB is the area of ​​the overlapping portion of the first contour area A and the second contour area B, and the first edge area S A is the non-overlapping area in the first contour area A, and the second edge area S B is the area of ​​the non-overlapping region in the second contour region.

[0085] Step 204 : Obtain the displacement change rate of the first monitoring target according to the overlapping area, the first edge area, and the second edge area.

[0086] In monitoring of the body trunk, the main movements of the trunk exemplarily include leaning forward (toward the camera), leaning backward (away from the camera), and tilting left and right.

[0087] The main feature of the trunk leaning forward or backward in the image is the change in area size. When the trunk leans forward, the first edge area S A Small or even zero, the second edge area S B Greater than zero, overlapping area S AB Increase, meet the above conditions can be basically determined as the body leaning forward, the displacement change rate can be determined as the preset value.

[0088] During the process of the trunk leaning back, the first edge area S A Greater than zero, the second edge area S B Small or even zero, the overlapping area S AB If the above conditions are met, it can be basically determined that the body is leaning back, and the displacement change rate can also be determined as a preset value.

[0089] In some embodiments, the left and right tilt of the body can be controlled by the first edge area S A Or the second edge area S B and overlapping area S AB The displacement change rate is obtained by the ratio of . The larger the displacement, the smaller the overlapping area.

[0090] In this embodiment, the displacement change rate of the trunk is determined using the following mathematical expression for the movement amplitude:

[0091]

[0092] Wherein, ΔS is the displacement change rate.

[0093] In the above mathematical expression, the first edge area S A and the second edge area S B At the same time, calculations are introduced. In addition to the left and right displacements of the body, the area changes caused by the forward and backward leaning of the torso can also be reflected by the displacement change rate.

[0094] In this application, the motion amplitude index is determined based on the displacement change rate, and an exemplary explanation includes dividing the displacement change rate into levels according to a unified scoring system. The larger the displacement change rate value, the larger the level of the motion amplitude. When the level exceeds a certain threshold, an alarm message can be generated to guide driving.

[0095] The driver motion amplitude monitoring method provided in the above embodiment recognizes the image of a specific area of ​​the driver, calculates the area change of the specific area, obtains its displacement change rate, and determines the motion amplitude index based on the displacement change rate to identify the driving motion amplitude, making it convenient for the monitoring system to provide driving guidance based on its motion amplitude.

[0096] As an implementation of the above embodiment, the driver's hand movements may be monitored to improve the accuracy of motion amplitude recognition.

[0097] Exemplarily, after acquiring the first target image and the second target image according to the first target image set, the method further includes:

[0098] The coordinates of the positioning point of the second monitoring target in the image are determined according to the target image (the first target image and the second target image). The second monitoring target includes the driver's hand area. The positioning point can be a feature point of the second monitoring target, i.e., the hand area. In this embodiment, the center point of the hand area can be used as the positioning point.

[0099] The positioning points include a first positioning point obtained according to the first target image and a second positioning point obtained according to the second target image;

[0100] Determine the offset rate of the second monitoring target according to the coordinates of the first positioning point and the second positioning point, such as Figure 4 As shown, the distance between the two positioning points is determined according to the coordinates of the first positioning point C and the second positioning point D, and the time difference is obtained by the shooting time T1 of the first target image and the shooting time T2 of the second target image. The offset rate is obtained according to the distance and the time difference. When the offset rate is low, it can be considered that the driver's action is relatively smooth.

[0101] The motion amplitude index is determined based on the offset rate of the second monitoring target and the displacement change rate of the first monitoring target. The offset rate and displacement change rate can be graded using a unified scoring system (e.g., a 10-point system where the displacement change rate is 0 and the displacement change rate grade is 0, or the offset rate grade is 0 when the offset rate is 0; 10 points is 100% displacement change rate defined for a specific scenario, or the offset rate exceeds a set threshold). The motion amplitude index is obtained by comprehensively weighting the grades determined based on the two parameters. When the index exceeds one or more set thresholds, it indicates that the driver's body or hand movements are too large, and there may be improper driving behavior or potential driving hazards, and is used to generate warning information.

[0102] Using multi-target comprehensive monitoring can obtain more accurate monitoring results.

[0103] As an implementation method, the coordinates of the first positioning point and the second positioning point are obtained in the following manner:

[0104] determining a closed contour curve of a second monitoring target according to the target image;

[0105] Obtaining a binarization matrix according to the target image, wherein the values ​​of the points in the binarization matrix located within the closed contour curve are set to 1, and the values ​​of the points located outside the closed contour curve are set to 0;

[0106] The coordinates of the positioning point (x c ,y c ):

[0107]

[0108]

[0109] in:

[0110]

[0111]

[0112]

[0113] Wherein, V(i, j) is the value at point (i, j) in the binarized matrix.

[0114] In another embodiment, image recognition is used to obtain the rectangular area where the driver's hand is located, the coordinates of the four vertices of the rectangular area are obtained, and the coordinates of the diagonal intersection are calculated based on the coordinates of the four vertices as the positioning point of the hand area.

[0115] As an implementation of the above embodiment, it also includes a step of obtaining the offset of the first positioning point or the second positioning point from the steering wheel area to determine the motion amplitude index according to the offset of the second monitoring target and the displacement change rate of the first monitoring target.

[0116] The steering wheel area can be a set area in the first target image set, and the area where the steering wheel is located in the image set obtained by the fixed camera is basically consistent. In one embodiment, the offset can be the closest distance between the first positioning point and the edge of the steering wheel area. In more cases, the offset can be determined based on the closest distance between the second positioning point and the edge of the steering wheel area.

[0117] In some embodiments, the line connecting the first and second positioning points is used as the hand trajectory, and the sum of the shortest distances between the hand trajectory as a whole and the steering wheel area (or the sum of the distances between the characteristic points of the steering wheel area, such as the center point) is used as the offset. For example, Figure 4 The area of ​​the minimum triangle connecting area E shown is used as the offset. Compared with the method of using the straight-line distance between a single positioning point and the steering wheel area as the offset, the calculation result of this embodiment is more accurate.

[0118] It can be understood that when the driver drives normally with both hands on the steering wheel, the positioning point is in the steering wheel area and the offset is regarded as 0.

[0119] In this embodiment, the motion amplitude index can be determined based on the offset of the second monitoring target and the displacement change rate of the first monitoring target. The offset and displacement change rate can be graded using a unified scoring system, and the grades determined based on the two parameters can be comprehensively weighted to obtain the motion amplitude index.

[0120] It is understandable that in some embodiments, the motion amplitude index can also be determined based on the offset rate and offset amount of the hand area and the displacement change rate of the torso area, and the three can be comprehensively analyzed to obtain a more accurate motion amplitude index.

[0121] On the other hand, the second monitoring target includes the driver's left and right hand areas, and the first offset rate and the second offset rate can be obtained respectively. The two can be weighted in equal proportion to obtain the final offset rate. Similarly, the first offset and the second offset are obtained based on the left and right hand areas, and the two are weighted in equal proportion to obtain the final offset.

[0122] In one embodiment, the facial color level change rate is used as one of the driving monitoring parameters. The driver abnormality monitoring method of this embodiment is as follows: Figure 5 The color level change rate is obtained as shown, including:

[0123] Step 301 : Acquire a second target image set, wherein each image in the second target image set includes at least a facial area of ​​the driver, and acquire a third target image and a fourth target image based on the second target image set.

[0124] Exemplarily, the second target image set can be obtained by an image acquisition module, such as an image recognition device, which is arranged at the driving position of the vehicle. The image recognition device continuously shoots the driving area and extracts all images with the driver's facial area as the second target image set after the driver enters the cockpit. It can be understood that the first target image set and the second target image set can be collected by the same device, and even the first target image set and the second target image set can be the same image set.

[0125] In some embodiments, two adjacent frames of images are obtained as the third target image and the fourth target image.

[0126] Step 302: Count the changes in the number of pixels of the same color scale in the target area in the third target image and the fourth target image to obtain a color scale change rate. When the target area is a facial area, the color scale change rate is the facial color scale change rate corresponding to the facial area.

[0127] For example, in this embodiment, the entire facial region of the driver in the image is used as the target region for analysis. In some embodiments, some characteristic sub-regions in the facial region may also be used as analysis targets.

[0128] It can be understood that in the target image (the third target image or the fourth target image), the outer contour closed curve of the driver's facial area can be obtained through, for example, chromatic aberration boundaries and image edge detection algorithms, and contrast and reinforced through dynamic contours and static contours, and the area within the outer contour closed curve is used as the driver's facial area, and the area outside the closed curve is used as the environmental area.

[0129] Among them, the color level of each pixel in the facial area is determined according to the brightness division, which can also be called grayscale in some implementation methods. The number of pixels at the same color level in the third target image and the fourth target image is counted respectively, and the change in the number of pixels at the same color level is obtained by subtraction. In this embodiment, based on the time interval between the two frames of target images, the speed of change of the number of pixels at the same color level is obtained as the color level change rate. In other embodiments, based on the number of all pixels in the target area in the two frames of target images, the proportion of pixels whose color level has changed is obtained as the color level change rate.

[0130] It can be understood that the facial color scale change rate reflects the degree of change in the brightness of the driver's facial skin color. For example, when the facial color scale change rate exceeds a certain threshold, it is judged that the driver's facial skin color has undergone a significant change, such as pale skin color, and it can be considered that the driver has health abnormalities.

[0131] Compared with portable health monitoring devices such as bracelets, the use of the above-mentioned image recognition method to obtain health status can, on the one hand, achieve contactless monitoring; on the other hand, the advantage is that after the driver leaves the vehicle, the status monitoring results will no longer be output, which is conducive to protecting the driver's privacy and security.

[0132] On the other hand, compared with contact sensors installed on the steering wheel, seat, etc., the monitoring method provided by the present application does not require the driver to maintain contact, will not affect the driver's movements, and is conducive to driving safety.

[0133] In one embodiment, based on the time interval between two frames of target images, the speed of change of the number of pixels of the same color level is obtained as the color level change rate, such as Figure 6 As shown, the following steps are included:

[0134] Step 401: Based on the decomposition of the third target image, a first single-channel image of the target area in the target channel is obtained. For example, if the third target image is a color image in RGB space, the color level of the image's R channel can be selected as a feature value for calculation to determine the driver's health status. For example, by setting the color level values ​​of both the G and B channels in the first target image to 0, a first single-channel image containing only the color level value of the R channel can be obtained.

[0135] Perform histogram statistics on the color levels of the pixels of the first single-channel image to obtain a first single-channel histogram, such as Figure 7 As shown in a, the first single-channel histogram can be a frequency histogram or a count histogram. In this embodiment, the first single-channel image has color levels as the horizontal axis, and the ratio of the number of pixels in each color level interval to the total number of pixels in the target area (facial area) is the vertical axis.

[0136] On the other hand, according to the decomposition of the fourth target image, a second single-channel image of the target area under the target channel is obtained, and the color levels of the pixels of the second single-channel image are subjected to histogram statistics to obtain a second single-channel histogram, such as Figure 7 As shown in b.

[0137] Step 402: Based on the comparison between the first single-channel histogram and the second single-channel histogram, the numerical difference of each group of intervals is obtained, and the sum is calculated to obtain the total amount of color level change, for example Figure 7 The non-overlapping parts of the first single-channel histogram and the second single-channel histogram shown in c;

[0138] Step 403: Acquire the sampling time interval of the third target image and the fourth target image, and obtain the color level change rate according to the total amount of color level change and the sampling time interval.

[0139] It can be understood that within the same time interval, the greater the total amount of color level change and the greater the obtained color level change rate, the more drastic the change in the driver's facial skin color is, and the greater the possibility of health abnormalities.

[0140] In some embodiments, a multi-channel color image is used as the target image, and statistics are performed on the color levels of each channel to identify changes in the driver's facial skin color.

[0141] For example, the total amount of color level change of the target area, i.e., the facial area, under each target channel is obtained; the single-channel color level change rate is obtained based on the total amount of color level change under each target channel and the sampling time interval, and the color level change rate is obtained by taking the average of the single-channel color level change rates under each target channel.

[0142] For example, for the R, G, and B channels of the RGB image space, by retaining the color level of one channel and setting the color level values ​​of the other two channels to zero, we can obtain an R channel image with only the color level value of the R channel, a G channel image with only the color level value of the G channel, and a B channel image with only the B channel image, respectively. Then, we perform histogram statistics to obtain the R channel histogram, G channel histogram, and B channel histogram.

[0143] The total amount of R channel color level change is obtained by summing the same color level differences of the R channel histograms obtained from the two frames of target images, and based on this, the R channel color level change rate is obtained.

[0144] The total amount of G channel color level change is obtained by summing the color level differences of the G channel histograms obtained from the two frames of target images, and based on this, the G channel color level change rate is obtained.

[0145] The total amount of B channel color level change is obtained by summing the color level differences of the B channel histograms obtained from the two frames of the target image, and based on this, the B channel color level change rate is obtained.

[0146] The final color level change rate is obtained by taking the average of the R channel color level change rate, the G channel color level change rate, and the B channel color level change rate.

[0147] The above method performs multi-channel statistics on color images to improve the accuracy of judgment.

[0148] During the driving of the vehicle, the driver's facial light environment will be affected by external light. In this application, the accuracy of judgment is improved by eliminating the influence of environmental factors.

[0149] In one embodiment, the target area further includes a reference area, which may be an environmental area other than the facial area, and may be the entire environmental area or a part of the environmental area. The reference area may also be a characteristic area in the facial area.

[0150] The present application obtains a reference color level change rate corresponding to the reference area by performing statistics based on the change in the number of pixels of the same color level in the reference area in the third target image and the fourth target image;

[0151] Whether the facial color scale change rate is valid is judged according to the numerical value of the reference color scale change rate, so as to judge the driver's health status according to the valid facial color scale change rate.

[0152] Taking a color image in RGB space as an example, according to the decomposition of the third target image, a first single-channel image of the reference area is obtained, and the first single-channel image includes a first R channel image, a first G channel image, and a first B channel image. The first R channel histogram, the first G channel histogram, and the first B channel histogram corresponding to the reference area in the third target image are further obtained by statistics.

[0153] According to the decomposition of the fourth target image, a second single-channel image of the reference area is obtained, the first single-channel image includes a second R-channel image, a second G-channel image, and a second B-channel image, and further statistics are obtained to obtain a second R-channel histogram, a second G-channel histogram, and a second B-channel histogram corresponding to the reference area in the fourth target image.

[0154] The color level change rate of the R channel of the reference area is obtained based on the sampling time interval by summing the difference between the color levels of the first and second R channel histograms; the color level change rate of the G channel of the reference area is obtained based on the sampling time interval by summing the difference between the same color levels in the first and second G channel histograms; the color level change rate of the B channel of the reference area is obtained based on the sampling time interval by summing the difference between the same color levels in the first and second B channel histograms; and the color level change rate of the reference area is obtained based on the sampling time interval. The color level change rate of the reference area is then obtained by averaging the color level change rates of the R, G, and B channels, i.e., the reference color level change rate.

[0155] It can be understood that a validity threshold can be set in the vehicle system. When the reference color scale change rate is less than the validity threshold, it can be considered that the impact of environmental factors in the first target image and the second target image is low, that is, the facial color scale change rate is considered valid. The facial color scale change rate is output only when the reference color scale change rate is valid, otherwise it is not output.

[0156] In one embodiment, the color level of each pixel is divided into 0-255, and the color level change rate can be obtained according to the following mathematical expression:

[0157]

[0158] Wherein, f is the rate of change of the color scale of the target area, S is the total number of pixels in the target area, z′(m) is the number of pixels of the mth color scale in the fourth target image, z(m) is the number of pixels of the mth color scale in the third target image, where 0≤m≤255, △t is the sampling time interval between the third target image and the fourth target image, where the color scale of the pixel points is divided into 0-255.

[0159] For example, the facial color level change rate of the facial area is obtained using the following mathematical expression:

[0160]

[0161] In the above mathematical expression, f1 is the facial color level change rate, z′1(m) is the value of the mth class interval (color level value is m) in the second single-channel histogram corresponding to the facial area, z1(m) is the value of the mth class interval in the first single-channel histogram corresponding to the facial area, and S1 is the total number of pixels in the facial area.

[0162] It can be understood that when calculating the facial area, the value of each pixel in the ambient area is set to "0" and the value of each pixel in the facial area is set to "1". A two-dimensional matrix is ​​generated. The value U(i, j) at the pixel point (i, j) satisfies the set state. Therefore, the area of ​​the facial area (the total number of pixels in the facial area) is the sum of all pixel values ​​in this matrix, that is:

[0163]

[0164] The same method can be used to obtain the total number of pixels corresponding to the reference area and the reference color level change rate.

[0165] In one embodiment, a respiratory health parameter is used as one of the driving monitoring parameters, and the respiratory health parameter includes one of respiratory rate and respiratory depth, or a combination of both.

[0166] For example, in this embodiment, the respiratory frequency and respiratory depth are acquired by image recognition.

[0167] First, the third target image set is obtained, and the outer contour of the driver's body is obtained through color difference boundary and image edge detection algorithms, and then compared and reinforced through dynamic contours and static contours.

[0168] The outer contour includes at least the driver's chest or shoulder width, and the area within the chest is designated as the chest area, the area (total number of pixels) of the chest area or the shoulder width is continuously obtained, and the period of area change or shoulder width change is further obtained as the breathing frequency; on the other hand, the area change (rate of change) of the chest area or the change in shoulder width is obtained as the breathing depth.

[0169] In the above embodiments, driving monitoring parameters are obtained through image recognition, thereby achieving non-sensing monitoring without affecting the driver's driving.

[0170] In one embodiment, Figure 8 As shown, the infrared plus RGB dual-lens module obtains the above-mentioned driving monitoring parameters, and can also obtain the driver's body temperature parameters, emotional parameters, and fatigue parameters to participate in abnormal monitoring. The emotional parameters and fatigue parameters can be obtained through the output of the neural network model; in this embodiment, multiple parameters are used as input to conduct a comprehensive assessment of the driver's health and driving status, and driving abnormality reporting is executed when an abnormality occurs.

[0171] As an implementation of the above-mentioned multiple embodiments, it also includes pre-processing of the target image set, that is, verifying the validity of the data to filter out interfering data and improve the accuracy of the calculation results.

[0172] The data validity verification includes checking the data validity of the target image, and is described by taking the first target image and the second target image in the first target image set as an example, including:

[0173] determining a third contour area of ​​a third monitoring target based on the first target image, and determining a fourth contour area of ​​the third monitoring target based on the second target image, wherein the third monitoring target includes a face and chest of the driver;

[0174] respectively acquiring the facial area and the chest area in the third contour region and the fourth contour region;

[0175] When the facial area in the fourth contour area is greater than or equal to the facial area in the third contour area, and the chest area in the fourth contour area is less than or equal to the chest area in the third contour area, determining that the target image is valid; or

[0176] When the facial area in the fourth contour region is smaller than or equal to the facial area in the third contour region, and the chest area in the fourth contour region is larger than or equal to the chest area in the third contour region, the target image is determined to be valid.

[0177] It can be understood that in this embodiment, the movement of the driver's body away from or close to the camera can be regarded as a normal action, which causes the facial area and chest area to increase or decrease synchronously. The chest area change data calculated at this time interferes with the monitoring results. The target image in this scene can be regarded as invalid, and the chest area data will not be output to participate in the subsequent calculation of breathing frequency, breathing depth, movement amplitude parameters, etc.

[0178] In one embodiment, further verification of the data is included, including checking the validity of the chest area data, including:

[0179] The displacement offset of the face is determined according to the coordinates of the facial positioning points in the third contour area and the fourth contour area; and the normal vector offset is determined according to the normal vector of the face in the third contour area and the fourth contour area.

[0180] When the displacement offset of the face is less than the displacement threshold, and the normal vector offset is less than the normal vector threshold, it is determined that the chest area data in the third contour region and the fourth contour region are valid.

[0181] It is understandable that the coordinates of the facial center point relative to the vehicle coordinate system and the facial normal vector can be obtained through calibration and sufficient sample learning. This part serves as a reference value for body and chest detection. When the face is in a relatively stable position, the detection data of the change in the size of the chest area is meaningful, which can further determine the data validity of the first target image and the second target image.

[0182] It should be understood that although Figure 1 、 Figure 2 、 Figure 5 、 Figure 6 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 、 Figure 2 、 Figure 5 、 Figure 6 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0183] In one embodiment, Figure 9As shown, a driver abnormality monitoring device is provided, comprising: an acquisition module 501, an evaluation and calculation module 502, a judgment module 503 and a warning execution module 504, wherein:

[0184] An acquisition module 501 is used to acquire N different driving monitoring parameters of the driver, where N ≥ 2;

[0185] an evaluation calculation module 502, configured to obtain a comprehensive evaluation value based on the N different driving monitoring parameters;

[0186] A judgment module 503 is used to judge whether the driving monitoring parameter is within a parameter abnormal range or whether the comprehensive evaluation value is within a comprehensive abnormal range;

[0187] The warning execution module 504 is used to execute the first warning instruction based on the abnormal driving monitoring parameter when at least one type of the driving monitoring parameter is in the corresponding parameter abnormal range; otherwise, when the comprehensive evaluation value is in the comprehensive abnormal range, execute the second warning instruction.

[0188] The above-mentioned driver abnormality monitoring device obtains multiple driving monitoring parameters of the driver. When a single parameter is within the parameter abnormal range, the warning instruction is directly executed based on the parameter; when all parameters are within the normal range, a comprehensive evaluation is performed to obtain a comprehensive evaluation value, and a health warning is issued based on the comprehensive evaluation value; the driver abnormality monitoring method provided in this application realizes both single parameter judgment and comprehensive judgment process, thereby improving the accuracy of abnormality monitoring.

[0189] In one embodiment, the acquisition module 501 includes an image recognition module, and acquiring the driver's motion amplitude parameter as one of the driving monitoring parameters includes:

[0190] Acquire a first target image set;

[0191] Obtaining a first target image and a second target image based on the first target image set, and determining a first contour region corresponding to a first monitoring target in the first target image and a second contour region of the first monitoring target in the second target image, respectively, wherein the first monitoring target includes at least a partial torso region of the driver;

[0192] Obtaining an overlapping area, a first edge area, and a second edge area according to the first contour area and the second contour area, wherein the first edge area is the area of ​​a non-overlapping area in the first contour area, and the second edge area is the area of ​​a non-overlapping area in the second contour area;

[0193] The displacement change rate of the first monitoring target is obtained according to the overlapping area, the first edge area, and the second edge area, and the motion amplitude parameter is determined according to the displacement change rate.

[0194] In one embodiment, the acquisition module 501 calculates the displacement change rate using the following mathematical expression:

[0195]

[0196] Wherein, ΔS is the displacement change rate, S A is the first edge area, S B is the second edge area, S AB is the overlapping area.

[0197] In one embodiment, the acquisition module 501 determines the coordinates of a positioning point of a second monitoring target according to the target image, including obtaining a first positioning point according to the first target image and obtaining a second positioning point according to the second target image, wherein the second monitoring target includes a hand area of ​​the driver;

[0198] The offset rate of the second monitoring target is determined according to the coordinates of the first positioning point and the second positioning point, so as to determine the motion amplitude parameter according to the offset rate of the second monitoring target and the displacement change rate of the first monitoring target.

[0199] In one embodiment, the acquisition module 501 acquires the facial color level change rate as one of the driving monitoring parameters, including:

[0200] Acquire a second target image set, wherein each image in the second target image set includes at least the driver's face, and acquire a third target image and a fourth target image based on the second target image set;

[0201] According to the third target image and the fourth target image, the change in the number of pixels of the same color level in the target area is counted to obtain a color level change rate, wherein the target area includes a facial area, and the color level change rate includes the facial color level change rate corresponding to the facial area.

[0202] As an implementation of the above embodiment, the following further includes:

[0203] Obtaining a reference color scale change rate corresponding to the reference area by performing statistics based on a change in the number of pixels of the same color scale in the reference area in the third target image and the fourth target image;

[0204] Whether the facial color scale change rate is valid is determined according to the numerical value of the reference color scale change rate, so as to output a valid facial color scale change rate.

[0205] In one embodiment, the color scale change rate is obtained according to the following mathematical expression:

[0206]

[0207] Wherein, f is the rate of change of the color scale of the target area, S is the total number of pixels in the target area, z′(m) is the number of pixels of the mth color scale in the fourth target image, z(m) is the number of pixels of the mth color scale in the third target image, where 0≤m≤255, △t is the sampling time interval between the third target image and the fourth target image, where the color scale of the pixel points is divided into 0-255.

[0208] The specific definitions of the driver anomaly monitoring device can be found in the definitions of the driver anomaly monitoring method above and will not be repeated here. Each module in the aforementioned driver anomaly monitoring device may be implemented in whole or in part through software, hardware, or a combination thereof. Each of these modules may be embedded in or independent of a processor within a computer device in hardware form, or may be stored in a computer device memory in software form, allowing the processor to call and execute the corresponding operations of each module.

[0209] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 10 As shown. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory.

[0210] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0211] Obtain N different driving monitoring parameters of the driver, where N ≥ 2;

[0212] When at least one type of the driving monitoring parameter is within a corresponding parameter abnormal range, executing a first warning instruction based on the abnormal driving monitoring parameter;

[0213] Otherwise, a comprehensive evaluation value is obtained based on the N different driving monitoring parameters, and when the comprehensive evaluation value is within a comprehensive abnormal range, a second early warning instruction is executed.

[0214] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0215] The driver's motion amplitude parameter is obtained as one of the driving monitoring parameters.

[0216] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0217] The facial color scale change rate is obtained as one of the driving monitoring parameters.

[0218] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0219] Obtain N different driving monitoring parameters of the driver, where N ≥ 2;

[0220] When at least one type of the driving monitoring parameter is within a corresponding parameter abnormal range, executing a first warning instruction based on the abnormal driving monitoring parameter;

[0221] Otherwise, a comprehensive evaluation value is obtained based on the N different driving monitoring parameters, and when the comprehensive evaluation value is within a comprehensive abnormal range, a second early warning instruction is executed.

[0222] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0223] The driver's motion amplitude parameter is obtained as one of the driving monitoring parameters.

[0224] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0225] The facial color scale change rate is obtained as one of the driving monitoring parameters.

[0226] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0227] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0228] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A driver abnormality monitoring method, characterized in that: include: Obtain N different driving monitoring parameters of the driver, where N ≥ 2; When at least one type of the driving monitoring parameter is within a corresponding parameter abnormal range, executing a first warning instruction based on the abnormal driving monitoring parameter; otherwise, obtaining a comprehensive evaluation value based on the N different driving monitoring parameters, and executing a second warning instruction when the comprehensive evaluation value is within a comprehensive abnormal range; The obtaining of N different driving monitoring parameters of the driver includes: Acquiring the driver's motion amplitude parameter as one of the driving monitoring parameters includes: Acquire a first target image set; Obtaining a first target image and a second target image based on the first target image set, and determining a first contour region corresponding to a first monitoring target in the first target image and a second contour region of the first monitoring target in the second target image, respectively, wherein the first monitoring target includes at least a partial torso region of the driver; Obtaining an overlapping area, a first edge area, and a second edge area according to the first contour area and the second contour area, wherein the first edge area is the area of ​​a non-overlapping area in the first contour area, and the second edge area is the area of ​​a non-overlapping area in the second contour area; The displacement change rate of the first monitoring target is obtained according to the overlapping area, the first edge area, and the second edge area, and the motion amplitude parameter is determined according to the displacement change rate.

2. The driver abnormality monitoring method according to claim 1, characterized in that: Obtaining the displacement change rate of the first monitoring target according to the overlapping area, the first edge area, and the second edge area includes calculating the displacement change rate using the following mathematical expression: Wherein, ΔS is the displacement change rate, S A is the first edge area, S B is the second edge area, S AB is the overlapping area.

3. The driver abnormality monitoring method according to claim 1, characterized in that: After acquiring the first target image and the second target image according to the first target image set, the method further includes: determining coordinates of a positioning point of a second monitoring target according to the target image, including obtaining a first positioning point according to the first target image and obtaining a second positioning point according to the second target image, wherein the second monitoring target includes a hand area of ​​the driver; The offset rate of the second monitoring target is determined according to the coordinates of the first positioning point and the second positioning point, so as to determine the motion amplitude parameter according to the offset rate of the second monitoring target and the displacement change rate of the first monitoring target.

4. The driver abnormality monitoring method according to claim 3, characterized in that: The step of determining the coordinates of the positioning point of the second monitoring target according to the target image includes: determining a closed contour curve of a second monitoring target according to the target image; Obtaining a binarization matrix according to the target image, wherein the values ​​of the points in the binarization matrix located within the closed contour curve are set to 1, and the values ​​of the points located outside the closed contour curve are set to 0; The coordinates of the positioning point (x c ,y c ): in: Wherein, V(i, j) is the value at point (i, j) in the binarized matrix.

5. The driver abnormality monitoring method according to claim 1, characterized in that: The obtaining of N different driving monitoring parameters of the driver includes: Obtaining a facial color level change rate as one of the driving monitoring parameters includes: Acquire a second target image set, wherein each image in the second target image set includes at least the driver's face, and acquire a third target image and a fourth target image based on the second target image set; According to the third target image and the fourth target image, the change in the number of pixels of the same color level in the target area is counted to obtain a color level change rate, wherein the target area includes a facial area, and the color level change rate includes a facial color level change rate corresponding to the facial area.

6. The driver abnormality monitoring method according to claim 5, characterized in that: The target area further includes a reference area, and the method of performing statistics based on the change in the number of pixels of the same color level in the target area in the third target image and the fourth target image to obtain the color level change rate further includes: Obtaining a reference color scale change rate corresponding to the reference area by performing statistics based on a change in the number of pixels of the same color scale in the reference area in the third target image and the fourth target image; Whether the facial color scale change rate is valid is determined according to the numerical value of the reference color scale change rate, so as to output a valid facial color scale change rate.

7. The driver abnormality monitoring method according to any one of claims 5 to 6, characterized in that: Obtaining a color scale change rate based on a statistical change in the number of pixels of the same color scale in the target area in the third target image and the fourth target image includes obtaining the color scale change rate according to the following mathematical expression: Wherein, f is the rate of change of the color scale of the target area, S is the total number of pixels in the target area, z′(m) is the number of pixels of the mth color scale in the fourth target image, z(m) is the number of pixels of the mth color scale in the third target image, where 0≤m≤255, △t is the sampling time interval between the third target image and the fourth target image, where the color scale of the pixel points is divided into 0-255.

8. A driver abnormality monitoring device, characterized in that: The device comprises: An acquisition module, used to acquire N different driving monitoring parameters of the driver, where N ≥ 2; an evaluation calculation module, configured to obtain a comprehensive evaluation value based on the N different driving monitoring parameters; A judgment module, configured to judge whether the driving monitoring parameter is within a parameter abnormal range or whether the comprehensive evaluation value is within a comprehensive abnormal range; a warning execution module, configured to execute a first warning instruction based on the abnormal driving monitoring parameter when at least one type of the driving monitoring parameter is within a corresponding parameter abnormal range, and otherwise execute a second warning instruction when the comprehensive evaluation value is within a comprehensive abnormal range; The obtaining of N different driving monitoring parameters of the driver includes: Acquiring the driver's motion amplitude parameter as one of the driving monitoring parameters includes: Acquire a first target image set; Obtaining a first target image and a second target image based on the first target image set, and determining a first contour region corresponding to a first monitoring target in the first target image and a second contour region of the first monitoring target in the second target image, respectively, wherein the first monitoring target includes at least a partial torso region of the driver; Obtaining an overlapping area, a first edge area, and a second edge area according to the first contour area and the second contour area, wherein the first edge area is the area of ​​a non-overlapping area in the first contour area, and the second edge area is the area of ​​a non-overlapping area in the second contour area; The displacement change rate of the first monitoring target is obtained according to the overlapping area, the first edge area, and the second edge area, and the motion amplitude parameter is determined according to the displacement change rate.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images