Driver distraction state early warning method and device based on progressive residual error regression network
Through the method based on progressive residual regression network, the driver's face images are acquired and processed in real time, and the problem of insufficient line of sight detection in the prior art is solved, thereby achieving a more efficient and accurate driver distraction state Evaluate.
Patent Information
- Application Number
- CN202411413580.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-11
- Publication Date
- 2025-02-07
AI Technical Summary
The prior art has complexity and high computational burden in driver line of sight detection, which is difficult to ensure real-time performance, and the detection accuracy is insufficient, so it is impossible to effectively identify the driver's distraction state.
Using a method based on progressive residual regression network, the driver's face image is acquired in real time, feature extraction and attention mechanism processing are performed, and the first-stage and second-stage feature maps are generated. Then, by processing the head posture information and line of sight direction in parallel, it is input into a pre-constructed distraction state warning model based on vehicle speed to evaluate whether the driver is in a distraction state.
It improves the accuracy and real-time performance of driver distraction status evaluation, simplifies the detection process, and enhances the ability to quickly identify and intervene in the driver's distraction status.
Smart Images
Figure CN119399739A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of vehicle safety warning, and in particular to a method and device for warning a driver's distracted state based on a progressive residual regression network. Background Art
[0002] Although the current intelligent driving technology has ushered in significant development, humans will still play a key role in vehicle driving for some time in the foreseeable future. Especially when faced with long periods of monotonous, repetitive or high-load road conditions, drivers are prone to fatigue accumulation and distraction. These abnormal driving conditions greatly threaten driving safety. In order to reduce the risk of traffic accidents, accurate monitoring of the driver's status becomes particularly important. Among them, the driver's line of sight is a key indicator that directly reflects his attention allocation. Its detection is of vital importance for timely detection and warning of distraction.
[0003] According to the existing relevant literature, on the one hand, gaze detection needs to crop the partial images of the face area, left / right eye area and mouth area from the front image of the driver captured by the camera, or use advanced facial feature point detection algorithms to locate feature points such as the corners of the eyes and mouth. Then, gaze detection is performed based on the partial images or feature points. This process increases the complexity and computational burden of detection, and it is difficult to effectively ensure the real-time performance of detection. On the other hand, most algorithms directly regress the gaze direction at the output end, and the regression search space is large, which increases the difficulty and complexity of the algorithm and reduces the accuracy of gaze detection. Therefore, it is urgent to explore a more direct gaze detection method that can better balance real-time and accuracy, so as to achieve rapid identification and intervention of driver distraction. Summary of the invention
[0004] One of the purposes of the present invention is to provide a method and device for warning driver distraction based on a progressive residual regression network, which solves the technical problems of cumbersome intermediate processing such as local image cropping or feature point extraction of the original image before sight detection, as well as insufficient detection accuracy. The present invention defines different sight warning areas and duration thresholds according to different vehicle speeds, and evaluates whether the driver is in a distracted state in combination with the driver's head posture and sight direction, effectively improving the accuracy of the driver's distraction state assessment.
[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:
[0006] In a first aspect, the present invention provides a method for warning a driver's distracted state based on a progressive residual regression network, the method comprising the following steps:
[0007] Acquire the facial image of the driver in the vehicle in real time;
[0008] Extracting features from the face image to generate a first-stage feature map, and processing it through an attention mechanism to generate a second-stage feature map;
[0009] The two-stage feature graphs are processed by two parallel threads to process head posture information and sight direction respectively, to generate a first processing result based on the head posture information and a second processing result based on the sight direction;
[0010] The first processing result and the second processing result are input into a pre-built vehicle speed-based distraction state warning model, and if the warning threshold is exceeded, a driver distraction state warning is issued.
[0011] According to the above technical solution, after acquiring the face image, it is necessary to analyze the face image. By analyzing the head posture information and line of sight direction information in the image, multi-layer processing of the image makes the image clearer and can effectively extract the required features.
[0012] Furthermore, the steps of extracting features from the face image to generate a first-stage feature map, and processing the face image through an attention mechanism to generate a second-stage feature map also include:
[0013] Perform feature extraction on the face image based on the residual network ResNet50 to generate a first-stage feature map;
[0014] After the first-stage feature map is processed by maximum pooling, average pooling and multi-layer perception respectively, it is connected in parallel on the channel to obtain the channel weight coefficient of each channel of the first-stage feature map, and the channel weight coefficient is multiplied by the first-stage feature map to obtain the intermediate feature map;
[0015] The intermediate feature map is processed by maximum pooling and average pooling, connected in series on the channel and subjected to a convolution operation, and the spatial weight coefficient of the intermediate feature map in space is obtained. The spatial weight coefficient is multiplied by the intermediate feature map to obtain a two-stage feature map.
[0016] According to the above technical solution, when processing head posture information, it is implemented through a fully connected neural network, and the data related to the head posture information can be directly obtained.
[0017] Furthermore, in the step of processing the two-stage feature map through two parallel threads to process head posture information and line of sight direction respectively, generating a first processing result based on the head posture information and a second processing result based on the line of sight direction, the step of processing the two-stage feature map through head posture information to generate a first processing result based on head information includes: using the two-stage feature map as the input of a fully connected neural network, performing regression processing, and outputting a first processing result based on an angle prediction value of the head posture, wherein the first processing result includes angle prediction values based on the pitch angle, yaw angle and roll angle of the head posture.
[0018] According to the above technical solution, the information processing of the line of sight direction is realized through a progressive residual regression network, which effectively improves the data processing effect of the line of sight direction.
[0019] Furthermore, in the step of processing the two-stage feature map through two parallel threads to process head posture information and line of sight direction respectively, and generating a first processing result based on the head posture information and a second processing result based on the line of sight direction, the step of processing the two-stage feature map through line of sight direction to generate a second processing result based on the line of sight direction includes: performing regression processing on the two-stage feature map through a progressive residual regression network, and generating a second processing result based on the line of sight direction.
[0020] According to the above technical solution, in order to obtain relevant data of the line of sight direction, a progressive residual regression network is introduced to construct the area of the line of sight, thereby achieving accurate positioning.
[0021] Furthermore, after the two-stage feature map is regressed by the progressive residual regression network, the step of generating a second processing result based on the line of sight direction specifically includes:
[0022] After regressing the two-stage feature map through a progressive residual regression network, the interval range of the sight direction is divided into two sub-range areas based on the yaw angle and the pitch angle of the sight direction, wherein the interval range is represented by a rectangular area constructed by the sub-range areas of the two angles;
[0023] The sub-range area of the yaw angle in the line of sight direction and the sub-range area of the pitch angle in the line of sight direction are evenly divided according to the size of δ1, and the pitch angle range is evenly divided according to the size of δ2, to obtain a series of small rectangular block sub-areas;
[0024] Encode the sub-range area of the yaw angle and the sub-range area of the pitch angle in the sight direction, and represent them with M and N respectively;
[0025] After flattening the two-stage feature map through a progressive residual regression network, the fully connected layer and the softmax activation function are sequentially connected to classify and estimate the small rectangular block sub-regions, and generate a grid region encoding where the line of sight is located, that is, a rough position representation [m, n], where the yaw angle is in the sub-region encoded as m, and the pitch angle is in the sub-region encoded as n;
[0026] In the sub-region coded as [m,n], the residual values res_y and res_p of the yaw angle and pitch angle are regressed respectively;
[0027] The second processing result based on the line of sight direction is generated by decoding, wherein the decoding formula is as follows:
[0028]
[0029] Where gaze represents the final sight direction, which is expressed by yaw angle and pitch angle.
[0030] According to the above technical solution, the focal loss function is used to implement classification estimation, and the smoothed mean absolute error loss function is used to implement classification regression processing.
[0031] Furthermore, it is characterized in that when classifying and estimating the small rectangular block sub-region, the focal loss function F is used cls As the classification loss function, in the steps of regressing the residual values res_y and res_p of the yaw angle and pitch angle respectively, the smoothed mean absolute error loss function F is used. reg The regression loss function as the residual value.
[0032] Furthermore, the loss function of the progressive residual regression network is:
[0033]
[0034] res yaw =yaw′-δ1·ag yaw
[0035] res pitch = pitch′-δ2·ag pitch
[0036]
[0037] Among them, yaw′ and pitch′ are the true values of the yaw angle and pitch angle in the line of sight, respectively, and ag yaw ,ag pitch are the true values of the yaw angle and pitch angle in the line of sight, res yaw 、res pitchare the true values of the residual values of the yaw angle and pitch angle in the line of sight direction, They are the forward prediction values, L represents the loss function of the progressive residual regression network, Represents the floor function.
[0038] According to the above technical solution, comparing the processing results with the pre-set warning range helps to improve the warning effect.
[0039] Furthermore, the first processing result and the second processing result are input into a pre-built vehicle speed-based distraction state warning model. If the warning threshold is exceeded, the step of performing a driver distraction state warning comprises:
[0040] Pre-build the sight warning range and duration threshold model based on vehicle speed information, and generate warning levels of different urgency and their corresponding warning signals;
[0041] The first processing result and the second processing result are input into a pre-constructed distraction state warning model based on vehicle speed. If the time that the predicted value of the head posture information based on the first processing result is within the warning range is greater than or equal to the duration threshold or the time that the line of sight direction of the second processing result is within the line of sight warning range is greater than or equal to the duration threshold, a driver distraction state warning is issued.
[0042] In a second aspect, the present invention further proposes a driver distraction state warning device based on a progressive residual regression network, the device is applied to the distraction state warning method as described in the first aspect, and the device comprises:
[0043] An acquisition module, used for acquiring a face image of a driver in a vehicle in real time;
[0044] A first processing module is used to extract features from the face image to generate a first-stage feature map, and to process the face image through an attention mechanism to generate a second-stage feature map;
[0045] A second processing module is used to process the head posture information and the sight line direction of the two-stage feature map through two parallel processing threads, respectively, to generate a first processing result based on the head posture information and a second processing result based on the sight line direction;
[0046] The warning recognition module is used to input the first processing result and the second processing result into a pre-built distraction state warning model based on vehicle speed, and issue a driver distraction state warning if the warning threshold is exceeded.
[0047] In a third aspect, the present invention further provides an electronic device, characterized in that the electronic device comprises:
[0048] At least one processor; and a memory in communication with the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the driver distraction state warning described in the first aspect.
[0049] Compared with the general method system, the method proposed in the present invention is more concise and accurate, which is specifically reflected in:
[0050] (1) The line of sight detection method proposed in the present invention can improve the detection accuracy of the line of sight direction while simplifying the detection process, thereby enhancing the real-time and accuracy of judging the driver's distracted state.
[0051] (2) The line of sight range is divided into sub-regions and encoded, and a progressive residual regression network is designed. The line of sight detection is divided into two stages: coarse positioning and refinement, making the line of sight detection more accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 A schematic diagram of the process flow of the driver distraction state warning method based on the progressive residual regression network proposed by the present invention;
[0053] Figure 2 A schematic diagram of the technical direction of the driver distraction state warning method based on the progressive residual regression network proposed by the present invention;
[0054] Figure 3 A schematic diagram showing the direction of sight in the driver distraction state warning method based on a progressive residual regression network proposed by the invention;
[0055] Figure 4 A schematic diagram of the grid division of the sight area and the residual value regression in the driver distraction state warning method based on the progressive residual regression network proposed by the invention;
[0056] Figure 5 A schematic diagram of the driver distraction state determination process in the driver distraction state warning method based on the progressive residual regression network proposed in the invention;
[0057] Figure 6 This is a framework diagram of the driver distraction state warning device based on the progressive residual regression network proposed in the invention. DETAILED DESCRIPTION
[0058] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.
[0059] According to the purpose of the present invention, this embodiment provides a driver distraction state warning method based on a progressive residual regression network, such as Figure 1 As shown, the distraction state warning method includes the following steps:
[0060] S101. Real-time acquisition of the driver's facial image in the vehicle;
[0061] S102. Extracting features from the face image to generate a first-stage feature map, and processing the face image with an attention mechanism to generate a second-stage feature map;
[0062] S103. Processing the head posture information and the sight line direction of the two-stage feature graphs through two parallel processing threads respectively, generating a first processing result based on the head posture information and a second processing result based on the sight line direction;
[0063] S104: Input the first processing result and the second processing result into a pre-built vehicle speed-based distraction state warning model, and if the warning threshold is exceeded, issue a driver distraction state warning.
[0064] In step S101, in order to obtain the facial image of the driver in the vehicle, the original image of the driver's head or upper body can be obtained from the front of the driver through a camera. In practice, other image acquisition devices, such as infrared imaging devices, can also be used to acquire images. The purpose of this embodiment is to acquire images, so the specific acquisition method is not specifically limited.
[0065] In step S102, it is considered that the residual network deepens the feature extraction dimension and can effectively weaken the problems such as gradient explosion / disappearance caused by the increase in network depth. The present invention uses the residual network ResNet50 (Residual Network) to extract features from the original image and obtain a first-stage feature map. On the basis of the first-stage feature map, an attention mechanism is introduced to process and generate a second-stage feature map. Specifically, the attention mechanism of this embodiment adopts a convolutional block attention module (CBAM) to highlight the feature weights of areas with a greater effect on line of sight detection (eyes, mouth, face, etc.), weaken some irrelevant area features, and obtain a second-stage feature map.
[0066] In this embodiment, the first-stage feature map is processed by maximum pooling and average pooling respectively, and then connected in parallel on the channel through a multi-layer perception module to obtain the weight coefficient of each channel of the first-stage feature map. The channel weight coefficient is multiplied by the first-stage feature map to obtain an intermediate feature map; the intermediate feature map is then processed by maximum pooling and average pooling, and then connected in series on the channel, and then subjected to a convolution operation to obtain the weight coefficient of the intermediate feature map in space. The spatial weight coefficient is multiplied by the intermediate feature map to obtain a second-stage feature map.
[0067] like Figure 2 As shown, in this embodiment, the two-stage feature map obtained from the above content is divided into two parallel threads for separate processing. One thread enters the head posture information processing link, and the other thread enters the line of sight direction processing link, thereby generating processing results based on head posture information and based on line of sight direction respectively, and then analyzing and comparing the processing results with the warning threshold, so as to accurately determine whether a warning occurs.
[0068] In this embodiment, after the two-stage feature map enters the head posture information processing link through the first thread, the driver's head posture information is obtained (the head posture is represented by pitch angle (pitch), yaw angle (yaw) and roll angle (roll)). Specifically, the two-stage feature map is used as the input of the fully connected neural network, and after regression processing, the first processing result based on the angle prediction value of the head posture is output, wherein the first processing result includes the angle prediction value based on the pitch angle, yaw angle and roll angle of the head posture.
[0069] In this embodiment, after the second-stage feature map enters the line of sight direction processing link through the second thread, the driver's line of sight direction is obtained. It should be noted that in this processing process, the second-stage feature map is regressed through the progressive residual regression network to achieve the regression of the line of sight direction and generate a second processing result based on the line of sight direction. Figure 3The driver's line of sight direction can be represented by two angles: yaw and pitch.
[0070] After the two-stage feature map is regressed by the progressive residual regression network, the sight direction value has a certain range, and the interval range of the sight direction is divided into two sub-range areas based on the yaw angle and pitch angle of the sight direction, such as Figure 4 As shown, the interval range is represented by a rectangular area constructed by sub-range areas of two angles. The sub-range area of the yaw angle in the line of sight and the sub-range area of the pitch angle in the line of sight are evenly divided according to the size of δ1, and the pitch angle range is evenly divided according to the size of δ2, to obtain a series of small rectangular block sub-areas. It should be noted that the present embodiment provides a uniform division into rectangular grids. In practice, the probability of the driver's line of sight appearing in the area with a larger probability (such as the schematic diagram) can also be counted. Figure 4 The gray elliptical area (shown in the figure) divides the mesh into denser areas.
[0071] The sub-range area of the yaw angle and the sub-range area of the pitch angle in the sight direction are encoded in the form of [0, 1, ..., X], respectively. The grid area in the sight direction is represented by the form of [M, N], and M and N are used to represent the encoding of the yaw angle and the pitch angle, respectively.
[0072] After flattening the two-stage feature map through the progressive residual regression network, the fully connected layer and the softmax activation function are connected in sequence, and the sub-regions where the yaw angle and the pitch angle are located are classified and estimated respectively, and the grid region code where the line of sight direction is located is generated. Since the sub-region code is an integer, the minimum scales of the yaw angle and the pitch angle are δ1 and δ2 respectively. Therefore, the line of sight direction is a rough value, that is, a rough position representation [m, n], and more precise positioning is required within the minimum scale, where the yaw angle is in the sub-region encoded as m, and the pitch angle is in the sub-region encoded as n.
[0073] Because the sight direction can only exist in one sub-region, its positive and negative samples are unbalanced. In order to enhance the performance of classification estimation, the focal-loss function is used as the classification loss function of this module, and its calculation formula is as follows:
[0074]
[0075] Among them, F cls represents the classification loss, γ=2,P t is the predicted probability. If the sample label value y is a positive sample, P t Represents the probability P that the prediction model is a positive class. Otherwise, P t It means the probability that the prediction model is a negative class is 1-P.
[0076] In the sub-region coded as [m, n] in this embodiment, the residual values res_y and res_p of the yaw angle and the pitch angle are regressed respectively, and the regression loss function of the residual value uses the smooth mean absolute error loss function (Smooth L1-loss), and its calculation formula is as follows:
[0077]
[0078] Among them, x and y represent the predicted value and the true value respectively.
[0079] In this embodiment, in order to obtain network weight parameters with good performance, the loss function of the progressive residual regression network is set to:
[0080]
[0081] res yaw =yaw′-δ1·ag yaw
[0082] res pitch = pitch′-δ2·ag pitch
[0083]
[0084] Among them, yaw′ and pitch′ are the true values of the yaw angle and pitch angle in the line of sight, respectively, and ag yaw 、ag pitch are the true values of the yaw angle and pitch angle in the line of sight, res yaw 、res pitch are the true values of the residual values of the yaw angle and pitch angle in the line of sight direction, They are the forward prediction values, L represents the loss function of the progressive residual regression network, Represents the floor function.
[0085] Because the yaw angle and pitch angle are encoded in the above content, they need to be decoded to output the accurate sight direction, and the second processing result based on the sight direction is generated by decoding, wherein the decoding formula is as follows:
[0086]
[0087] Where gaze represents the final sight direction, which is expressed by yaw angle and pitch angle.
[0088] The above content clearly and in detail describes the results of processing the head posture information and the line of sight direction information. In this embodiment, the collected data is analyzed by building an early warning model to determine whether there is an early warning, such as Figure 5 As shown, specifically including:
[0089] Pre - construct a line - of - sight warning range and duration threshold model based on vehicle speed information, and generate warning levels of different emergency degrees and their corresponding warning signals;
[0090] Input the first processing result and the second processing result into a pre - constructed distracted - state warning model based on vehicle speed. If the time that the line - of - sight direction based on the second processing result is within the line - of - sight warning range is greater than or equal to the duration threshold, or the time that the predicted value of the head pose based on the first processing result is within the warning range is greater than or equal to the duration threshold, then a warning for the driver's distracted state is given.
[0091] In this embodiment, in the distracted - state warning model of vehicle speed, vehicle speed is combined with the warning range for analysis. The higher the vehicle speed, the lower the tolerance for judging driver distraction, and a more stringent warning range is required. For example, when the vehicle speed is V1, set the line - of - sight warning area as R1, the head warning area as S1, and the duration threshold as T1. When the vehicle speed is V2, set the line - of - sight warning area as R2, the head warning area as S2, and the duration threshold as T2. The logic for threshold setting is that if V1 < V2, then R2 > R1, S2 > S1, and T1 > T2. The warning judgment basis is: when the vehicle speed is V1, if the duration T that the line - of - sight direction G is within the area R1 is greater than or equal to the duration threshold T1, or the duration T that the head pose A is within the area S1 is greater than or equal to the duration threshold T1, the system issues a distraction alarm.
[0092] It should be noted that the distracted - state warning model can be based on the threshold range obtained after analyzing multiple types of collected data, and thus be used as a standard to compare and analyze the data extracted from the face images collected during implementation, and combine vehicle speed information to classify the levels of warning signals, thereby fully considering the impact of the current vehicle speed on driving safety. By defining different line - of - sight warning areas and duration thresholds according to different vehicle speeds, and combining the driver's head posture and line - of - sight direction, it is possible to evaluate whether the driver is in a distracted state, effectively improving the accuracy of evaluating the driver's distracted state.
[0093] This embodiment provides a driver distraction state warning method based on a progressive residual regression network, which uses a neural network to extract image features to obtain a first-stage feature map; then, based on the attention mechanism, the key features of the sight line detection task are highlighted to obtain a second-stage feature map; then, the head posture information processing and the sight line direction information processing are performed separately, and the head posture is parsed in the head posture information processing; in the sight line direction information processing, the sight line area is gridded and encoded, and the progressive residual regression network is used to achieve accurate positioning of the sight line direction. Finally, a driver distraction state evaluation method is constructed by comprehensively considering multi-dimensional data such as vehicle speed, head posture and sight line direction.
[0094] According to the purpose of the present invention, the present invention also proposes a driver distraction state warning device based on a progressive residual regression network, and the device is applied to the distraction state warning method, such as Figure 6 As shown, the device 60 comprises:
[0095] An acquisition module 61 is used to acquire a face image of a driver in a vehicle in real time;
[0096] A first processing module 62 is used to extract features from the face image to generate a first-stage feature map, and to process the face image through an attention mechanism to generate a second-stage feature map;
[0097] The second processing module 63 is used to process the head posture information and the sight line direction of the two-stage feature graphs through two parallel processing threads, respectively, to generate a first processing result based on the head posture information and a second processing result based on the sight line direction;
[0098] The warning recognition module 64 is used to input the first processing result and the second processing result into a pre-built vehicle speed-based distraction state warning model, and issue a driver distraction state warning if the vehicle speed exceeds the warning threshold.
[0099] In a third aspect, the present invention further provides an electronic device, characterized in that the electronic device comprises:
[0100] At least one processor; and a memory in communication with the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the steps of the distraction state warning method.
[0101] The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present invention is within the protection scope of the present invention.
Claims
1. A driver distraction state warning method based on a progressive residual regression network, characterized in that: The method comprises the following steps: acquiring a facial image of a driver in a vehicle in real time; The facial image is subjected to feature extraction to generate a first-stage feature map, and is subjected to attention mechanism processing to generate a second-stage feature map; the second-stage feature map is subjected to head posture information processing and sight direction processing respectively by two parallel processing threads to generate a first processing result based on the head posture information and a second processing result based on the sight direction; The first processing result and the second processing result are input into a pre-built vehicle speed-based distraction state warning model, and if the warning threshold is exceeded, a driver distraction state warning is issued.
2. The driver distraction state warning method based on progressive residual regression network according to claim 1 is characterized in that: The steps of extracting features from the face image to generate a first-stage feature map, and processing the face image through an attention mechanism to generate a second-stage feature map also include: Perform feature extraction on the face image based on the residual network ResNet50 to generate a first-stage feature map; After the first-stage feature map is processed by maximum pooling, average pooling and multi-layer perception respectively, it is connected in parallel on the channel to obtain the channel weight coefficient of each channel of the first-stage feature map, and the channel weight coefficient is multiplied by the first-stage feature map to obtain the intermediate feature map; The intermediate feature map is processed by maximum pooling and average pooling, connected in series on the channel and subjected to a convolution operation, and the spatial weight coefficient of the intermediate feature map in space is obtained. The spatial weight coefficient is multiplied by the intermediate feature map to obtain a two-stage feature map.
3. The driver distraction state warning method based on progressive residual regression network according to claim 2 is characterized in that: In the step of processing the two-stage feature map through two parallel threads to process the head posture information and the line of sight direction respectively, and generating a first processing result based on the head posture information and a second processing result based on the line of sight direction, the step of processing the two-stage feature map through the head posture information to generate the first processing result based on the head information includes: using the two-stage feature map as the input of a fully connected neural network, performing regression processing, and outputting a first processing result based on the angle prediction value of the head posture, wherein the first processing result includes angle prediction values based on the pitch angle, yaw angle and roll angle of the head posture.
4. The driver distraction state warning method based on progressive residual regression network according to claim 3 is characterized in that: In the step of processing the two-stage feature map through two parallel threads to process the head posture information and the line of sight direction respectively, and generating a first processing result based on the head posture information and a second processing result based on the line of sight direction, the step of processing the two-stage feature map through the line of sight direction to generate the second processing result based on the line of sight direction includes: regressing the two-stage feature map through a progressive residual regression network to generate a second processing result based on the line of sight direction.
5. The driver distraction state warning method based on progressive residual regression network according to claim 4 is characterized in that: After the two-stage feature map is regressed by the progressive residual regression network, the step of generating a second processing result based on the line of sight direction specifically includes: After regressing the two-stage feature map through a progressive residual regression network, the interval range of the sight direction is divided into two sub-range areas based on the yaw angle and the pitch angle of the sight direction, wherein the interval range is represented by a rectangular area constructed by the sub-range areas of the two angles; The sub-range area of the yaw angle in the line of sight direction and the sub-range area of the pitch angle in the line of sight direction are evenly divided according to the size of δ1, and the pitch angle range is evenly divided according to the size of δ2, to obtain a series of small rectangular block sub-areas; The sub-range area of the yaw angle and the sub-range area of the pitch angle in the sight direction are encoded and represented by M and N respectively; after flattening the two-stage feature map through a progressive residual regression network, the fully connected layer and the softmax activation function are sequentially connected to classify and estimate the small rectangular block sub-areas, and a grid area encoding of the sight direction is generated, that is, a rough position representation [m, n], wherein the yaw angle is in the sub-area encoded as m, and the pitch angle is in the sub-area encoded as n; In the sub-region coded as [m,n], the residual values res_y and res_p of the yaw angle and pitch angle are regressed respectively; The second processing result based on the line of sight direction is generated by decoding, wherein the decoding formula is as follows: Where gaze represents the final sight direction, which is expressed by yaw angle and pitch angle.
6. According to claim 5, the method for warning driver distraction based on progressive residual regression network is characterized in that: When classifying and estimating small rectangular block sub-regions, the focal loss function F is used cls As the classification loss function, in the steps of regressing the residual values res_y and res_p of the yaw angle and pitch angle respectively, the smoothed mean absolute error loss function F is used. reg The regression loss function as the residual value.
7. The method for warning driver distraction based on progressive residual regression network according to claim 6, characterized in that: The loss function of the progressive residual regression network is: <h2 style=";text-align:left;direction:ltr">res<h2 style=";text-align:left;direction:ltr"> yaw <h2 style=";text-align:left;direction:ltr"> =yaw′-δ1·ag<h2 style=";text-align:left;direction:ltr"> yaw res pitch =pitch′-δ2·ag pitch Among them, yaw′ and pitch′ are the true values of the yaw angle and pitch angle in the line of sight, respectively, and ag yaw 、ag pitch are the true values of the yaw angle and pitch angle in the line of sight, res yaw 、res pitch are the true values of the residual values of the yaw angle and pitch angle in the line of sight direction, They are the forward prediction values, L represents the loss function of the progressive residual regression network, Represents the floor function.
8. According to claim 7, the method for warning driver distraction based on progressive residual regression network is characterized in that: The first processing result and the second processing result are input into a pre-built vehicle speed-based distraction state warning model. If the vehicle speed exceeds the warning threshold, the step of performing a driver distraction state warning comprises: Pre-build the sight warning range and duration threshold model based on vehicle speed information, and generate warning levels of different urgency and their corresponding warning signals; The first processing result and the second processing result are input into a pre-constructed distraction state warning model based on vehicle speed. If the time that the predicted value of the head posture information based on the first processing result is within the warning range is greater than or equal to the duration threshold or the time that the line of sight direction of the second processing result is within the line of sight warning range is greater than or equal to the duration threshold, a driver distraction state warning is issued.
9. A driver distraction warning device based on a progressive residual regression network, characterized in that: The device is applied to the distraction state warning method according to any one of claims 1 to 8, and the device comprises: An acquisition module, used for acquiring a face image of a driver in a vehicle in real time; A first processing module is used to extract features from the face image to generate a first-stage feature map, and to process the face image through an attention mechanism to generate a second-stage feature map; A second processing module is used to process the head posture information and the sight line direction of the two-stage feature map through two parallel processing threads, respectively, to generate a first processing result based on the head posture information and a second processing result based on the sight line direction; The warning recognition module is used to input the first processing result and the second processing result into a pre-built distraction state warning model based on vehicle speed, and issue a driver distraction state warning if the warning threshold is exceeded.
10. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory in communication with the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the driver distraction state warning according to any one of claims 1 to 8.