A video driver fatigue detection method based on deep ensemble network

Abstract

The present invention discloses a video driver fatigue detection method based on a deep integration network, which relates to the field of video recognition technology, including: acquiring a driver's driving image, identifying obstructions during the driver's driving process, and determining the type and obstruction position of the obstruction; determining a feature range of facial feature points based on the obstruction position of the obstruction, determining regional facial feature points in the feature range, establishing a facial model, and marking the regional facial feature points and the obstruction area of the obstruction in the facial model; determining the obstruction feature points of the obstruction, marking the obstruction feature points, and acquiring the change position of the obstruction feature points within a period of time, generating a trajectory diagram of the change of the obstruction feature points, and determining the support for the obstruction according to the trajectory diagram; the present application determines the driver's state during driving by acquiring the driver's facial state changes, acquires the driver's facial data, and determines the facial feature points.

Description

Technical Field

[0001] The present invention relates to the field of video recognition technology, and in particular to a video driver fatigue detection method based on a deep integration network. Background Art

[0002] Fatigue driving refers to a phenomenon in which a driver's driving ability declines due to physical and mental fatigue after prolonged driving. This condition affects the driver's attention, reaction speed, and judgment, increasing the risk of traffic accidents.

[0003] The effects of fatigue driving on drivers include: slow reaction, decreased judgment, blurred vision, operational errors, etc.

[0004] Fatigue driving not only threatens the safety of the driver himself, but also endangers other road users. It is considered one of the main causes of traffic accidents.

[0005] Announcement No. CN112101103A discloses a video driver fatigue detection method based on a deep integration network. The framework includes a style transfer module, a facial key point detection module, and a classification module. The style transfer module consists of a codec generation network, which is used to restore the color information of the input infrared video frame and output a color video frame. The facial key point detection module adopts a fully convolutional neural network structure, taking the infrared video frame and the color video frame generated by the style transfer module as input, locates the facial key points, and outputs a mask feature map. The classification module consists of a 3D convolutional neural network, which integrates the spatiotemporal information of the infrared video sequence, the color information of the color video sequence, and the saliency information of the mask feature sequence to determine the driver's fatigue state. Compared with existing fatigue driving detection algorithms, the present invention has a high detection rate and a low false alarm rate, and can be used for driver fatigue detection under infrared surveillance video. The invention has important application value in the field of intelligent transportation.

[0006] Fatigue driving detection determines whether the driver is driving fatigued based on the driver's state and posture during driving. Since fatigue driving is different from normal driving, whether the driver is driving fatigued is determined by identifying characteristics that are different from normal driving.

[0007] When the driver's face is obscured while driving, the existing fatigue detection method cannot obtain the driver's facial features. When performing fatigue detection, the driver's driving status can only be judged by the driver's local facial features and head posture, which is not accurate enough for detecting fatigue driving. Summary of the Invention

[0008] One of the objectives of the present invention is to provide a video driver fatigue detection method based on a deep integration network, which obtains the changes in the driver's facial features during driving, establishes a facial feature model of the driver, and determines the changes in the driver's obscured facial features based on the changes in the driver's facial muscles, thereby detecting the driver's current driving status.

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions: A video driver fatigue detection method based on a deep integration network, comprising:

[0010] Acquire the driver's driving image, identify the obstructions during the driver's driving process, and determine the type and location of the obstructions;

[0011] Determining a feature range of facial feature points based on an occlusion position of the occluder, determining regional facial feature points within the feature range, establishing a facial model, and marking the regional facial feature points and the occlusion area of the occluder in the facial model;

[0012] Determine the occlusion feature points of the occlusion object, mark the occlusion feature points, obtain the change position of the occlusion feature points over a period of time, generate a trajectory map of the occlusion feature point changes, and determine the support for the occlusion object based on the trajectory map;

[0013] The facial features inside the occluder are determined by the different supported states of the occluder, and the occluded facial feature points are generated inside the occluder of the facial model. Based on the occluded facial feature points and regional facial feature points, a complete facial feature point is constructed, and the state of the facial feature points is identified to analyze the driver's driving state.

[0014] In one or more embodiments of the present invention, multiple facial recognition points are determined, and occlusions during the driver's driving process are determined using the multiple facial recognition points. First, driving images of the driver are obtained from different frames of the driving image. Whether the most recent driving image contains a complete driver's facial image is determined, and facial recognition points are identified on the complete driver's facial image.

[0015] determining whether the driver's facial image contains complete facial recognition points;

[0016] The edge of the occluded object is marked for the driving image with the occluded object, and the edge position of the occluded object is determined based on the driving images of multiple different frames.

[0017] In one or more embodiments of the present invention, the type of the obstruction is determined by the obstruction area and the obstruction position of the obstruction:

[0018] Marking edge positions of occluders in driving images of multiple different frames to determine an occluder contour area, and determining a position of the facial area where the occluder contour area is located;

[0019] The type of occluder is determined by combining the occluder outline area and the position of the facial area.

[0020] In one or more embodiments of the present invention, regional facial feature points within a feature range are determined:

[0021] Generate a facial image, block the facial image using the occluder outline area, and determine the range of facial features that are not blocked;

[0022] Determine the facial feature range and determine whether the facial feature points are included in the facial feature range;

[0023] When the facial feature range includes facial feature points, the facial feature points in the facial feature range are marked as position area facial feature points;

[0024] When the facial feature unit does not contain a facial feature point, the prominent points in the facial feature range are determined as regional facial feature points.

[0025] In one or more embodiments of the present invention, occlusion feature points of an obstruction are determined, the occlusion feature points in the driving image are marked, and the changing positions of the occlusion feature points over a period of time are captured;

[0026] Generate a change trajectory diagram of the occlusion feature points according to the change position of the occlusion feature points, wherein the change trajectory diagram determines the position change of the occlusion feature points relative to the driver's facial coordinate point;

[0027] The changes in the driver's facial muscles are determined based on the position changes of the occluded feature points in the change trajectory graph and the facial feature points in the driver area.

[0028] In one or more embodiments of the present invention, the method for determining the driver's facial state by observing the change of the feature points blocked by the obstruction is as follows:

[0029] First, a trajectory diagram of the occlusion feature points is obtained to determine the degree of fluctuation of the occlusion feature points. A three-dimensional coordinate system is established based on the driver's facial coordinate points. The occlusion feature points are then brought into the three-dimensional coordinate system. The trajectory diagram of the occlusion feature points corresponds to different coordinates of the occlusion feature points.

[0030] Calculate the distance between the facial coordinate point and the occlusion feature point, determine the relationship between the occlusion feature point and the facial coordinate point, and analyze the facial position corresponding to the occlusion feature point;

[0031] The facial state corresponding to the undulating state of the occluded feature point is determined according to the facial position corresponding to the occluded feature point.

[0032] In one or more embodiments of the present invention, a fixed distance from the driver's facial coordinate point is determined as an occlusion feature point, and a change trajectory of the occlusion feature point is determined:

[0033] Establish a projection surface, where there is a gap between the projection surface and the facial coordinate point and the occlusion feature point, and the axis of the coordinate point is perpendicular to the projection surface;

[0034] Determine the distance between the facial coordinate point and the occlusion feature point based on the facial coordinate point and the occlusion feature point in the projection surface;

[0035] Taking the occlusion feature point in the projection surface as the emission point, the position of the occlusion object is determined in the reverse direction along the axis perpendicular to the projection surface;

[0036] The distance between the emission point of the projection surface and the occlusion object is obtained, and the change trajectory of the occlusion feature point is determined to generate a change trajectory map.

[0037] In one or more embodiments of the present invention, a point in the occluder is determined as an occlusion feature point, and a change trajectory of the occlusion feature point is determined:

[0038] Determine the position of the occluded feature point in the occluder and calculate the distance d between the occluded feature point and the facial coordinate point based on the three-dimensional coordinate system:

[0039]

[0040] Among them, (x1, y1, z1) is the coordinate of the occluded feature point, and (0, 0, 0) is the coordinate of the facial coordinate point;

[0041] Determine the angles α, β, and γ between the straight line formed by connecting the occluded feature points and the facial coordinate points and the X-axis, Y-axis, and Z-axis:

[0042]

[0043] Among them, α is the angle between the straight line between the occluded feature point and the facial coordinate point and the X axis, β is the angle between the straight line between the occluded feature point and the facial coordinate point and the Y axis, and γ is the angle between the straight line between the occluded feature point and the facial coordinate point and the Z axis, generating a tilt angle mark (α, β, γ);

[0044] A change trajectory diagram of the occluded feature points is generated according to the angle and distance changes of the straight line between the occluded feature points and the facial coordinate points.

[0045] In one or more embodiments of the present invention, the facial muscle states corresponding to the occlusion feature points are analyzed based on a change trajectory diagram of the occlusion feature points in the occlusion, the facial muscle changes corresponding to the changes in the occlusion feature points over a period of time are obtained, and the facial muscle changes are divided into a normal state and a fatigue state;

[0046] Calculate the fatigue state frequency f in facial muscle changes i :

[0047]

[0048] Where n is the number of times the fatigue state occurs in time period T.

[0049] In one or more embodiments of the present invention, occluded facial feature points in a facial model are constructed based on changes in facial muscles, the positions of the occluded facial feature points are determined, and the occluded facial feature points are combined with regional facial feature points to generate a complete facial model. The changes in the facial feature points in the facial model are used to further examine the degree of relaxation of the driver's facial muscles and determine the driver's driving status.

[0050] Through the above technical solution, the present invention has the following beneficial effects:

[0051] 1. This application obtains the changes in the driver's facial state, determines the driver's state during driving, obtains the driver's facial data, and determines the facial feature points. Based on the facial feature points, a driver's facial model is established. When the driver's facial feature points change, the driver's facial changes can be determined based on the changes in the state of the facial obstruction.

[0052] 2. Establish a facial model based on facial feature points, determine the occlusion feature points of the occlusion object, determine the support of the driver's face for the occlusion object based on the position changes of the occlusion feature points, analyze the driver's facial state, adjust the facial model, and determine the driver's driving state based on the muscle state in the facial model and the position changes of the facial feature points.

[0053] 3. Determine the driver's state corresponding to the movement range of the driver's facial feature points through the position of unobstructed facial feature points. Analyze the driver's driving state based on the movement range of the facial feature points. When the facial muscles move and pull on the driver's face, the facial feature points change. The corresponding facial feature points reflect the changes in the facial muscles and determine the degree of muscle relaxation.

[0054] 4. Analyze the video to determine the changes in the occlusion feature points in the occlusion over a period of time. Determine the facial muscle movement corresponding to the change in the occlusion feature points based on the change state, appearance time and changes in facial feature points over a period of time. Determine the position of the facial feature points within the occlusion of the occlusion, and simulate through the facial model to determine the driver's driving status. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Flow chart of the detection method of the present invention;

[0056] Figure 2 is a schematic diagram of a facial model of the present invention, Figure 2 A in the middle is the unobstructed facial model. Figure 2 Middle B is the facial model image when the lower area of the face is blocked. DETAILED DESCRIPTION

[0057] The following drawings illustrate various embodiments of the present invention. For clarity, many practical details are included in the following description. However, it should be understood that these practical details are not intended to limit the present invention. In other words, in some embodiments of the present invention, these practical details are optional. Furthermore, features from different embodiments may be interchangeably applicable, where practically possible.

[0058] Unless otherwise defined, all words used herein (including technical and scientific terms) have their ordinary meanings as understood by those skilled in the art. Furthermore, the definitions of the above-mentioned words in commonly used dictionaries should be interpreted in the context of this specification as having the same meanings as those in the relevant field of the present invention. Unless otherwise explicitly defined, these words should not be interpreted as having idealized or overly formal meanings.

[0059] See also Figure 1-Figure 2 The present invention provides a video driver fatigue detection method based on a deep integration network, which establishes a facial model through the driver's facial feature points, determines the state of facial muscles blocked by the occluder according to the deformation state of the occluder, and analyzes the driver's driving state.

[0060] In one embodiment, the fatigue detection method includes:

[0061] Acquire the driver's driving image, identify the obstructions during the driver's driving process, and determine the type and location of the obstructions;

[0062] Determining a feature range of facial feature points based on an occlusion position of the occluder, determining regional facial feature points within the feature range, establishing a facial model, and marking the regional facial feature points and the occlusion area of the occluder in the facial model;

[0063] Determine the occlusion feature points of the occlusion object, mark the occlusion feature points, obtain the change position of the occlusion feature points over a period of time, generate a trajectory map of the occlusion feature point changes, and determine the support for the occlusion object based on the trajectory map;

[0064] The facial features inside the occluder are determined by the different supported states of the occluder, and the occluded facial feature points are generated inside the occluder of the facial model. Based on the occluded facial feature points and regional facial feature points, a complete facial feature point is constructed, and the state of the facial feature points is identified to analyze the driver's driving state.

[0065] In this embodiment, when the driver's face is obscured, the recognition of the driver's facial feature points is incomplete. By determining the changing state of the obscured facial feature points through the change of the obstruction, a complete facial feature point can be quickly constructed according to the change of the obstruction to complete the driver's facial features obscured by the obstruction, thereby further ensuring the stability of the recognition of the driver's driving status.

[0066] During the driving recognition process, since the driver's head state is changing, in order to ensure the accuracy of recognition, a coordinate point is established based on the driver's head. This coordinate point is used to locate the driver. Even if the driver's head moves, the recognition stability of the feature point position can be guaranteed.

[0067] By constructing occlusion feature points, changes in occluding objects can be determined. Similarly, for example, for items such as masks, changes in the mask on the driver's face can be determined by constructing occlusion feature points. Changes in the driver's face will support the mask. Therefore, changes in the mask's occlusion feature points correspond to changes in the driver's face.

[0068] In one embodiment, multiple facial recognition points are determined, and occlusions during the driver's driving process are determined using the multiple facial recognition points. First, driving images of the driver in different frames are obtained to determine whether the most recent driving image contains a complete driver's facial image. Facial recognition points are then identified on the complete driver's facial image.

[0069] determining whether the driver's facial image contains complete facial recognition points;

[0070] The edge of the occluded object is marked for the driving image with the occluded object, and the edge position of the occluded object is determined based on the driving images of multiple different frames.

[0071] In this embodiment, multiple facial recognition points are used to determine whether there is an obstruction on the driver's face. Among them, the most common and easiest objects to obstruct the driver's face are masks, glasses, sunglasses and other items. Therefore, by determining multiple facial recognition points to determine whether there is an obstruction on the driver's face, the driver's face can be identified more quickly.

[0072] In one embodiment, the type of the obstruction is determined by the obstruction area and obstruction position of the obstruction:

[0073] Marking edge positions of occluders in driving images of multiple different frames to determine an occluder contour area, and determining a position of the facial area where the occluder contour area is located;

[0074] The type of occluder is determined by combining the occluder outline area and the position of the facial area.

[0075] In this embodiment, the driver's facial area is divided into three areas: upper, middle and lower, which correspond to the driver's forehead, eye and mouth areas respectively. By determining the position of the obstruction contour area, different obstructions can be identified more quickly to facilitate determining the type of obstruction.

[0076] In one embodiment, regional facial feature points in a feature range are determined:

[0077] Generate a facial image, block the facial image using the occluder outline area, and determine the range of facial features that are not blocked;

[0078] Determine the facial feature range and determine whether the facial feature points are included in the facial feature range;

[0079] When the facial feature range includes facial feature points, the facial feature points in the facial feature range are marked as position area facial feature points;

[0080] When the facial feature unit does not contain a facial feature point, the prominent points in the facial feature range are determined as regional facial feature points.

[0081] For example, when the driver has a mask and sunglasses on his face, and the sunglasses are large, causing the driver's eyes and eyebrows to be blocked, and the facial feature range does not include the set facial feature points, the raised positions on both sides of the cheekbones are determined as regional facial feature points.

[0082] In this embodiment, by determining the regional facial feature points within the facial feature range, the driver's facial muscle state can be determined when performing driver facial recognition, that is, by changing the regional facial feature points, the driver's facial muscle changes can be determined in combination with the changes in the state of the obstruction.

[0083] The changes in the driver's facial muscles correspond to the driver's facial state, and the relaxation of the driver's facial muscles corresponding to the normal driving state and the fatigue driving state are inconsistent. Therefore, the driver's driving state can be determined based on the driver's facial muscle state.

[0084] In one embodiment, occlusion feature points of an obstruction are determined, the occlusion feature points in the driving image are marked, and the changing positions of the occlusion feature points over a period of time are captured;

[0085] Generate a change trajectory diagram of the occlusion feature points according to the change position of the occlusion feature points, wherein the change trajectory diagram determines the position change of the occlusion feature points relative to the driver's facial coordinate point;

[0086] The changes in the driver's facial muscles are determined based on the position changes of the occluded feature points in the change trajectory graph and the facial feature points in the driver area.

[0087] In this embodiment, the driver will support the obstruction. When the driver's facial muscles change, the position of the obstruction can be affected, thereby causing the obstruction to change. The change of the obstruction will cause the position of the obstruction feature point relative to the driver's facial coordinate point to change. Since the obstruction is supported by the driver's face, the change of the obstruction feature point on the driver's face corresponds to the change of the driver's facial muscles.

[0088] In one embodiment, the method for determining the driver's facial state by observing the changes in the feature points blocked by the occluder is as follows:

[0089] First, a trajectory diagram of the occlusion feature points is obtained to determine the degree of fluctuation of the occlusion feature points. A three-dimensional coordinate system is established based on the driver's facial coordinate points. The occlusion feature points are then brought into the three-dimensional coordinate system. The trajectory diagram of the occlusion feature points corresponds to different coordinates of the occlusion feature points.

[0090] Calculate the distance between the coordinate point and the occluded feature point, determine the relationship between the occluded feature point and the facial coordinate point, and analyze the facial position corresponding to the occluded feature point;

[0091] The facial state corresponding to the undulating state of the occluded feature point is determined according to the facial position corresponding to the occluded feature point.

[0092] In this embodiment, the facial state corresponds to the change of facial muscles, and the facial muscles support the obstruction, causing the position of the obstruction to change, thereby determining the driver's driving state over a period of time.

[0093] To ensure stability during the detection process, two methods are used to identify the occlusion feature points. The first method is to set a fixed distance from the driver's facial coordinate point as the location of the occlusion feature point and identify the undulation of the occlusion at the location of the occlusion feature point.

[0094] The second method is to determine a certain point in the occlusion as the occlusion feature point. In this case, the distance between the occlusion feature point and the facial coordinate point changes with the change of the driver's facial muscles.

[0095] In one embodiment, a fixed distance from the driver's facial coordinate point is determined as an occlusion feature point, and a change trajectory of the occlusion feature point is determined:

[0096] Establish a projection surface, where there is a gap between the projection surface and the facial coordinate point and the occlusion feature point, and the axis of the coordinate point is perpendicular to the projection surface;

[0097] Determine the distance between the facial coordinate point and the occlusion feature point based on the facial coordinate point and the occlusion feature point in the projection surface;

[0098] Taking the occlusion feature point in the projection surface as the emission point, the position of the occlusion object is determined in the reverse direction along the axis perpendicular to the projection surface;

[0099] The distance between the emission point of the projection surface and the occlusion object is obtained, and the change trajectory of the occlusion feature point is determined to generate a change trajectory map.

[0100] In this embodiment, based on the distance between the projection surface and the obstruction, the support status of the driver at that position for the obstruction can be determined, and the changes in the facial muscles of the obstruction can be determined. When the face is obstructed, the support position and distance of the driver for the obstruction can be analyzed, and the muscle status inside the obstruction can be determined accordingly.

[0101] In one embodiment, a point in the occlusion object is determined as an occlusion feature point, and a change trajectory of the occlusion feature point is determined:

[0102] Determine the position of the occluded feature point in the occluder and calculate the distance d between the occluded feature point and the facial coordinate point based on the three-dimensional coordinate system:

[0103]

[0104] Among them, (x1, y1, z1) is the coordinate of the occluded feature point, and (0, 0, 0) is the coordinate of the facial coordinate point;

[0105] Determine the angles α, β, and γ between the straight line formed by connecting the occluded feature points and the facial coordinate points and the X-axis, Y-axis, and Z-axis:

[0106]

[0107] Among them, α is the angle between the straight line between the occluded feature point and the facial coordinate point and the X axis, β is the angle between the straight line between the occluded feature point and the facial coordinate point and the Y axis, and γ is the angle between the straight line between the occluded feature point and the facial coordinate point and the Z axis, generating a tilt angle mark (α, β, γ);

[0108] A change trajectory diagram of the occluded feature points is generated according to the angle and distance changes of the straight line between the occluded feature points and the facial coordinate points.

[0109] In this embodiment, the changing trajectory of the occlusion feature points corresponds to the degree of change of the occlusion at different positions, and the position changes of the occlusion feature points at different positions correspond to the changes in the face of the driver inside the occlusion.

[0110] The combination of multiple occlusion feature points can reflect the changes in the driver's facial muscles. During the process of driver fatigue driving detection, even if the driver's face is occluded, the changes in the driver's facial muscles can be determined.

[0111] In one embodiment, the facial muscle states corresponding to the occlusion feature points are analyzed based on a change trajectory diagram of the occlusion feature points in the occlusion, the facial muscle changes corresponding to the changes in the occlusion feature points over a period of time are obtained, and the facial muscle changes are divided into a normal state and a fatigue state;

[0112] Calculate the fatigue state frequency f in facial muscle changes i :

[0113]

[0114] Where n is the number of times the fatigue state occurs in time period T.

[0115] In this embodiment, by calculating the frequency of fatigue state, the driving state of the driver in a period of time can be determined, and after analyzing the facial muscle changes, the facial muscle changes are divided into normal state and fatigue state, and the fatigue state can be statistically analyzed.

[0116] In another embodiment, the muscle states are subdivided, for example, blinking, yawning and other muscle states, and the blinking and yawning frequencies are calculated respectively.

[0117] Exemplarily, facial muscle change simulation is performed by the following steps:

[0118] 1. Establish facial geometry model

[0119] First, a detailed geometric model of the face needs to be built, including the skin layer, muscle layer, and bone structure. These layers can be created using 3D scanning technology or manual modeling.

[0120] 2. Define the muscle model

[0121] Use geometric or physical models to define facial muscles. Common methods include:

[0122] Mass-spring model: The muscle is regarded as a spring, and the movement of the muscle is achieved by simulating the contraction and expansion of the spring.

[0123] Finite Element Analysis: Use the finite element method to simulate the physical properties and movement of muscles.

[0124] 3. Bind muscles and skin

[0125] Bind the muscle model to the facial geometry so that the movement of the muscles can affect the deformation of the skin. This can be achieved using skeletal animation technology.

[0126] 4. Simulate muscle movement

[0127] By controlling the contraction and expansion of muscles, you can simulate changes in facial expressions. This can be achieved using keyframe animation or physics simulation.

[0128] In one embodiment, occluded facial feature points in a facial model are constructed based on changes in facial muscles, the positions of the occluded facial feature points are determined, and the occluded facial feature points are combined with regional facial feature points to generate a complete facial model. The changes in the facial feature points in the facial model are used to further examine the degree of relaxation of the driver's facial muscles and determine the driver's driving status.

[0129] In this embodiment, the position of the occluded facial feature points is determined by the change of the occluded feature points, and a complete facial feature point change state is generated. The occlusion of the driver's face by the occluded object is lost, and the driver's facial muscle state can be detected more accurately.

[0130] In summary, the technical solutions disclosed in the above embodiments of the present invention have at least the following advantages:

[0131] 1. This application obtains the changes in the driver's facial state, determines the driver's state during driving, obtains the driver's facial data, and determines the facial feature points. Based on the facial feature points, a driver's facial model is established. When the driver's facial feature points change, the driver's facial changes can be determined based on the changes in the state of the facial obstruction.

[0132] 2. Establish a facial model based on facial feature points, determine the occlusion feature points of the occlusion object, determine the support of the driver's face for the occlusion object based on the position changes of the occlusion feature points, analyze the driver's facial state, adjust the facial model, and determine the driver's driving state based on the muscle state in the facial model and the position changes of the facial feature points.

[0133] 3. Determine the driver's state corresponding to the movement range of the driver's facial feature points through the position of unobstructed facial feature points. Analyze the driver's driving state based on the movement range of the facial feature points. When the facial muscles move and pull on the driver's face, the facial feature points change. The corresponding facial feature points reflect the changes in the facial muscles and determine the degree of muscle relaxation.

[0134] 4. Analyze the video to determine the changes in the occlusion feature points in the occlusion over a period of time. Determine the facial muscle movement corresponding to the change in the occlusion feature points based on the change state, appearance time and changes in facial feature points over a period of time. Determine the position of the facial feature points within the occlusion of the occlusion, and simulate through the facial model to determine the driver's driving status.

[0135] Although the present invention is disclosed in conjunction with the above embodiments, it is not intended to limit the present invention. Anyone familiar with this technology can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be based on the definition of the attached claims.

Claims

1. A video driver fatigue detection method based on deep integration network, characterized in that: include: Acquire the driver's driving image, identify the obstructions during the driver's driving process, and determine the type and location of the obstructions; Determining a feature range of facial feature points based on an occlusion position of the occluder, determining regional facial feature points within the feature range, establishing a facial model, and marking the regional facial feature points and the occlusion area of the occluder in the facial model; Determine the occlusion feature points of the occlusion object, mark the occlusion feature points, obtain the change position of the occlusion feature points over a period of time, generate a trajectory map of the occlusion feature point changes, and determine the support for the occlusion object based on the trajectory map; The facial features inside the occluder are determined by the different supported states of the occluder, and the occluded facial feature points are generated inside the occluder of the facial model. Based on the occluded facial feature points and regional facial feature points, a complete facial feature point is constructed, and the state of the facial feature points is identified to analyze the driver's driving state. The method for determining the driver's facial state by observing the changes in feature points blocked by occluders is as follows: First, a trajectory diagram of the occlusion feature points is obtained to determine the degree of fluctuation of the occlusion feature points. A three-dimensional coordinate system is established based on the driver's facial coordinate points. The occlusion feature points are then brought into the three-dimensional coordinate system. The trajectory diagram of the occlusion feature points corresponds to different coordinates of the occlusion feature points. Calculate the distance between the facial coordinate point and the occluded feature point coordinate, determine the relationship between the occluded feature point and the facial coordinate point, and analyze the facial position corresponding to the occluded feature point; Determining the facial state corresponding to the fluctuation state of the occluded feature point according to the facial position corresponding to the occluded feature point, and determining the change trajectory of the occluded feature point to generate a change trajectory graph includes: Determine a point in the occlusion as an occlusion feature point, and determine the change trajectory of the occlusion feature point: Generate a change trajectory diagram of the occluded feature points according to the angle and distance changes of the straight line between the occluded feature points and the facial coordinate points; Alternatively, a point at a fixed distance from the driver's facial coordinate point is determined as an occlusion feature point, and the change trajectory of the occlusion feature point is determined: Establish a projection surface, the facial coordinate points and the occlusion feature points do not belong to the projection surface, and project them vertically onto the projection surface; Determining the distance between the facial coordinate point and the occlusion feature point based on the point where the facial coordinate point in the projection plane is projected onto the projection plane and the point where the occlusion feature point is projected onto the projection plane; The point where the occlusion feature point in the projection plane is projected on the projection plane is used as the emission point, and the position of the occlusion object is determined in the reverse direction along the axis of the occlusion feature point perpendicular to the projection plane; The distance between the emission point of the projection surface and the occlusion object is obtained, and the change trajectory of the occlusion feature point is determined to generate a change trajectory map.

2. The method for detecting driver fatigue via video based on a deep integration network according to claim 1, characterized in that: Determine multiple facial recognition points, and use the multiple facial recognition points to determine the occlusions during the driver's driving process. First, obtain driving images of different frames in the driver's driving image, identify whether the most recent driving image contains a complete driver's facial image, and perform facial recognition point recognition on the complete driver's facial image; determining whether the driver's facial image contains complete facial recognition points; The edge of the occluded object is marked for the driving image with the occluded object, and the edge position of the occluded object is determined based on the driving images of multiple different frames.

3. The method for detecting driver fatigue via video based on a deep integration network according to claim 2, characterized in that: Determine the type of occluder by its occlusion area and occlusion position: Marking edge positions of occluders in driving images of multiple different frames to determine an occluder contour area, and determining a position of the facial area where the occluder contour area is located; The type of occluder is determined by combining the occluder outline area and the position of the facial area.

4. The method for detecting driver fatigue via video based on a deep integration network according to claim 3, characterized in that: Determine regional facial landmarks within the feature range: Generate a facial image, block the facial image using the occluder outline area, and determine the range of facial features that are not blocked; Determine the facial feature range and determine whether the facial feature points are included in the facial feature range; When the facial feature range includes facial feature points, the facial feature points in the facial feature range are marked as regional facial feature points; When the facial feature range does not include facial feature points, prominent points in the facial feature range are determined as regional facial feature points.

5. The method for detecting driver fatigue via video based on a deep integration network according to claim 4, characterized in that: Determine the occlusion feature points of the obstruction, mark the occlusion feature points in the driving image, and obtain the changing position of the occlusion feature points over a period of time; Generating a change trajectory diagram of the occlusion feature points according to the change position of the occlusion feature points, so as to determine the position change of the occlusion feature points relative to the driver's facial coordinate points; The changes in the driver's facial muscles are determined based on the position changes of the occluded feature points in the change trajectory graph and the facial feature points in the driver area.

6. The method for detecting driver fatigue via video based on a deep integration network according to claim 1, characterized in that: The steps to generate the change trajectory of the occlusion feature points based on the angle and distance changes of the straight line between the occlusion feature points and the facial coordinate points are as follows: Determine the position of the occluded feature point in the occluder and calculate the distance d between the occluded feature point and the facial coordinate point based on the three-dimensional coordinate system: Among them, (x1, y1, z1) is the coordinate of the occluded feature point, and (0, 0, 0) is the coordinate of the facial coordinate point; Determine the angles α, β, and γ between the straight line formed by connecting the occluded feature points and the facial coordinate points and the X-axis, Y-axis, and Z-axis: Generates tilt angle markers (α, β, γ).

7. The method for detecting driver fatigue via video based on a deep integration network according to claim 5, characterized in that: Analyze the facial muscle states corresponding to the occlusion feature points based on the change trajectory of the occlusion feature points in the occlusion, obtain the facial muscle changes corresponding to the changes in the occlusion feature points over a period of time, and divide the facial muscle changes into normal state and fatigue state; Calculate the fatigue state frequency f in facial muscle changes i : Where n is the number of times the fatigue state occurs in time period T.

8. The method for detecting driver fatigue via video based on a deep integration network according to claim 5, characterized in that: Based on the changes in facial muscles, the occluded facial feature points in the facial model are constructed, the positions of the occluded facial feature points are determined, and the occluded facial feature points are combined with the regional facial feature points to generate a complete facial model. The changes in the facial feature points in the facial model are used to further detect the degree of relaxation of the driver's facial muscles and determine the driver's driving status.

Images