Distraction detection method, electronic device and computer-readable storage medium

Abstract

The present application discloses a distraction detection method, an electronic device, and a computer-readable storage medium, the method comprising: obtaining key point information of a face region on a current frame, determining angle detection information of the face region based on the key point information; converting the key point information into a three-dimensional space based on the angle detection information to obtain three-dimensional coordinates of the key points, and determining sight line detection results based on the three-dimensional coordinates of the key points; determining the height deviation between the three-dimensional space and the two-dimensional plane of the face region based on the three-dimensional coordinates of the key points and the key point information; obtaining a distraction detection result based on the sight line detection result, the height deviation, and the angle detection information corresponding to some historical frames before the current frame. In the above manner, the present application can improve the accuracy of distraction detection.

Description

Technical Field

[0001] The present application relates to the field of safe driving technology, and in particular to a distraction detection method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the popularization of vehicles and the development of vehicle intelligence, how to ensure the safety of drivers and passengers has received more and more attention. Among them, distracted driving is a major safety hazard. Conventional distraction detection methods are easily affected by environmental factors and may miss or misdetect, resulting in a low accuracy rate of distraction detection. In view of this, how to improve the accuracy of distraction detection has become an urgent problem to be solved. Summary of the invention

[0003] The main technical problem solved by the present application is to provide a distraction detection method, an electronic device and a computer-readable storage medium, which can improve the accuracy of distraction detection.

[0004] To solve the above technical problems, the first aspect of the present application provides a distraction detection method, including: obtaining key point information of a face area on a current frame, and determining angle detection information of the face area based on the key point information; converting the key point information into a three-dimensional space based on the angle detection information to obtain three-dimensional coordinates of the key points, and determining a line of sight detection result based on the three-dimensional coordinates of the key points; determining a height deviation between the three-dimensional space and the two-dimensional plane of the face area based on the three-dimensional coordinates of the key points and the key point information; and obtaining a distraction detection result based on the line of sight detection result, the height deviation, and the angle detection information corresponding to some historical frames before the current frame.

[0005] To solve the above technical problems, the second aspect of the present application provides an electronic device, which includes: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method described in the first aspect.

[0006] In order to solve the above technical problem, the third aspect of the present application provides a computer-readable storage medium on which program data is stored. When the program data is executed by a processor, the method described in the first aspect is implemented.

[0007] The above scheme obtains the current frame and obtains key point information from the face area of ​​the current frame, detects the angle of the face area based on the key point information, and determines the angle detection information of the face area. Based on the angle detection information, the key point information is converted into a three-dimensional space to obtain the three-dimensional coordinates of the key point information in the three-dimensional space, and the sight line is detected based on the three-dimensional coordinates of the key point to determine the sight line detection result. Based on the three-dimensional coordinates of the key point and the key point information, the height deviation between the estimated height of the face area in the three-dimensional space and the two-dimensional plane is determined. Based on the sight line detection result, the height deviation, and the angle detection information corresponding to some historical frames before the current frame, whether distraction occurs is detected to obtain the distraction detection result, so that when the detection is based on the sight line detection result, the height deviation can be integrated to compensate the detection result, and the angle detection information corresponding to the historical frame can be combined to determine whether the angle detection information changes frequently in a short time, so as to obtain a more accurate distraction detection result by integrating multiple aspects of information, reduce the influence of environmental factors such as height deviation and lighting conditions, and improve the accuracy of distraction detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0009] Figure 1 It is a flow chart of an implementation method of the distraction detection method of the present application;

[0010] Figure 2 It is a flow chart of another embodiment of the distraction detection method of the present application;

[0011] Figure 3 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0012] Figure 4 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments, and different implementation methods can be adaptively combined. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0015] The distraction detection method provided in the present application uses view analysis to detect whether the driver is distracted, and its corresponding execution subject is a terminal that can call and process view data.

[0016] See also Figure 1 , Figure 1 : is a flow chart of an embodiment of a distraction detection method of the present application, the method comprising:

[0017] S101: Acquire key point information of a face region in a current frame, and determine angle detection information of the face region based on the key point information.

[0018] Specifically, a current frame is acquired and key point information is obtained from a face region of the current frame. An angle of the face region is detected based on the key point information to determine angle detection information of the face region.

[0019] In some implementation scenarios, a current frame is obtained, face detection is performed on the current frame using a pre-trained face detection model to obtain a face area on the current frame, key point detection is performed on the face area using a pre-trained key point recognition model to obtain key point information on the face area, the line of sight of the human eye in the face area is estimated based on the key point information, and the angle detection information of the face area is determined.

[0020] In some implementation scenarios, a current frame is obtained, the current frame and key point detection prompts are input into a large language model, and a face region and key point information on the face region output by the large language model are obtained, wherein the large language model is fine-tuned using a sample image including a sample face. A head posture of the face region is estimated based on the key point information, and angle detection information of the face region is determined.

[0021] It should be noted that the key point information at least includes the key points of the face. In some scenes, the key point information also includes the key points of the human eyes marked from the key points of the face. The angle detection information at least includes the pitch angle (Pitch) and the yaw angle (Yaw). Among them, the roll angle (Roll) commonly used in posture estimation has a very small effect on the focus of attention. Therefore, the roll angle can be ignored.

[0022] S102: Based on the angle detection information, the key point information is converted into a three-dimensional space to obtain the three-dimensional coordinates of the key points, and the sight line detection result is determined based on the three-dimensional coordinates of the key points.

[0023] Specifically, the key point information is converted into three-dimensional space based on the angle detection information to obtain the three-dimensional coordinates of the key point information in the three-dimensional space, and the line of sight is detected based on the three-dimensional coordinates of the key point to determine the line of sight detection result.

[0024] In some implementation scenarios, the corresponding vehicle coordinate system in the three-dimensional space is obtained, and the key point information is converted to the vehicle coordinate system in the three-dimensional space based on the angle detection information to obtain the three-dimensional coordinates of the key points of the key point information in the vehicle coordinate system. The angle and landing point of the line of sight are detected based on the three-dimensional coordinates of the key points to determine the line of sight detection result.

[0025] In some implementation scenarios, a rigid body model matching the facial area is obtained, and based on the angle detection information and the rigid body model, a mapping relationship for converting key point information to three-dimensional space is determined. Based on the mapping relationship, the key point information is converted to three-dimensional space to obtain the three-dimensional coordinates of the key points. Based on the three-dimensional coordinates of the key points, the direction of the line of sight in the three-dimensional space is detected to determine the line of sight detection result.

[0026] It is understandable that the head posture or eye state of the face area can be analyzed more accurately in three-dimensional space, thereby obtaining a more accurate line of sight detection result.

[0027] S103: Determine the height deviation between the three-dimensional space and the two-dimensional plane of the face region based on the three-dimensional coordinates of the key points and the key point information.

[0028] Specifically, based on the three-dimensional coordinates of the key points and the key point information, a height deviation between the heights estimated in the three-dimensional space and the two-dimensional plane of the face region is determined.

[0029] In some implementation scenarios, the height of the face area in the three-dimensional space is determined based on the vertical height corresponding to the three-dimensional coordinates of the key points, and the height of the face area on the two-dimensional plane is determined based on the position of the key point information in the current frame, and the height deviation between the estimated heights of the face area in the three-dimensional space and the two-dimensional plane is obtained.

[0030] In some implementation scenarios, the vertical heights corresponding to at least some of the key points in the three-dimensional coordinates of the key points are obtained, the pixel coordinates of the corresponding key points in the current frame are obtained, the plane heights of the corresponding key points on the two-dimensional plane are determined based on the pixel coordinates, and the height deviation between the three-dimensional space and the two-dimensional plane of the face area is determined based on the vertical heights and plane heights corresponding to the corresponding key points.

[0031] It is understandable that the height of the face area corresponds to the driver's height and sitting posture. Since there are errors in estimating the height both on the two-dimensional plane of the image and in the three-dimensional space, the height deviation can feedback the difference between the estimated results in the two dimensions, so as to use the height deviation to correct the detection results, thereby improving the robustness of the detection results.

[0032] S104: Obtain a distraction detection result based on the sight line detection result, the height deviation, and the angle detection information corresponding to some historical frames before the current frame.

[0033] Specifically, based on the sight line detection result, the height deviation, and the angle detection information corresponding to some historical frames before the current frame, whether distraction occurs is detected to obtain the distraction detection result.

[0034] In some implementation scenarios, the line of sight detection result is highly compensated based on the height deviation to correct the line of sight detection result. The confidence of the corrected line of sight detection result of the current frame is determined based on the angle detection information corresponding to some historical frames before the current frame. When the confidence exceeds the confidence threshold, the distraction detection result is obtained based on the corrected line of sight detection result. When the confidence does not exceed the confidence threshold, the distraction detection result is set to not detected.

[0035] In some implementation scenarios, a line of sight angle threshold corresponding to a line of sight detection result is obtained, and the line of sight angle threshold is adjusted based on the height deviation to obtain the adjusted line of sight angle threshold, thereby obtaining a line of sight angle threshold that adaptively changes with the height deviation; based on the angle detection information corresponding to some historical frames before the current frame, the maximum deviation angle between the maximum value and the minimum value in the angle detection information within multiple consecutive frame intervals is determined, and the number of deviations when the maximum deviation angle exceeds the deviation threshold is counted; when the number of deviations does not exceed the number threshold, a distraction detection result is obtained based on the line of sight detection result and the adjusted line of sight angle threshold; when the number of deviations exceeds the number threshold, the distraction detection result is set to not detected.

[0036] It is understandable that when performing detection based on line of sight detection results, the height deviation can be integrated to compensate for the detection results, and the angle detection information corresponding to the historical frame can be combined to determine whether the angle detection information changes frequently in a short period of time, thereby integrating multiple aspects of information to obtain more accurate distraction detection results, reducing the impact of environmental factors such as height deviation and lighting conditions, and improving the accuracy of distraction detection.

[0037] In a specific implementation scenario, the height deviation is obtained based on the difference between the height obtained in the three-dimensional space and the height obtained on the two-dimensional plane. When the absolute value of the height deviation exceeds the deviation threshold, at least one of the line of sight detection result and its corresponding line of sight angle threshold is corrected based on the numerical value of the height deviation. When the absolute value of the height deviation does not exceed the deviation threshold, the current line of sight detection result and its corresponding line of sight angle threshold are kept unchanged.

[0038] In a specific implementation scenario, there is a preset number of historical frames before the current frame. The current frame is used as the starting point to search for the maximum and minimum values ​​in the angle detection information according to the preset frame interval, and the historical frame corresponding to any extreme value of the maximum and minimum values ​​is used as a new starting point. The angle detection information with the largest difference with the new starting point within the preset frame interval is searched, and the maximum deviation angle within the frame interval is determined, thereby continuously searching forward until the preset number of historical frames are traversed.

[0039] The above scheme obtains the current frame and obtains key point information from the face area of ​​the current frame, detects the angle of the face area based on the key point information, and determines the angle detection information of the face area. Based on the angle detection information, the key point information is converted into a three-dimensional space to obtain the three-dimensional coordinates of the key point information in the three-dimensional space, and the sight line is detected based on the three-dimensional coordinates of the key point to determine the sight line detection result. Based on the three-dimensional coordinates of the key point and the key point information, the height deviation between the estimated height of the face area in the three-dimensional space and the two-dimensional plane is determined. Based on the sight line detection result, the height deviation, and the angle detection information corresponding to some historical frames before the current frame, whether distraction occurs is detected to obtain the distraction detection result, so that when the detection is based on the sight line detection result, the height deviation can be integrated to compensate the detection result, and the angle detection information corresponding to the historical frame can be combined to determine whether the angle detection information changes frequently in a short time, so as to obtain a more accurate distraction detection result by integrating multiple aspects of information, reduce the influence of environmental factors such as height deviation and lighting conditions, and improve the accuracy of distraction detection.

[0040] See also Figure 2 , Figure 2 : is a flow chart of another embodiment of the distraction detection method of the present application, the method comprising:

[0041] S201: Acquire key point information of a face region in a current frame, and determine angle detection information of the face region based on the key point information.

[0042] Specifically, a current frame is acquired and key point information is obtained from a face region of the current frame. An angle of the face region is detected based on the key point information to determine angle detection information of the face region.

[0043] In some implementation scenarios, a current frame is obtained and face detection is performed on the current frame to obtain a face area; wherein, when the face area is occluded beyond an area threshold, occlusion warning information is generated; key point detection is performed on the face area to obtain face key points, the eye key points among the face key points are determined, and the eye occlusion coefficients corresponding to the eye key points are obtained; based on the face key points, the eye key points and their corresponding eye occlusion coefficients, the angle detection information of the face area is determined.

[0044] Specifically, the entire detection process is divided into first-order detection, second-order detection and third-order detection, wherein the first-order detection corresponds to the face area, the second-order detection corresponds to the key point information, and the third-order detection corresponds to the angle detection information.

[0045] It can be understood that the current frame is obtained and face detection is performed on the current frame to obtain the face area in the current frame, and it is determined whether the face area is occluded and exceeds the area threshold. If so, an occlusion warning message is generated, otherwise the current frame is retained to perform key point detection on the face area.

[0046] It should be noted that due to the different installation positions of the acquisition equipment, there will be different types of occlusion, including environmental occlusion and self-occlusion. Environmental occlusion includes the occlusion of the face caused by the steering wheel turning or the occlusion caused by the front passenger's body. Self-occlusion includes the occlusion from the driver's own attributes. Occlusion detection can be based on a binary classification model to determine whether there is occlusion, thereby achieving first-order detection.

[0047] Furthermore, key point detection is performed on the face area to obtain face key points, and eye key points are obtained from the face key points to locate the area of ​​interest corresponding to the eye key points. Occlusion detection is performed on the eye key points to determine the eye occlusion coefficient corresponding to the eye key points, thereby achieving accurate detection of the eye area.

[0048] It is understandable that the key points of the face, the key points of the eyes, and the eye occlusion coefficient can all be obtained using a pre-trained model to achieve second-order detection. Among them, the key points of the face can be obtained based on the dlib face library, or based on other open source face libraries, and this application does not make specific restrictions on this.

[0049] Furthermore, the angle of the face area is detected by utilizing the face key points, the eye key points with higher precision on the face key points, and the eye occlusion coefficients corresponding to the face key points, so as to determine the angle detection information of the face area and improve the accuracy of the angle detection information.

[0050] It should be noted that the third-order detection is implemented in two branches, namely, attention detection based on head posture and attention detection based on sight. The specific branch to be used is determined by the human eye occlusion coefficient obtained by the second-order detection. When the human eye is occluded, the head posture can be used as a more reliable detection result that can feedback the sight line. Among them, the results output by the two branches are angle detection information, and the angle detection information includes at least pitch angle and yaw angle.

[0051] S202: Based on the angle detection information, the key point information is converted into a three-dimensional space to obtain the three-dimensional coordinates of the key points, and the sight line detection result is determined based on the three-dimensional coordinates of the key points.

[0052] Specifically, the key point information is converted into three-dimensional space based on the angle detection information to obtain the three-dimensional coordinates of the key point information in the three-dimensional space, and the line of sight is detected based on the three-dimensional coordinates of the key point to determine the line of sight detection result.

[0053] In some implementation scenarios, a rigid body model matching the facial area is obtained, and the three-dimensional coordinates of the key points of the key point information in the three-dimensional space are determined based on the key point information, angle detection information and the rigid body model; in response to the feedback from the human eye occlusion coefficient that the eyes are not occluded, the line of sight angle of the eyes is estimated based on the three-dimensional coordinates of the key points to obtain a line of sight detection result; in response to the feedback from the human eye occlusion coefficient that the eyes are occluded, the posture of the head is estimated based on the three-dimensional coordinates of the key points to obtain the estimated head posture, and the line of sight detection result is determined based on the estimated head posture.

[0054] Specifically, a rigid body model matching the face area is obtained, and the key point information is mapped from a two-dimensional plane to a three-dimensional space using key point information, angle detection information, and the rigid body model to determine the three-dimensional coordinates of the key point information in the three-dimensional space.

[0055] It should be noted that the spatial mapping relationship between two-dimensional and three-dimensional conforms to the pinhole camera model and has the following corresponding homography relationship:

[0056] (1)

[0057] Among them, the internal parameter matrix of the acquisition device is , the external parameter matrix is , and the coordinates corresponding to the key points on the two-dimensional plane are Through the key point information has been obtained, Provided by the rigid body model, the extrinsic parameter matrix can be obtained after obtaining the intrinsic parameter matrix.

[0058] Furthermore, based on the three-dimensional rigid body motion properties, a mapping relationship can be obtained. Using the mapping relationship, all key point information can be converted to the coordinate system in the three-dimensional space, that is, all face key points are converted to the vehicle coordinate system corresponding to the vehicle in the three-dimensional space. The above process is expressed by the following formula:

[0059] (2)

[0060] in, That is the three-dimensional coordinates of the key points in the vehicle coordinate system.

[0061] It can be understood that when the human eye occlusion coefficient feedback shows that the eyes are not occluded, the line of sight angle of the eyes is directly estimated based on the three-dimensional coordinates of the key points to obtain the line of sight detection result, thereby obtaining a more accurate line of sight detection result in the three-dimensional space; when the human eye occlusion coefficient feedback shows that the eyes are occluded, the posture of the head is estimated based on the three-dimensional coordinates of the key points to obtain a more reliable estimated head posture, thereby estimating the line of sight based on the estimated head posture to determine the line of sight detection result, so as to facilitate detection when the eyes are occluded and reduce the probability of missed detection.

[0062] S203: Determine the height deviation between the three-dimensional space and the two-dimensional plane of the face region based on the three-dimensional coordinates of the key points and the key point information.

[0063] Specifically, based on the three-dimensional coordinates of the key points and the key point information, a height deviation between the heights estimated in the three-dimensional space and the two-dimensional plane of the face region is determined.

[0064] In some implementation scenarios, based on the three-dimensional coordinates of the key points, a first estimated height of the face area in the three-dimensional space is determined, and based on the key point information, a second estimated height of the face area in the two-dimensional plane is determined; based on the first estimated height and the second estimated height, the height deviation between the face area in the three-dimensional space and the two-dimensional plane is determined.

[0065] Specifically, the height of the face area in the three-dimensional space is estimated based on the three-dimensional key point coordinates to obtain a first estimated height, and the height of the face area in the two-dimensional plane is estimated based on the key point information to obtain a second estimated height. Based on the first estimated height and the second estimated height, the height difference between the three-dimensional space and the two-dimensional plane of the face area is determined, so that the height difference can feed back the estimated error between different dimensions to facilitate error compensation.

[0066] Optionally, the first estimated height can be obtained by weighted summing of all key point information in the vertical direction, that is, the first estimated height The second estimated height can be obtained based on the position of the facial contour obtained from all key point information in the current frame.

[0067] In a specific implementation scenario, based on key point information, a second estimated height of the face area in a two-dimensional plane is determined, including: based on the key point information, determining the contour height of the face area in a reference pixel plane; based on the contour height and multiple reference images in the reference pixel plane, determining the second estimated height of the face area in the two-dimensional plane; wherein any two reference images include test faces of different reference heights.

[0068] Specifically, based on the key point information, the contour height of the face area on the reference pixel plane corresponding to the current frame is determined, and multiple reference images in the reference pixel plane are obtained, wherein the multiple reference images include test faces of different reference heights, that is, the test subjects corresponding to the test faces in the reference images have different heights, and based on the contour height and the multiple reference images, the height matched by the contour height is estimated, thereby obtaining a more reliable second estimated height on the two-dimensional plane, so as to compensate for the height of the driver in the three-dimensional space.

[0069] It should be noted that the contour height of the face area in the vertical direction of the reference pixel plane is obtained based on the face key points, and the second estimated height can be obtained by using the least squares method based on the reference faces of different reference heights on the reference pixel plane. , thus obtaining the height deviation .

[0070] S204: Based on the height deviation, at least one of the current line of sight detection result and the line of sight angle threshold is adjusted to obtain an adjusted line of sight detection result and line of sight angle threshold.

[0071] Specifically, the line of sight detection result corresponds to a line of sight angle threshold. After obtaining the height deviation, at least one of the current line of sight detection result and the line of sight angle threshold can be adaptively adjusted to improve the accuracy of the distraction detection result.

[0072] It is understandable that in different implementation scenarios, you can choose to adjust only the line of sight detection result, you can choose to adjust only the line of sight angle threshold, or you can choose to adjust both the line of sight detection result and the line of sight angle threshold.

[0073] S205: Obtain a preset number of historical frames before the current frame and their corresponding angle detection information, take the current frame as the starting point, and determine the maximum deviation angles between historical frames within multiple frame intervals in sequence, until the preset number of historical frames are traversed to obtain the number of deviations when the maximum deviation angle exceeds the deviation threshold.

[0074] Specifically, a preset number of historical frames before the current frame and the angle detection information corresponding to the historical frames are obtained. Starting from the current frame, the maximum deviation angle obtained within the corresponding frame interval is determined based on the angle detection information corresponding to the historical frames within multiple frame intervals, and it is determined whether the maximum deviation angle exceeds the deviation threshold, thereby counting the number of deviations until the final number of deviations is obtained after traversing the preset number of historical frames. Therefore, the number of deviations can feedback the number of times the angle detection information before the current frame has a large deviation, thereby determining whether there is a frequent change in sight within a short period of time.

[0075] It should be noted that when the angle detection information includes the pitch angle and the yaw angle, the maximum deviation angles of the pitch angle and the yaw angle need to be calculated respectively, and the number of deviations is accumulated only when the maximum deviation angles of the pitch angle and the yaw angle within the frame interval are both greater than the deviation threshold.

[0076] Optionally, when searching forward from the current frame as the starting point, a fixed frame interval N can be set, so that the search is performed in the order of 1 to N, N+1 to 2N before the current frame, and so on. The starting point can also be re-determined based on the angle extreme value found within the frame interval and then search forward.

[0077] In some implementation scenarios, the maximum deviation angles between historical frames within multiple frame intervals are determined in sequence until a preset number of historical frames are traversed to obtain the number of deviations in which the maximum deviation angle exceeds a deviation threshold, including: obtaining a first angle extreme value and a second angle extreme value in historical frames within a first frame interval before a current frame, and determining the maximum deviation angle within the first frame interval; using the historical frame corresponding to the first angle extreme value as a first marking point, obtaining the second angle extreme value in historical frames within a second frame interval before the first marking point, and determining the maximum deviation angle within the second frame interval; using the historical frame corresponding to the second angle extreme value as a second marking point, obtaining the first angle extreme value in historical frames within a third frame interval before the second marking point, and determining the maximum deviation angle within the third frame interval; returning to the step of using the historical frame corresponding to the first angle extreme value as a first marking point, obtaining the second angle extreme value in historical frames within the second frame interval before the first marking point, and determining the maximum deviation angle within the second frame interval, until a preset number of historical frames are traversed to obtain the number of deviations in which the maximum deviation angle exceeds the deviation threshold.

[0078] Specifically, the first angle extreme value and the second angle extreme value in the historical frames within the first frame number interval before the current frame are obtained, and the maximum deviation angle within the first frame number interval is determined based on the absolute difference between the first angle extreme value and the second angle extreme value, wherein the second angle extreme value is the minimum value when the first angle extreme value is the maximum value, and conversely, the second angle extreme value is the maximum value when the first angle extreme value is the minimum value.

[0079] Furthermore, based on the current first angle extreme value, the historical frame corresponding to the first angle extreme value is obtained as the first marking point, the second angle extreme value is searched in the historical frame within the second frame interval before the first marking point, and based on the absolute difference between the first angle extreme value and the second angle extreme value, the maximum deviation angle within the second frame interval is determined.

[0080] Further, based on the current second angle extreme value, the historical frame corresponding to the second angle extreme value is obtained as the second marking point, the first angle extreme value is searched in the historical frame within the third frame interval before the second marking point, and based on the absolute difference between the first angle extreme value and the second angle extreme value, the maximum deviation angle within the third frame interval is determined.

[0081] It can be understood that after traversing a preset number of historical frames using the second frame interval and the third frame interval, the number of deviations in which the maximum deviation angle exceeds the deviation threshold can be counted, and during the traversal process, the extreme value is used as a new starting point to more accurately count the number of deviations, so that the number of deviations can accurately feedback whether the angle detection information of the historical frames is erratic.

[0082] It should be noted that in different implementation scenarios, the specific values ​​of the first frame interval, the second frame interval and the third frame interval can be customized, and this application does not impose any specific restrictions on this.

[0083] Optionally, at least the first frame number interval is related to the overexposure degree and backlight degree of the current frame. Wherein, before obtaining the key point information of the face area on the current frame and determining the angle detection information of the face area based on the key point information, it also includes: obtaining the current frame, determining the quality score, overexposure score and backlight score of the current frame; wherein the overexposure score feedbacks the overexposure degree of the current frame, and the backlight score feedbacks the backlight degree of the current frame; in response to the quality score exceeding the quality threshold, resetting and saving the overexposure score and backlight score; in response to the quality score not exceeding the quality threshold, and the overexposure score exceeding the overexposure threshold, discarding the current frame and generating overexposure warning information; in response to the quality score not exceeding the quality threshold, and the overexposure score not exceeding the overexposure threshold, saving the overexposure score and backlight score.

[0084] Specifically, the current frame is pre-screened, and after the current frame is obtained, the quality score, overexposure score and backlight score of the current frame are determined, among which the overexposure score can feedback the overexposure degree of the current frame, and the backlight score can feedback the overexposure degree of the current frame, and it is judged whether the quality score exceeds the quality threshold. When the quality score exceeds the quality threshold, the overexposure score and the backlight score are reset and saved.

[0085] Furthermore, when the quality score does not exceed the quality threshold and the overexposure score exceeds the overexposure threshold, the current frame is discarded and overexposure warning information is generated, so that warning information is generated in time when the current frame has insufficient picture information due to overexposure. When the quality score does not exceed the quality threshold and the overexposure score does not exceed the overexposure threshold, the overexposure score and backlight score are saved.

[0086] It is understandable that when the overexposure is high, the pixels on the current frame will be whiter overall, and when the backlight is high, erratic light spots are likely to appear on the current frame. After obtaining the overexposure score and the backlight score, the first frame interval can be adaptively adjusted. When the overexposure score and the backlight score are large, the first frame interval can be increased to obtain the maximum deviation angle in a longer time as much as possible, reducing the impact of unreliable results.

[0087] Optionally, the second frame interval and the third frame interval may also be adaptively adjusted based on the overexposure degree and backlight degree of the current frame, and the present application does not impose any specific restrictions on this.

[0088] S206: Based on the adjusted gaze detection result, the gaze angle threshold, and the number of deviations, a distraction detection result and corresponding distraction warning information are obtained; wherein, when the distraction detection result indicates that distraction exists and the number of deviations exceeds the number threshold, the distraction warning information is ignored.

[0089] Specifically, based on the relationship between the adjusted gaze detection result and the gaze angle threshold, the distraction detection result and its corresponding distraction warning information are determined. Based on whether there are frequent changes in gaze in a short period of time as reflected by the number of deviations, it is determined whether the distraction warning information is accurate, thereby reducing the probability of false detection due to overexposure and backlighting.

[0090] It is understandable that it is determined whether the number of deviations exceeds the number threshold. If it exceeds and the distraction detection result is that distraction exists, the distraction warning information will be ignored, thereby discarding the unreliable result and reducing the probability of false detection.

[0091] In summary, the distraction detection method based on any of the above embodiments can produce more accurate distraction detection results, reduce the impact of environmental factors such as height deviation and lighting conditions, and improve the accuracy of distraction detection.

[0092] See also Figure 3 , Figure 3 It is a structural diagram of an embodiment of an electronic device of the present application, wherein the electronic device 30 includes a memory 301 and a processor 302 coupled to each other, wherein the memory 301 stores program data (not shown), and the processor 302 calls the program data to implement the method in any of the above embodiments. For descriptions of related contents, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0093] See also Figure 4 , Figure 4 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 40 stores program data 400. When the program data 400 is executed by a processor, the method in any of the above embodiments is implemented. For descriptions of related contents, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0094] It should be noted that the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present implementation scheme.

[0095] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0096] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0097] It should be noted that if the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to collect his or her personal information; or on the device for processing personal information, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0098] The above description is only an implementation method of the present application, and does not limit the protection scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the protection scope of the present application.

Claims

1. A distraction detection method, characterized in that: The method comprises: Acquire key point information of a face region on a current frame, and determine angle detection information of the face region based on the key point information; Based on the angle detection information, the key point information is converted into a three-dimensional space to obtain three-dimensional coordinates of the key points, and the sight line detection result is determined based on the three-dimensional coordinates of the key points; Determine the height deviation of the face region between the three-dimensional space and the two-dimensional plane based on the three-dimensional coordinates of the key points and the key point information; A distraction detection result is obtained based on the sight line detection result, the height deviation, and angle detection information corresponding to some historical frames before the current frame.

2. The distraction detection method according to claim 1, characterized in that: The step of acquiring key point information of a face region on a current frame and determining angle detection information of the face region based on the key point information includes: Acquire the current frame and perform face detection on the current frame to obtain the face area; wherein, when the face area is blocked beyond the area threshold, an occlusion warning message is generated; Performing key point detection on the face area to obtain face key points, determining eye key points among the face key points and obtaining eye occlusion coefficients corresponding to the eye key points; Based on the facial key points, the eye key points and their corresponding eye occlusion coefficients, angle detection information of the facial area is determined.

3. The distraction detection method according to claim 2, characterized in that: The determining of the three-dimensional coordinates of the key points of the key point information in the three-dimensional space based on the angle detection information, and determining the sight line detection result based on the three-dimensional coordinates of the key points includes: Acquire a rigid body model matched with the face region, and determine the three-dimensional coordinates of the key points of the key point information in the three-dimensional space based on the key point information, the angle detection information and the rigid body model; In response to the eye occlusion coefficient feedback that the eye is not occluded, estimating the eye sight angle based on the three-dimensional coordinates of the key points to obtain the sight line detection result; In response to the eye occlusion coefficient feedback that the eye is occluded, the posture of the head is estimated based on the three-dimensional coordinates of the key points to obtain the estimated head posture, and the line of sight detection result is determined based on the estimated head posture.

4. The distraction detection method according to claim 1, characterized in that: The determining, based on the three-dimensional coordinates of the key points and the key point information, a height deviation between the three-dimensional space and the two-dimensional plane of the face region includes: Based on the three-dimensional coordinates of the key points, determining a first estimated height of the face area in the three-dimensional space, and based on the key point information, determining a second estimated height of the face area in the two-dimensional plane; Based on the first estimated height and the second estimated height, a height deviation of the face region between a three-dimensional space and a two-dimensional plane is determined.

5. The distraction detection method according to claim 4, characterized in that: The determining, based on the key point information, a second estimated height of the face region in the two-dimensional plane includes: Based on the key point information, determining the contour height of the face area on the reference pixel plane; Based on the contour height and multiple reference images in the reference pixel plane, a second estimated height of the face area in the two-dimensional plane is determined; wherein any two of the reference images include test faces of different reference heights.

6. The distraction detection method according to claim 1, characterized in that: The sight line detection result corresponds to a sight line angle threshold; The step of obtaining a distraction detection result based on the sight line detection result, the height deviation, and the angle detection information corresponding to a portion of historical frames before the current frame includes: Based on the height deviation, at least one of a current sight line detection result and a sight line angle threshold is adjusted to obtain an adjusted sight line detection result and a sight line angle threshold; Obtain a preset number of historical frames before the current frame and their corresponding angle detection information, and take the current frame as the starting point to sequentially determine the maximum deviation angles between historical frames within a plurality of frame intervals until the preset number of historical frames are traversed to obtain the number of deviations at which the maximum deviation angle exceeds the deviation threshold; Based on the adjusted gaze detection result, gaze angle threshold and the number of deviations, the distraction detection result and its corresponding distraction warning information are obtained; wherein, when the distraction detection result indicates that distraction exists and the number of deviations exceeds the number threshold, the distraction warning information is ignored.

7. The distraction detection method according to claim 6, characterized in that: The step of sequentially determining the maximum deviation angles between the historical frames within the plurality of frame number intervals until the preset number of historical frames are traversed to obtain the number of deviations at which the maximum deviation angle exceeds the deviation threshold comprises: Acquire a first angle extreme value and a second angle extreme value in a historical frame within a first frame number interval before a current frame, and determine a maximum deviation angle within the first frame number interval; Taking the historical frame corresponding to the first angle extreme value as the first marking point, obtaining the second angle extreme value in the historical frame within a second frame interval before the first marking point, and determining the maximum deviation angle within the second frame interval; Taking the historical frame corresponding to the second angle extreme value as the second marking point, obtaining the first angle extreme value in the historical frame within a third frame interval before the second marking point, and determining the maximum deviation angle within the third frame interval; Return to the step of taking the historical frame corresponding to the first angle extreme value as the first marking point, obtaining the second angle extreme value in the historical frame within the second frame interval before the first marking point, and determining the maximum deviation angle within the second frame interval, until traversing the preset number of historical frames and obtaining the number of deviations when the maximum deviation angle exceeds the deviation threshold.

8. The distraction detection method according to claim 7, characterized in that: At least the first frame number interval is related to the overexposure degree and the backlight degree of the current frame; Before acquiring key point information of the face area on the current frame and determining angle detection information of the face area based on the key point information, the method further includes: Acquire the current frame, and determine a quality score, an overexposure score, and a backlight score of the current frame; wherein the overexposure score feeds back the overexposure degree of the current frame, and the backlight score feeds back the backlight degree of the current frame; In response to the quality score exceeding a quality threshold, resetting and saving the overexposure score and the backlight score; In response to the quality score not exceeding the quality threshold and the overexposure score exceeding the overexposure threshold, discarding the current frame and generating overexposure warning information; In response to the quality score not exceeding the quality threshold and the overexposure score not exceeding the overexposure threshold, the overexposure score and the backlight score are saved.

9. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images