A driving state monitoring method and device

Abstract

The disclosure provides a driving state monitoring method and device, and relates to the technical field of automobile safety. The method comprises the following steps: acquiring an image, performing face recognition on the image, and obtaining a feature sequence corresponding to the image; determining a driving state corresponding to the image based on the feature sequence; updating a fatigue image queue of a user according to the driving state corresponding to the image, wherein the fatigue image queue comprises a plurality of images with a fatigue state, and the update rule of the fatigue image queue is a first-in first-out rule; and determining the driving state of the user according to the updated fatigue image queue. The disclosure improves the accuracy and robustness of fatigue detection during driving.

Description

Technical Field

[0001] This disclosure relates to the field of automotive safety technology, and in particular to a driving status monitoring method and device. Background Technology

[0002] Currently, with the continuous development of the automotive industry, more and more families are choosing to travel by car, leading to a continuous increase in traffic accidents. Among these accidents, fatigued driving is a significant contributing factor. Continuing to drive when fatigued can cause drowsiness, weakness in the limbs, poor concentration, impaired judgment, and even mental confusion or momentary memory loss. This can result in delayed or premature actions, lapses in operation, or improper correction timing, all of which greatly increase the risk of road accidents. Therefore, fatigue detection is crucial for driving safety.

[0003] However, the accuracy and robustness of fatigue detection methods in related technologies are low. Summary of the Invention

[0004] This disclosure provides a driving status monitoring method, apparatus, electronic device, and storage medium.

[0005] According to a first aspect of this disclosure, a driving state monitoring method is provided, the method comprising: acquiring an image and performing face recognition on the image to obtain a feature sequence corresponding to the image; determining the driving state corresponding to the image based on the feature sequence; updating a user's fatigue image queue according to the driving state corresponding to the image, the fatigue image queue containing multiple frames of images in a fatigued driving state, the update rule of the fatigue image queue being a first-in-first-out rule; and determining the user's driving state according to the updated fatigue image queue.

[0006] In some embodiments, determining the driving state corresponding to an image based on a feature sequence includes: obtaining a feature sequence corresponding to the image based on the obtained feature values ​​of the face parts, wherein the face parts include the eyes and the mouth; when the feature value of the eyes in the feature sequence indicates that both eyes are closed, or when the feature value of the eyes in the feature sequence indicates that one eye is closed and the feature value of the mouth indicates that the mouth is open, the driving state corresponding to the image is determined to be a fatigued state; otherwise, the driving state corresponding to the image is determined to be a non-fatigue state.

[0007] In some embodiments, updating the user's fatigue image queue according to the driving state corresponding to the image includes: when the driving state corresponding to the image is fatigued, inserting the image at the head of the fatigue image queue, wherein the timestamp corresponding to the image is the latest time in the fatigue image queue.

[0008] In some embodiments, non-fatigue includes a first non-fatigue state and a second non-fatigue state. Determining the driving state corresponding to the image as non-fatigue includes: when the feature values ​​of the eyes and / or mouth in the feature sequence indicate that they are not recognized, determining the driving state corresponding to the image as the first non-fatigue state; when the driving state is neither a fatigue state nor the first non-fatigue state, determining the driving state as the second non-fatigue state.

[0009] In some embodiments, when the driving state corresponding to the image is non-fatigue or unknown, a predetermined number of images are deleted from the tail of the fatigue image queue.

[0010] In some embodiments, when the driving state corresponding to the image is a non-fatigue state, deleting a predetermined number of images from the tail of the fatigue image queue includes: when the driving state corresponding to the image is a second non-fatigue state, deleting a first predetermined number of images from the tail of the fatigue image queue; and when the driving state corresponding to the image is a first non-fatigue state, deleting a second predetermined number of images from the tail of the fatigue image queue; wherein the first predetermined number of frames is different from the second predetermined number of frames.

[0011] In some embodiments, determining the user's driving state based on the updated fatigue image queue includes: determining the user's driving state as fatigued when the number of image frames in the fatigue image queue is greater than or equal to a preset threshold; and determining the user's driving state as normal driving when the number of image frames in the fatigue image queue is less than the preset threshold.

[0012] According to embodiments of this disclosure, by acquiring an image and performing facial recognition on the image to obtain the feature sequence corresponding to the image; based on the feature sequence, the driving state corresponding to the image is determined; according to the driving state corresponding to the image, the user's fatigue image queue is updated, the fatigue image queue contains multiple frames of images in the fatigue state, and the update rule of the fatigue image queue is a first-in-first-out rule; according to the updated fatigue image queue, the user's driving state is determined, thereby improving the accuracy and robustness of fatigue detection during driving.

[0013] According to a second aspect of this disclosure, a driving state monitoring device is provided, the device comprising: an acquisition unit for acquiring an image and performing face recognition on the image to obtain a feature sequence corresponding to the image; a first determination unit for determining the driving state corresponding to the image based on the feature sequence; an update unit for updating a user's fatigue image queue according to the driving state corresponding to the image, the fatigue image queue containing multiple frames of images in a fatigued driving state, the update rule of the fatigue image queue being a first-in-first-out rule; and a second determination unit for determining the user's driving state according to the updated fatigue image queue.

[0014] According to embodiments of this disclosure, an image is acquired through a driving state monitoring device, and facial recognition is performed on the image to obtain the feature sequence corresponding to the image; based on the feature sequence, the driving state corresponding to the image is determined; according to the driving state corresponding to the image, the user's fatigue image queue is updated, the fatigue image queue contains multiple frames of images in a fatigued driving state, and the update rule of the fatigue image queue is a first-in-first-out rule; according to the updated fatigue image queue, the user's driving state is determined, thereby improving the accuracy and robustness of fatigue detection during driving.

[0015] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0016] At least one processor; and

[0017] A memory that is communicatively connected to at least one processor; wherein,

[0018] The memory stores instructions that can be executed by at least one processor, such that the at least one processor is able to perform the method described in the first aspect above.

[0019] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the method of the first aspect described above.

[0020] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method as described in the first aspect above.

[0021] According to embodiments of this disclosure, by acquiring an image and performing facial recognition on the image to obtain the feature sequence corresponding to the image; based on the feature sequence, the driving state corresponding to the image is determined; according to the driving state corresponding to the image, the user's fatigue image queue is updated, the fatigue image queue contains multiple frames of images in the fatigue state, and the update rule of the fatigue image queue is a first-in-first-out rule; according to the updated fatigue image queue, the user's driving state is determined, thereby improving the accuracy and robustness of fatigue detection during driving.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0023] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0024] Figure 1A flowchart illustrating a driving status monitoring method provided in an embodiment of this disclosure;

[0025] Figure 2 A flowchart illustrating a driving status monitoring method provided in an embodiment of this disclosure;

[0026] Figure 3 A flowchart illustrating a specific feature combination method provided in an embodiment of this disclosure;

[0027] Figure 4 A flowchart illustrating a specific time-series queue decay method provided in this embodiment of the disclosure;

[0028] Figure 5 This is a schematic diagram of the structure of a driving status monitoring device provided in an embodiment of the present disclosure;

[0029] Figure 6 A schematic block diagram of an example electronic device 600 provided for embodiments of this disclosure. Detailed Implementation

[0030] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0031] A driving status monitoring method, apparatus, electronic device, and storage medium according to embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0032] Currently, with the continuous development of the automotive industry, more and more families are choosing to travel by car, leading to a continuous increase in traffic accidents. Among these accidents, fatigued driving is a significant contributing factor. Continuing to drive when fatigued can cause drowsiness, weakness in the limbs, poor concentration, impaired judgment, and even mental confusion or momentary memory loss. This can result in delayed or premature actions, lapses in operation, or improper correction timing, all of which greatly increase the risk of road accidents. Therefore, fatigue detection is crucial for driving safety.

[0033] However, the accuracy and robustness of fatigue detection methods in related technologies are low.

[0034] Specifically, existing technologies mainly fall into two categories: one is to obtain the final fatigue state by simply combining fatigue characteristics; the other is to set fatigue characteristic thresholds and use rules to control fatigue output.

[0035] For example, related technologies use simple combinations of speed, head posture, eye posture, and mouth posture to determine the final fatigue state, or use video algorithms to directly analyze the task status of videos and use weighted average calculations to determine whether the worker is fatigued by the frequency of the opening and closing of their eyes and mouth reaching a threshold within a certain period of time.

[0036] However, driver fatigue is a time-series process, and related technologies only use feature combinations to determine whether fatigue is present. Using feature combinations from a single image to make a judgment is very unreasonable and has very low accuracy for the entire fatigue process.

[0037] Furthermore, while related technologies directly analyze videos and possess temporal information to some extent, this method is very slow and demands extremely high computational resources. Its method of using weighted averages to determine fatigue levels is severely affected by model accuracy and exhibits poor robustness.

[0038] This disclosure proposes a driving state monitoring scheme that can effectively avoid result errors caused by model accuracy, while also taking into account the impact of time series information on driver fatigue.

[0039] This disclosure is primarily used in driver monitoring systems (DMS) in the automotive field, such as Advanced Driving Assistance Systems (ADAS). The method proposed in this disclosure, as part of the DMS system, integrates the results of multiple algorithms in a time-series manner, greatly eliminating interference from algorithm errors in individual image frames. It can provide information on whether the driver is fatigued within a certain period of time. The method proposed in this disclosure can effectively improve the accuracy and anti-interference capability of the entire DMS system.

[0040] The focus of this disclosure is to obtain various information about the driver, combine the obtained information into features, and then use a queue decay method to determine whether the driver is in a state of fatigue based on the current mapping state.

[0041] To address the problems of the existing technology, this disclosure proposes a driving state monitoring method. This method involves acquiring images and performing facial recognition on them to obtain corresponding feature sequences. Based on the feature sequences, the driving state corresponding to the image is determined. The user's fatigue image queue is updated according to the driving state, and the fatigue image queue contains multiple frames of images indicating a fatigued driving state. The update rule for the fatigue image queue is a first-in, first-out (FIFO) rule. Based on the updated fatigue image queue, the user's driving state is determined, thus improving the accuracy and robustness of fatigue detection during driving.

[0042] The method proposed in this disclosure is mainly applied to vehicles, specifically to advanced driver assistance systems, i.e., driver monitoring systems, but is not limited to this in the embodiments of this disclosure.

[0043] The following describes in detail, with reference to the accompanying drawings, a driving status monitoring method, device, electronic equipment, storage medium, and program product proposed in this disclosure.

[0044] Figure 1 This is a flowchart illustrating a driving state monitoring method provided in an embodiment of this disclosure. This method can be used in driver monitoring systems, but is not limited to this embodiment. Figure 1 As shown, the method includes the following steps:

[0045] Step 101: Acquire the image and perform face recognition on the image to obtain the feature sequence corresponding to the image.

[0046] In one implementation of this disclosure, an infrared camera can be used to capture images of the driver, and single-frame images of the video can be obtained to locate the driver's eyes and mouth as feature points. The state of the acquired feature points is represented in the form of a feature sequence according to preset labels.

[0047] Step 102: Determine the driving state corresponding to the image based on the feature sequence.

[0048] In one embodiment of this disclosure, the driver's facial region is cropped and located, and fatigue characteristics of the eyes and mouth are combined as a standard for judging fatigue. Based on the feature sequence, fatigue is categorized into two types: fatigued state and non-fatigue state.

[0049] In this disclosure, the non-fatigue state includes a first non-fatigue state and a second non-fatigue state. The first non-fatigue state is an unknown state, meaning there is no clear image formation. This occurs when the algorithm malfunctions, resulting in cropping errors or incomplete cropping, making it impossible to recognize the user's facial features and thus impossible to confirm their identity. The second non-fatigue state is determined by assessment that the user is awake. This disclosure uses a classification model to classify the left eye, right eye, and mouth based on feature sequences. For example, when 001 (i.e., left eye open, right eye open, mouth closed) is the user's awake state.

[0050] Step 103: Update the user's fatigue image queue according to the driving state corresponding to the image. The fatigue image queue contains multiple frames of images in the driving state of fatigue. The update rule for the fatigue image queue is the first-in, first-out rule.

[0051] In one embodiment of this disclosure, a time-decaying fatigue state queue, i.e. queue decay, is maintained for the driving state corresponding to the image obtained in step 102, i.e., the single-frame fatigue state, where the queue is a first-in-first-out data structure.

[0052] In this method, frame extraction involves drawing one frame at a time interval. Queue decay refers to accumulating fatigue levels when they occur, but decreasing them when non-fatigue levels are detected. The queue decay method acknowledges that the algorithm is not 100% reliable, but it is reliable most of the time. When a misjudgment occurs, decay occurs if a non-fatigue prediction is subsequently made. If the algorithm cannot consistently maintain a significantly higher number of fatigue detections than non-fatigue detections, it will continuously decay without determining driver fatigue, no matter how long it takes. Only when the number of fatigue detections significantly exceeds the number of non-fatigue detections (i.e., decay stops) does the algorithm begin accumulating fatigue levels until a preset fatigue threshold is reached, at which point driver fatigue is determined.

[0053] Step 104: Determine the user's driving status based on the updated fatigue image queue.

[0054] In one embodiment of this disclosure, the driver is determined to be fatigued by comparing the number of image frames in the updated fatigue image queue with a preset threshold.

[0055] The images in the fatigue image queue are all in a fatigued state.

[0056] Therefore, according to the embodiments of this disclosure, by acquiring an image and performing face recognition on the image to obtain the feature sequence corresponding to the image; based on the feature sequence, the driving state corresponding to the image is determined; according to the driving state corresponding to the image, the user's fatigue image queue is updated, the fatigue image queue contains multiple frames of images in the fatigue state, and the update rule of the fatigue image queue is a first-in-first-out rule; according to the updated fatigue image queue, the user's driving state is determined, thereby improving the accuracy and robustness of fatigue detection during driving.

[0057] Figure 2 This is a flowchart illustrating a driving status monitoring method provided in an embodiment of the present disclosure. Figure 2 based on Figure 1 The illustrated embodiment further defines steps 101, 103, and 104. Figure 2 In the illustrated embodiment, step 101 includes steps 201, 202, and 203; step 103 includes steps 205 and 206; and step 104 includes step 207. For example... Figure 2 As shown, the method includes the following steps.

[0058] Step 201: Identify multiple facial features in the image, including the eyes and mouth.

[0059] In some embodiments of this disclosure, single-frame images can be obtained from a video, or single-frame images can be captured at certain intervals; this is not limited in the embodiments of this disclosure. Face recognition is performed on the single-frame image, that is, the left eye region, right eye region, and mouth region are cropped through facial key point detection.

[0060] Step 202: Obtain the feature values ​​of each facial region.

[0061] In some embodiments of this disclosure, the states of the left eye, right eye, and mouth are classified according to a classification model. The categories are open, closed, and unknown, which are represented by the numbers 0, 1, and 2. In the embodiments of this disclosure, the setting of the numbers is not limited.

[0062] The unknown state in the classification model arises because, in reality, not all images can be guaranteed to be clearly imaged. Since the eye and mouth regions are cropped using an algorithm, errors can occur, meaning the cropped image regions may be incorrect. When cropping is incorrect or incomplete, the image is classified as unknown.

[0063] Classification refers to a single frame of an image; each frame is classified. A single image obtained through face recognition will capture two eye regions and one mouth region. Each acquired single frame will be classified for these three regions.

[0064] Step 203: Obtain the feature sequence corresponding to the image based on the obtained feature values ​​of the face region.

[0065] In some embodiments of this disclosure, the feature values ​​of the obtained facial features are combined to obtain the feature sequence corresponding to the image.

[0066] Specifically, based on the acquired facial feature categories—left eye, right eye, and mouth—features are combined to form a feature sequence. In this embodiment, for example, 001 represents a state where the left eye is closed, the right eye is closed, and the mouth is open. The order of feature combination is not limited in this disclosure.

[0067] Step 204: Determine the driving state corresponding to the image based on the feature sequence.

[0068] In some embodiments of this disclosure, when the feature value of the eye in the feature sequence indicates that both eyes are closed, or when the feature value of the eye in the feature sequence indicates that one eye is closed and the feature value of the mouth indicates that the mouth is open, the driving state corresponding to the image is determined to be a fatigued state; otherwise, the driving state corresponding to the image is determined to be a non-fatigue state.

[0069] Based on the feature sequence mapping, fatigue is categorized into two types: fatigued state and non-fatigue state. For example, in this embodiment, setting 001 represents eyes closed and mouth open, which can be identified as a fatigued state; setting 110 represents eyes open and mouth closed, which can be identified as a non-fatigue state.

[0070] The non-fatigue state includes a first non-fatigue state and a second non-fatigue state. When the feature values ​​of the eyes and / or mouth in the feature sequence indicate that they are not recognized, the driving state corresponding to the image is determined to be the first non-fatigue state; when the driving state is neither the fatigue state nor the first non-fatigue state, the driving state is determined to be the second non-fatigue state.

[0071] When the user's driving state is determined to be other than the fatigue state and the first non-fatigue state mentioned above, it is identified as the second non-fatigue state.

[0072] Specifically, for a single frame image, by detecting facial key points and cropping out the left eye region, right eye region, and mouth region, the image will be composed of three numbers representing the left eye category, right eye category, and mouth category, respectively.

[0073] Each number has three options, resulting in 27 possible combinations: 000, 100, 200, 010, 020, 030, 110, 120, 130, and so on. For 001, the left eye is closed, the right eye is closed, and the mouth is open; this is defined as fatigue in this embodiment. In this embodiment, the states defined as fatigue are 001, 000, 011, and 101, a total of four cases. The other 23 cases are non-fatigue states.

[0074] In this embodiment of the disclosure, based on the set judgment conditions, such as Figure 3 As shown, facial landmark detection and classification are performed on a single frame image, including identifying the categories of the eyes (left eye and right eye) and the mouth. Then, feature combinations are performed on each part, and the fatigue category is determined by mapping the feature sequence of the feature combination to the fatigue category.

[0075] Step 205: When the driving state corresponding to the image is fatigued, insert the image at the head of the fatigue image queue, where the timestamp corresponding to the image is the latest time in the fatigue image queue.

[0076] In some embodiments of this disclosure, a queue of length L is set up, which is a first-in, first-out (FIFO) data structure. After obtaining feature combinations according to the above steps, the category of a single-frame image is obtained, divided into two categories: fatigued and non-fatigue.

[0077] Different enqueue or dequeue operations are performed according to the category. At each time step, when a fatigue frame occurs, the frame TN is inserted into the head of the queue.

[0078] Specifically, in this embodiment of the disclosure, for ease of understanding, the queue can be viewed as a container, characterized by the principle that what goes in first must come out first. Therefore, the existing states in the queue must be arranged in chronological order, so the head of the queue represents the most recent fatigue state.

[0079] Step 206: When the driving state corresponding to the image is a non-fatigue state, delete a predetermined number of images from the tail of the fatigue image queue.

[0080] In some embodiments of this disclosure, when the driving state corresponding to the image is the second non-fatigue state, images of a first predetermined number of frames are deleted from the tail of the fatigue image queue; when the driving state corresponding to the image is the first non-fatigue state, images of a second predetermined number of frames are deleted from the tail of the fatigue image queue; wherein, the first predetermined number of frames and the second predetermined number of frames are different and can be set according to actual conditions, and are not limited in this disclosure.

[0081] Specifically, different enqueue or dequeue operations are performed according to the category. At each moment, when the second non-fatigue frame appears, frame β is taken from the tail of the queue; when the first non-fatigue frame appears, frame α is taken from the tail of the queue. After a period of time, the queue will contain a certain number of fatigue frames. For the first and second non-fatigue frames, fatigue frames with the same or different frame numbers can be deleted from the tail of the queue. Deleting a predetermined number of frames involves deleting frames with older time sequences, i.e., frames that were first enqueued (tail frames).

[0082] Here, α and β can be the same value or different values, and this disclosure does not impose any restrictions. This can be understood as applying different degrees of decay to different non-fatigue states. In other words, when α and β are different values, different decay rates are applied, fully reflecting the influence of time on the current state and the different decay treatments based on the current state.

[0083] Step 207: When the number of image frames in the fatigue image queue is greater than or equal to a preset threshold, the user's driving state is determined to be fatigued; when the number of image frames in the fatigue image queue is less than the preset threshold, the user's driving state is determined to be normal driving.

[0084] In some embodiments of this disclosure, after each frame is retrieved from or inserted into the queue, fatigue frames in the current queue are judged, and a threshold γ is set. When the number of frames in the queue is greater than this threshold, the current state is judged to be fatigue driving state; when the number of frames in the queue is less than the threshold, the current state is judged to be normal driving state.

[0085] Here, the threshold γ represents the number of frames.

[0086] It is understood that the fatigue image queue in this disclosure is a first-in, first-out data structure, so the images in the fatigue image queue can be arranged in a certain time sequence.

[0087] In this embodiment of the disclosure, the fatigue state time-series queue decay is used as the basis, such as Figure 4 The diagram shows a flowchart of a specific time-series queue decay method.

[0088] In summary, according to the embodiments of this disclosure, a time-decaying fatigue state queue is maintained by combining features based on the driver's state at each moment, thereby determining whether the driver is in a state of fatigued driving. The time queue effectively evaluates the driver's state over a period of time, offering greater accuracy and robustness compared to single-frame image judgment. Furthermore, different decay rates are set according to different categories of the current state, with different decay rates applied to the first non-fatigue state and the second non-fatigue state (unknown state). This fully reflects the influence of time on the current state, avoids errors caused by certain frame algorithm judgment mistakes, and the different decay processing based on the current state is more consistent with reality.

[0089] Corresponding to the aforementioned driving status monitoring method, this disclosure also proposes a driving status monitoring device. Figure 5 This is a schematic diagram of the structure of a driving status monitoring device 500 provided in an embodiment of this disclosure. Figure 5 As shown, it includes:

[0090] The acquisition unit 510 is used to acquire an image and perform face recognition on the image to obtain the feature sequence corresponding to the image;

[0091] The first determining unit 520 is used to determine the driving state corresponding to the image based on the feature sequence;

[0092] The update unit 530 is used to update the user's fatigue image queue according to the driving state corresponding to the image. The fatigue image queue contains multiple frames of images in the driving state of fatigue. The update rule of the fatigue image queue is the first-in-first-out rule.

[0093] The second determining unit 540 is used to determine the user's driving status based on the updated fatigue image queue.

[0094] In some embodiments, the first determining unit 520 is configured to: determine the driving state corresponding to the image based on the feature sequence, including: obtaining the feature sequence corresponding to the image according to the obtained feature values ​​of the face parts, wherein the face parts include the eyes and the mouth; when the feature value of the eyes in the feature sequence indicates that both eyes are closed, or when the feature value of the eyes in the feature sequence indicates that one eye is closed and the feature value of the mouth indicates that the mouth is open, the driving state corresponding to the image is determined to be a fatigued state; otherwise, the driving state corresponding to the image is determined to be a non-fatigue state.

[0095] In some embodiments, the update unit 530 is configured to: insert the image into the head of the fatigue image queue when the driving state corresponding to the image is fatigue state, wherein the timestamp corresponding to the image is the latest time in the fatigue image queue.

[0096] In some embodiments, the first determining unit 520 is configured to: determine that the driving state corresponding to the image is non-fatigue, where non-fatigue includes a first non-fatigue state and a second non-fatigue state; when the feature values ​​of the eyes and / or mouth in the feature sequence indicate that they are not recognized, determine that the driving state corresponding to the image is the first non-fatigue state; and when the driving state is neither a fatigue state nor the first non-fatigue state, determine that the driving state is the second non-fatigue state.

[0097] In some embodiments, the update unit 530 is configured to: delete a predetermined number of images from the tail of the fatigue image queue when the driving state corresponding to the image is a non-fatigue state.

[0098] In some embodiments, the updating unit 530 is configured to: delete a first predetermined number of images from the tail of the fatigue image queue when the driving state corresponding to the image is a second non-fatigue state; and delete a second predetermined number of images from the tail of the fatigue image queue when the driving state corresponding to the image is a first non-fatigue state; wherein the first predetermined number of frames is different from the second predetermined number of frames.

[0099] In some embodiments, the second determining unit 540 is configured to: determine that the user's driving state is fatigued when the number of image frames in the fatigue image queue is greater than or equal to a preset threshold; and determine that the user's driving state is normal when the number of image frames in the fatigue image queue is less than the preset threshold.

[0100] In summary, according to the embodiments of this disclosure, an image is acquired through a driving state monitoring device, and facial recognition is performed on the image to obtain the feature sequence corresponding to the image; based on the feature sequence, the driving state corresponding to the image is determined; according to the driving state corresponding to the image, the user's fatigue image queue is updated, the fatigue image queue contains multiple frames of images in a fatigued driving state, and the update rule of the fatigue image queue is a first-in-first-out rule; according to the updated fatigue image queue, the user's driving state is determined, thereby improving the accuracy and robustness of fatigue detection during driving.

[0101] It should be noted that since the device embodiments of this disclosure correspond to the method embodiments described above, the foregoing explanations and descriptions of the method embodiments also apply to the device of this embodiment, and the principles are the same. For details not disclosed in the device embodiments, please refer to the method embodiments described above, and they will not be repeated in this disclosure.

[0102] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0103] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0104] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 602 or a computer program loaded from storage unit 608 into RAM (Random Access Memory) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. I / O (Input / Output) interface 605 is also connected to bus 604.

[0105] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0106] The computing unit 601 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as methods for monitoring driving conditions. For example, in some embodiments, the method for monitoring driving conditions may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the aforementioned driving state monitoring method by any other suitable means (e.g., by means of firmware).

[0107] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0108] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0109] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0110] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0111] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0112] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0113] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0114] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0115] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for monitoring driving status, characterized in that, The method includes: Acquire an image and perform face recognition on the image to obtain the feature sequence corresponding to the image; Based on the feature sequence, the driving state corresponding to the image is determined; wherein, the driving state corresponding to the image includes a fatigued state and a non-fatigue state; wherein, the non-fatigue state includes a first non-fatigue state and a second non-fatigue state; The user's fatigue image queue is updated according to the driving state corresponding to the image. The fatigue image queue contains multiple frames of images in the driving state of fatigue. The update rule of the fatigue image queue is a first-in-first-out rule. The user's driving status is determined based on the updated fatigue image queue; When the driving state corresponding to the image is the second non-fatigue state, the image of the first predetermined number of frames is deleted from the tail of the fatigue image queue. When the driving state corresponding to the image is the first non-fatigue state, the image of the second predetermined number of frames is deleted from the tail of the fatigue image queue; The first predetermined number of frames is different from the second predetermined number of frames.

2. The method according to claim 1, characterized in that, Determining the driving state corresponding to the image based on the feature sequence includes: Based on the obtained feature values ​​of the facial features, the feature sequence corresponding to the image is obtained, wherein the facial features include the eyes and the mouth; When the feature value of the eye in the feature sequence indicates that both eyes are closed, or when the feature value of the eye in the feature sequence indicates that one eye is closed and the feature value of the mouth indicates that the mouth is open, the driving state corresponding to the image is determined to be fatigued. Otherwise, the driving state corresponding to the image is determined to be non-fatigue.

3. The method according to claim 2, characterized in that, The step of updating the user's fatigue image queue based on the driving state corresponding to the image includes: When the driving state corresponding to the image is fatigued, the image is inserted at the head of the fatigue image queue, wherein the timestamp corresponding to the image is the latest time in the fatigue image queue.

4. The method according to claim 2, characterized in that, Determining that the driving state corresponding to the image is non-fatigue includes: When the feature values ​​of the eyes and / or mouth in the feature sequence indicate that they are not recognized, the driving state corresponding to the image is determined to be the first non-fatigue state; When the driving state is neither the fatigued state nor the first non-fatigue state, the driving state is determined to be the second non-fatigue state.

5. The method according to any one of claims 3 to 4, characterized in that, Determining the user's driving status based on the updated fatigue image queue includes: When the number of image frames in the fatigue image queue is greater than or equal to a preset threshold, the user's driving state is determined to be fatigued. When the number of image frames in the fatigue image queue is less than the preset threshold, the user's driving state is determined to be normal driving state.

6. A driving status monitoring device, characterized in that, include: An acquisition unit is used to acquire an image and perform face recognition on the image to obtain the feature sequence corresponding to the image; The first determining unit is configured to determine the driving state corresponding to the image based on the feature sequence; wherein the driving state corresponding to the image includes a fatigued state and a non-fatigue state; wherein the non-fatigue state includes a first non-fatigue state and a second non-fatigue state. The update unit is used to update the user's fatigue image queue according to the driving state corresponding to the image. The fatigue image queue contains multiple frames of images in the driving state of fatigue. The update rule of the fatigue image queue is a first-in-first-out rule. The second determining unit is used to determine the user's driving status based on the updated fatigue image queue; When the driving state corresponding to the image is the second non-fatigue state, the image of the first predetermined number of frames is deleted from the tail of the fatigue image queue. When the driving state corresponding to the image is the first non-fatigue state, the image of the second predetermined number of frames is deleted from the tail of the fatigue image queue; The first predetermined number of frames is different from the second predetermined number of frames.

7. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.

Images