Fatigue recognition method, device, equipment and medium

Abstract

The application discloses a fatigue recognition method, device, equipment and medium. It relates to the technical field of head-mounted devices. The method is applied to a head-mounted device, the head-mounted device is provided with a proximity sensor and an inertial sensor, and the method comprises the following steps: acquiring current information collected by the proximity sensor and head movement information of a wearer of the head-mounted device collected by the inertial sensor, the signal emission direction of the proximity sensor being directed towards the human eye; determining the blink frequency of the wearer according to the current information; determining at least one of the head-bowing frequency and the yawning frequency of the wearer according to the head movement information; and determining the fatigue state of the wearer according to at least one of the head-bowing frequency and the yawning frequency and the blink frequency. The application provides a method for accurately recognizing the fatigue state of the wearer without increasing the hardware cost of the head-mounted device.

Description

Technical Field

[0001] This application relates to the field of head-mounted device technology, and more specifically, to a fatigue recognition method, a fatigue recognition device, a head-mounted device, and a computer-readable storage medium. Background Technology

[0002] With the development of technology and the economy, more and more people are choosing to travel by car. To ensure people's safety, drivers must not drive while fatigued. Therefore, monitoring whether drivers are fatigued is particularly important.

[0003] Currently, the primary method for monitoring driver fatigue is through deep learning. Specifically, a deep learning network is trained to learn various human facial expressions indicating fatigue, generating a deep learning model. Based on the driver's facial expressions and the generated deep learning model, the driver's fatigue level is determined. However, this method suffers from limitations due to the high computational resource requirements for generating the deep learning model and the inability to comprehensively collect samples of human facial expressions indicating fatigue.

[0004] Therefore, how to provide another means of monitoring whether a vehicle driver is fatigued has become one of the urgent technical problems to be solved. Summary of the Invention

[0005] One objective of this application is to provide a new technical solution for fatigue identification.

[0006] According to a first aspect of this application, a fatigue recognition method is provided, applied to a head-mounted device, the head-mounted device being equipped with a proximity sensor and an inertial sensor, including:

[0007] The current information collected by the proximity sensor and the head movement information of the wearer of the head-mounted device collected by the inertial sensor are acquired, and the signal emission direction of the proximity sensor is directed toward the human eye;

[0008] Based on the current information, the wearer's blinking frequency is determined;

[0009] Based on the head movement information, at least one of the wearer's head-down frequency and yawning frequency is determined;

[0010] The wearer's fatigue state is determined based on at least one of the head-down frequency and the yawning frequency, as well as the blinking frequency.

[0011] Optionally, determining at least one of the wearer's head-down frequency and yawning frequency based on the head movement information includes:

[0012] The head-mounted device is equipped with a microphone to acquire sound information collected by the microphone;

[0013] The wearer's yawning frequency is determined based on the head movement information and the sound information;

[0014] And / or, based on the head movement information, determine the wearer's head-down frequency.

[0015] Optionally, the head-mounted device is equipped with a positioning sensor, and the method further includes, before acquiring the current information collected by the proximity sensor and the head movement information of the wearer of the head-mounted device collected by the inertial sensor:

[0016] The driving status of the wearer is determined based on the positioning sensor, and the driving status includes driving a vehicle and not driving a vehicle;

[0017] When the driving state is driving a vehicle, the step of acquiring the current information collected by the proximity sensor and the head movement information of the wearer of the head-mounted device collected by the inertial sensor is triggered.

[0018] Optionally, the step of triggering the acquisition of current information collected by the proximity sensor and head movement information of the wearer of the head-mounted device collected by the inertial sensor when the driving state is driving a vehicle includes:

[0019] When the driving state is driving a vehicle, determine the wearer's continuous driving time;

[0020] If the continuous driving time is less than or equal to a preset time, the step of acquiring the current information collected by the proximity sensor and the head movement information of the wearer of the head-mounted device collected by the inertial sensor is triggered.

[0021] The method further includes:

[0022] If the continuous driving duration exceeds a preset duration, it is determined that the wearer is in a state of fatigue.

[0023] Optionally, the wearer's fatigue state is determined based on at least one of the head-down frequency and the yawning frequency, as well as the blinking frequency, including:

[0024] The wearer is determined to be in a state of fatigue if at least one of the following conditions is met:

[0025] The first item is that the head-down frequency is greater than a first preset frequency, where the first preset frequency is the maximum value of the head-down frequency when the person is in a fatigued state (not fatigued).

[0026] The second item is that the yawning frequency is greater than a second preset frequency, where the second preset frequency is the maximum value of the yawning frequency when the person is in a fatigued state (not fatigued).

[0027] The third item is that the blinking frequency is greater than a third preset frequency, where the third preset frequency is the maximum blinking frequency in a non-fatigue state.

[0028] Optionally, the head-mounted device is equipped with an iris camera, and the method further includes, before acquiring the current information collected by the proximity sensor and the head movement information of the wearer of the head-mounted device collected by the inertial sensor:

[0029] Acquire the iris image of the wearer captured by the iris camera;

[0030] The iris image is sent to the vehicle that the wearer is about to drive, and the vehicle determines whether to perform the unlocking operation based on the iris image;

[0031] When the unlocking operation is performed, the acquisition of current information collected by the proximity sensor and head movement information of the wearer of the head-mounted device collected by the inertial sensor is triggered.

[0032] Optionally, the head-mounted device is equipped with a bone conduction speaker, and the method further includes:

[0033] When the wearer is in a state of fatigue, the bone conduction speaker is controlled to output a reminder message.

[0034] According to a second aspect of this application, a fatigue detection device is provided, applied to a head-mounted device, the head-mounted device being equipped with a proximity sensor and an inertial sensor, including:

[0035] The acquisition module is used to acquire the current information collected by the proximity sensor and the head movement information of the wearer of the head-mounted device collected by the inertial sensor, wherein the signal emission direction of the proximity sensor is towards the human eye;

[0036] The first determining module is used to determine the blinking frequency of the wearer based on the current information;

[0037] The second determining module is used to determine at least one of the wearer's head-down frequency and yawning frequency based on the head movement information;

[0038] The third determining module is used to determine the wearer's fatigue state based on at least one of the head-down frequency and the yawning frequency, as well as the blinking frequency.

[0039] According to a third aspect of this application, a head-mounted device is provided, the head-mounted device comprising the fatigue detection device, proximity sensor, and inertial sensor as described in the second aspect of this application;

[0040] Alternatively, the head-mounted device may include a memory, a processor, the proximity sensor, and the inertial sensor;

[0041] Wherein, the signal emission direction of the proximity sensor is toward the human eye, the proximity sensor is used to collect current information, the inertial sensor is used to collect head movement information of the wearer of the head-mounted device, the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the fatigue recognition method as described in any one of the first aspects.

[0042] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the fatigue identification method according to any one of the first aspects.

[0043] This application provides a fatigue recognition method applied to a head-mounted device. The head-mounted device is equipped with a proximity sensor and an inertial sensor. The method includes: acquiring current information collected by the proximity sensor and head movement information of the wearer collected by the inertial sensor, with the signal emission direction of the proximity sensor pointing towards the wearer's eyes; determining the wearer's blinking frequency based on the current information; determining at least one of the wearer's head-down frequency and yawning frequency based on the head movement information; and determining the wearer's fatigue state based on at least one of the head-down frequency, yawning frequency, and blinking frequency. This application provides a method for accurately recognizing the wearer's fatigue state without increasing the hardware cost of the head-mounted device. Furthermore, the method provided in this application can also enhance the functionality of the head-mounted device.

[0044] Other features and advantages of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present application and, together with their description, serve to explain the principles of the present application.

[0046] Figure 1 This is a block diagram of the hardware configuration of a head-mounted device for a fatigue recognition method according to an embodiment of this application;

[0047] Figure 2 This is a flowchart illustrating a fatigue identification method according to an embodiment of this application;

[0048] Figure 3 This is a schematic diagram of the structure of a head-mounted device according to an embodiment of this application;

[0049] Figure 4 This is a schematic diagram of the equivalent structure of an inertial sensor according to an embodiment of this application;

[0050] Figure 5a This is a schematic diagram illustrating the working principle of a proximity sensor when eyes are open, according to an embodiment of this application.

[0051] Figure 5b This is a schematic diagram illustrating the working principle of a proximity sensor in the case of closed eyes, according to an embodiment of this application.

[0052] Figure 6 This is a schematic diagram of the structure of a fatigue identification device according to an embodiment of this application;

[0053] Figure 7 This is a schematic diagram of the structure of a head-mounted device according to an embodiment of this application. Detailed Implementation

[0054] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0055] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0056] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0057] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0058] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0059] Figure 1This is a block diagram of the hardware configuration of a head-mounted device for a fatigue recognition method provided in an embodiment of this application.

[0060] The head-mounted device 1000 can be either smart glasses or a smart helmet. Specifically, the smart glasses can be AR smart glasses, and the smart helmet can be an AR smart helmet. It should be noted that... Figure 1 The device is shown as a smart glasses-like head-mounted device 1000.

[0061] The head-mounted device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and a sensor 1900, etc. The processor 1100 may be a central processing unit (CPU), a microprocessor (MCU), etc. The memory 1200 may include, for example, ROM (Read-Only Memory), RAM (Random Access Memory), or non-volatile memory such as a hard disk. The interface device 1300 may include, for example, a USB interface, a headphone jack, etc. The communication device 1400 may be capable of wired or wireless communication. The display device 1500 may be, for example, an LCD screen, a touch screen, etc. The input device 1600 may include, for example, a touch screen, a keyboard, etc. Users can input / output voice information through the speaker 1700 and the microphone 1800. The speaker 1700 may be, for example, a bone conduction speaker. The sensor 1900 may include, but is not limited to, proximity sensors, inertial sensors, positioning sensors, and image sensors. The image sensor may be an iris camera.

[0062] Despite Figure 1 The head-mounted device 1000 shows multiple devices, but this application may only involve some of them. For example, the head-mounted device 1000 may only involve the memory 1200 and the processor 1100.

[0063] In the embodiments applied in this application, the memory 1200 of the head-mounted device 1000 is used to store instructions for controlling the processor 1100 to execute the fatigue recognition method provided in the embodiments of this application.

[0064] In the above description, those skilled in the art can design instructions based on the scheme disclosed in this application. How the instructions control the processor to operate is well known in the art, and therefore will not be described in detail here.

[0065] This application provides a fatigue identification method, which is applied to, for example... Figure 1 The head-mounted device 1000 shown is equipped with a proximity sensor and an inertial sensor. For example... Figure 2As shown, the fatigue identification method provided in this application includes the following steps S2100-S2400:

[0066] S2100: Acquires current information collected by the proximity sensor and head movement information of the wearer of the head-mounted device collected by the inertial sensor. The signal transmission direction of the proximity sensor is towards the human eye.

[0067] In this application embodiment, the number of proximity sensors and inertial sensors is not limited. In one example, when the head-mounted device is smart glasses, the placement of the proximity sensors and inertial sensors can be as follows: Figure 3 As shown. Among them, Figure 3 The example shown uses a proximity sensor and an inertial sensor as two examples.

[0068] It is understandable that fatigue manifests in three ways on the human face: blinking, yawning, and head nodding (which occurs when nodding off during drowsiness). In this embodiment, the blinking frequency of the wearer wearing the head-mounted device is determined by current information collected by a proximity sensor. Furthermore, the yawning and head-nodding frequencies of the wearer are determined by head movement information collected by an inertial sensor.

[0069] It should be noted that when multiple proximity sensors are used, the wearer's blinking frequency is determined by averaging the current information collected from these sensors. Similarly, when multiple inertial sensors are used, the wearer's yawning and head-down frequencies are determined by averaging the head movement information collected from these sensors.

[0070] Furthermore, traditional head-mounted devices typically incorporate proximity sensors and inertial sensors. In conventional technology, proximity sensors detect whether a user is wearing the head-mounted device, while inertial sensors monitor the wearer's head posture to control the device for adaptive display. In this embodiment, the proximity sensor on the head-mounted device is reused to determine the wearer's blinking frequency. Additionally, the inertial sensor is reused to determine the wearer's yawning frequency and / or head-down frequency. This eliminates the need to increase the hardware cost of the head-mounted device.

[0071] The working principle of the proximity sensor is as follows:

[0072] The proximity sensor is implemented using an infrared light emitter and a photodiode. The infrared light emitter emits infrared light of the same intensity at a fixed frequency. When there are no obstructions within the emission range, the infrared light escapes without reflection. However, when there are obstructions within the emission range, part of the infrared light is refracted and part is reflected. The reflected infrared light returns along its original emission path and is sensed by the photodiode integrated with the infrared light emitter. Upon sensing the sudden arrival of infrared light, the photodiode generates a rapidly increasing current. The duration of the infrared light's travel from emission to sensing by the photodiode is proportional to the reflection path of the infrared light; the path from the obstruction to the infrared light emitter is the reflection path.

[0073] In addition, the working principle of an inertial sensor is as follows:

[0074] like Figure 4 As shown, an inertial sensor can be understood as a small steel ball inside a closed metal box. Any movement of the inertial sensor in any direction will cause the small steel ball to collide with the metal box wall, thereby generating a certain stress that can be detected. Different stresses have different directions and magnitudes, indicating different actions of the wearer of the inertial sensor.

[0075] S2200: Determine the wearer's blinking frequency based on current information.

[0076] In this embodiment, based on the working principle of the proximity sensor in S2100, the specific implementation of S2200 is as follows:

[0077] Due to the thickness of the eyelids, it leads to... Figure 5a As shown, with eyes open, the reflection path Lref of the infrared light emitted by the infrared emitter is compared to... Figure 5a As shown in the case with eyes closed, the reflection path Lref of the infrared light emitted by the infrared emitter is longer. That is, in Figure 5a As shown in the case of open eyes, the duration of infrared light emitted to the photodiode and sensed by the infrared light is compared to that shown in the case of open eyes. Figure 5a As shown, with eyes closed, the duration of infrared light emitted to the photodiode is longer, with a difference typically around 1 microsecond. Based on this, the number of increasing currents separated by approximately 1 microsecond is determined from the current information collected by the proximity sensor. This determined number is used as the wearer's blink count during the current information occurrence period. Furthermore, based on the aforementioned count, the wearer's blink count for a preset time period is calculated as the wearer's blink frequency. The preset time period can be set empirically, for example, 2 minutes.

[0078] It should be noted that a high-precision proximity sensor can be used in this embodiment. Compared to ordinary proximity sensors, high-precision proximity sensors emit infrared light at a higher frequency and with a higher emission intensity. Therefore, high-precision proximity sensors are more likely to detect fast-moving actions like blinking, where the infrared light reflection path changes little.

[0079] S2300: Based on head movement information, determine at least one of the wearer's head-down frequency and yawning frequency.

[0080] In this embodiment, the above-mentioned S2300 can be implemented in the following two ways:

[0081] The first approach includes the following S2310 and / or S2311:

[0082] S2310. The head-mounted device is equipped with a microphone to acquire sound information collected by the microphone and determine the wearer's yawning frequency based on head movement information and sound information.

[0083] In this embodiment, when a wearer yawns, their mouth opens wide for a period of time, and a yawning sound is emitted. Based on this, it can be determined whether the wearer has opened their mouth wide for a period of time based on head movement information, and whether the wearer has emitted a yawning sound based on sound information collected by the microphone, thus determining whether the wearer is yawning. Furthermore, the yawning frequency of the wearer can be determined based on the number of yawns within a preset time period. Therefore, the specific implementation of S2310 above can be as follows:

[0084] Based on head movement information, a motion curve is determined. If a sub-motion curve exists within the motion curve that matches the standard motion curve corresponding to a yawn based on an inertial sensor, and if the sound information collected by the microphone within the time period corresponding to this sub-motion curve contains a yawn sound, the number of times the wearer yawns is determined as the number of sub-motion curves in the head movement information. The number of times the wearer yawns within a preset time period is used to determine the wearer's yawn frequency. The standard motion curve corresponding to a yawn based on an inertial sensor is pre-stored in the head-mounted device after manual testing.

[0085] Of course, considering the computing resources available to computers, the wearer's yawning frequency can also be determined solely based on head movement information.

[0086] S2311. Determine the wearer's head-down frequency based on head movement information.

[0087] In this embodiment, the specific implementation of S2311 can be as follows: A motion curve is determined based on head movement information; if a sub-motion curve exists in the motion curve that matches the standard motion curve corresponding to head-down during fatigue based on an inertial sensor, the number of sub-motion curves matching the standard motion curve corresponding to head-down during fatigue is determined as the wearer's head-down frequency; the wearer's head-down frequency is determined by the number of head-downs within a preset time period. The standard motion curve corresponding to yawning based on an inertial sensor is pre-stored in the head-mounted device after manual testing.

[0088] S2400 determines the wearer's fatigue state based on at least one of the head-down frequency and yawning frequency, as well as the blinking frequency.

[0089] In one embodiment of this application, the above-mentioned S2400 can be specifically implemented by the following S2410:

[0090] S2410. The wearer is determined to be in a state of fatigue if at least one of the following conditions is met:

[0091] The first condition is that the head-down frequency is greater than the first preset frequency, which is the maximum head-down frequency when the person is in a fatigued state (not fatigued).

[0092] The second condition is that the yawning frequency is greater than the second preset frequency, which is the maximum value of the yawning frequency when the person is in a fatigued state (not fatigued).

[0093] The third item is that the blinking frequency is greater than the third preset frequency, which is the maximum blinking frequency when the person is not fatigued.

[0094] In one embodiment of this application, the first preset frequency, the second preset frequency, and the third preset frequency can be set based on experience.

[0095] In another embodiment of this application, the process of obtaining the third preset frequency can be as follows: S2410-1 to S2410-3:

[0096] S2410-1. Obtain the wearer's blink frequency curve, which reflects the wearer's daily blink frequency at different time periods within a preset duration.

[0097] In one embodiment of this application, the horizontal axis of the blink frequency curve can be a 24-hour period within a day, and the corresponding vertical axis is the blink frequency within the corresponding hourly period. The blink frequency curve can be obtained by recording and fitting the blinking data of the wearer over multiple consecutive days. For example, the wearer's blinking count between 08:00 and 09:00 each day for a week can be continuously recorded, and the average blinking count between 08:00 and 09:00 for the week can be used as the daily blinking frequency for the hourly period of 08:00 to 09:00.

[0098] Understandably, the blink frequency curve reflects the wearer's daily blinking habits. Thus, an accurate third preset frequency can be determined based on the blink frequency curve.

[0099] S2410-2, Obtain the time period during which blinking occurs.

[0100] S2410-3, The daily blinking frequency corresponding to the blinking frequency curve during the occurrence time period is taken as the third preset frequency.

[0101] In this embodiment, based on the blink frequency occurrence time period obtained in S2200 above, a daily blink frequency within a matching time period is selected from the blink frequency curve. The selected daily blink frequency is used as the third preset frequency.

[0102] In this embodiment of the application, if at least one of the above conditions is met, it is determined that the wearer is in a state of fatigue.

[0103] In another embodiment of this application, different items correspond to different scores. If the sum of the scores for the above three items is greater than a preset score, then the wearer is determined to be in a state of fatigue. For example:

[0104] In the first item, if the frequency of looking down is greater than the first preset frequency, the score for the first item is determined to be 3 points; otherwise, the score is determined to be 0 points.

[0105] In the second item, if the yawning frequency is greater than the second preset frequency, the score for the second item is determined to be 4 points; otherwise, the score is determined to be 0 points.

[0106] In the third item, if the blinking frequency is greater than the third preset frequency, the score for the third item is determined to be 3 points; otherwise, the score is determined to be 0 points.

[0107] If the sum of the scores for the first, second, and third items is greater than 5, then the wearer is determined to be in a state of fatigue.

[0108] In this embodiment of the application, by judging the wearer's yawning and / or head-down and blinking, the wearer's fatigue state can be accurately determined.

[0109] In summary, the fatigue recognition method provided in this application requires significantly less computing resources compared to fatigue state recognition methods based on deep learning. Furthermore, the fatigue recognition method provided in this application is implemented using a head-mounted device, which enhances the functionality of the head-mounted device.

[0110] This application provides a fatigue recognition method applied to a head-mounted device. The head-mounted device is equipped with a proximity sensor and an inertial sensor. The method includes: acquiring current information collected by the proximity sensor and head movement information of the wearer collected by the inertial sensor, with the signal emission direction of the proximity sensor pointing towards the wearer's eyes; determining the wearer's blinking frequency based on the current information; determining at least one of the wearer's head-down frequency and yawning frequency based on the head movement information; and determining the wearer's fatigue state based on at least one of the head-down frequency, yawning frequency, and blinking frequency. This application provides a method for accurately recognizing the wearer's fatigue state without increasing the hardware cost of the head-mounted device. Furthermore, the method provided in this application can also enhance the functionality of the head-mounted device.

[0111] In one embodiment of this application, such as Figure 3 As shown, a positioning sensor is provided on the head-mounted device. Based on this, the fatigue recognition method provided in this application embodiment further includes the following steps S2110 and S2111 before S2100:

[0112] S2110. Determine the wearer's driving status based on the positioning sensor.

[0113] Driving status includes both driving a vehicle and not driving a vehicle.

[0114] In one embodiment of this application, the positioning sensor may be exemplarily a GPS.

[0115] In one embodiment of this application, the specific implementation of S2110 above can be as follows: determining the wearer's position coordinates based on the positioning sensor; determining the wearer's speed based on the wearer's position coordinates; if the wearer's speed is greater than a preset speed, then determining the wearer's driving state as driving a vehicle, otherwise determining it as not driving a vehicle. The preset speed is typically the minimum speed during vehicle operation and can be set based on experience.

[0116] Of course, the specific implementation of S2110 above can also be as follows: determine the wearer's location coordinates based on the positioning sensor; if the wearer is determined to be on a highway or other road based on the wearer's location coordinates, then determine the wearer's driving status as driving a vehicle, otherwise determine it as not driving a vehicle.

[0117] S2111, When the driving state is driving a vehicle, trigger the step of acquiring current information collected by the proximity sensor and head movement information of the wearer of the head-mounted device collected by the inertial sensor.

[0118] It is understandable that fatigue detection is particularly important when the wearer is driving. Therefore, upon determining that the wearer is driving, the aforementioned S2100 is triggered, enabling proactive detection of the wearer's fatigue state. This enhances the intelligence of the fatigue detection method provided in this application embodiment.

[0119] Corresponding to S2111 above, if the driving state is a non-driving vehicle, then S2100 above will not be triggered, and fatigue recognition will not be performed. This reduces unnecessary computer resource consumption by the head-mounted device.

[0120] In one embodiment of this application, the above-mentioned S2111 can be specifically implemented by the following S2111-1 and S2111-2:

[0121] S2111-1. When driving, determine the wearer's continuous driving time.

[0122] S2111-2, When the continuous driving time is less than or equal to a preset time, trigger the step of acquiring the current information collected by the proximity sensor and the head movement information of the wearer of the head-mounted device collected by the inertial sensor.

[0123] In this embodiment, the preset duration is the maximum safe driving duration. Since the Road Traffic Safety Law stipulates that drivers must stop and rest for at least 20 minutes after driving for more than 4 hours, the preset duration can be set to 4 hours.

[0124] In this embodiment, if the continuous driving time is less than a preset time, it indicates that the wearer may be driving while fatigued. In this case, S2100 is triggered to actively identify the wearer's fatigue state. This improves the intelligence of the fatigue recognition method provided in this embodiment.

[0125] Corresponding to S2111-2 above, the fatigue identification method provided in this application embodiment further includes the following S2500:

[0126] S2500 determines that the wearer is in a state of fatigue when the continuous driving time exceeds the preset time.

[0127] In this embodiment, if the continuous driving time exceeds a preset duration, it indicates that the wearer has exceeded the maximum safe driving time and is in a state of fatigued driving. At this point, it is undoubtedly determined that the wearer is in a state of fatigue.

[0128] Based on any of the above embodiments, the fatigue identification method provided in this application further includes the following S2600:

[0129] S2600: When the wearer is in a state of fatigue, the bone conduction speaker outputs a reminder message.

[0130] In this embodiment, the head-mounted device is equipped with a bone conduction microphone with excellent sound transmission performance. The reminder message can be a voice message reminding the wearer that they are currently in a state of fatigue, such as "You are tired, please rest for a moment." The reminder message can also be refreshing music to help the wearer transition from a state of fatigue to a state of non-fatigue.

[0131] In this embodiment of the application, when the wearer is in a state of fatigue, a reminder message is output by controlling the bone conduction speaker so that the wearer can be aware that he is in a state of fatigue.

[0132] In one embodiment of this application, when the wearer is driving a vehicle and it is determined that the wearer is in a state of fatigue, the fatigue recognition method provided in this embodiment further includes the following S2700:

[0133] S2700: Send a fatigue driving upload instruction to the vehicle driven by the wearer of the head-mounted device, so that the vehicle can report the wearer's fatigue driving to the vehicle management equipment.

[0134] In one embodiment of this application, the vehicle management device may be exemplarily a highway management office.

[0135] The S2700 described above can be used to report the wearer's state of fatigue.

[0136] In another embodiment of this application, such as Figure 3 As shown, the head-mounted device provided in this embodiment of the application is equipped with an iris camera. Based on this, the above-mentioned S2100 can also be implemented through the following S2120-S2122:

[0137] S2120. Obtain the iris image of the wearer captured by the iris camera.

[0138] S2121. The iris image is sent to the vehicle to which the wearer is to drive, and the vehicle determines whether to perform the unlocking operation based on the iris image.

[0139] In this embodiment, the vehicle pre-stores the iris image of the authorized driver. The vehicle can communicate with the head-mounted device.

[0140] Once the iris camera captures an iris image, the head-mounted device sends this image to the vehicle. The vehicle then matches the iris image sent by the head-mounted device with a pre-stored image of a legitimate driver's iris. If a match is successful, it indicates that the wearer is the legitimate driver of the vehicle. At this point, the vehicle unlocks. This eliminates the need for the wearer to manually unlock the vehicle, further enhancing the functionality of the head-mounted device.

[0141] S2122. When performing the unlocking operation, trigger the acquisition of current information collected by the proximity sensor and head movement information of the wearer of the head-mounted device collected by the inertial sensor.

[0142] In this embodiment, after the vehicle successfully unlocks, it notifies the head-mounted device that the unlocking operation has been performed. Based on this, it can be known that the wearer is about to drive the vehicle. At this time, the above-mentioned S2100 is triggered to actively identify the wearer's fatigue state.

[0143] like Figure 6 As shown, this application embodiment also provides a fatigue recognition device 600, which is applied to a head-mounted device. The head-mounted device is equipped with a proximity sensor and an inertial sensor, including:

[0144] The acquisition module 610 is used to acquire the current information collected by the proximity sensor and the head movement information of the wearer of the head-mounted device collected by the inertial sensor, wherein the signal emission direction of the proximity sensor is towards the human eye;

[0145] The first determining module 620 is used to determine the blinking frequency of the wearer based on the current information;

[0146] The second determining module 630 is used to determine at least one of the wearer's head-down frequency and yawning frequency based on the head movement information.

[0147] The third determining module 640 is used to determine the wearer's fatigue state based on at least one of the head-down frequency and the yawning frequency, as well as the blinking frequency.

[0148] In one embodiment of this application, the head-mounted device is equipped with a microphone, and the second determining module is specifically used for:

[0149] Acquire the sound information collected by the microphone;

[0150] The wearer's yawning frequency is determined based on the head movement information and the sound information;

[0151] And / or, based on the head movement information, determine the wearer's head-down frequency.

[0152] In one embodiment of this application, the head-mounted device is equipped with a positioning sensor. Based on this, the fatigue recognition device 600 provided in this embodiment further includes:

[0153] The trigger module is used to determine the wearer's driving status based on the positioning sensor, the driving status including driving a vehicle and not driving a vehicle;

[0154] When the driving state is driving a vehicle, the step of acquiring the current information collected by the proximity sensor and the head movement information of the wearer of the head-mounted device collected by the inertial sensor is triggered.

[0155] In one embodiment of this application, the triggering module is specifically used for:

[0156] The driving status of the wearer is determined based on the positioning sensor, and the driving status includes driving a vehicle and not driving a vehicle;

[0157] When the driving state is driving a vehicle, the step of acquiring the current information collected by the proximity sensor and the head movement information of the wearer of the head-mounted device collected by the inertial sensor is triggered.

[0158] In this embodiment of the application, the fatigue recognition device 600 provided in this embodiment of the application further includes:

[0159] The fourth determining module is used to determine that the wearer is in a state of fatigue when the continuous driving time exceeds a preset time.

[0160] In one embodiment of this application, the third determining module 640 is specifically used for:

[0161] The wearer is determined to be in a state of fatigue if at least one of the following conditions is met:

[0162] The first item is that the head-down frequency is greater than a first preset frequency, where the first preset frequency is the maximum value of the head-down frequency when the person is in a fatigued state (not fatigued).

[0163] The second item is that the yawning frequency is greater than a second preset frequency, where the second preset frequency is the maximum value of the yawning frequency when the person is in a fatigued state (not fatigued).

[0164] The third item is that the blinking frequency is greater than a third preset frequency, where the third preset frequency is the maximum blinking frequency in a non-fatigue state.

[0165] In one embodiment of this application, the acquisition module 610 is further configured to:

[0166] Acquire the iris image of the wearer captured by the iris camera;

[0167] In this embodiment of the application, the fatigue recognition device 600 provided in this embodiment of the application further includes:

[0168] The transmitting module is used to send the iris image to the vehicle that the wearer is about to drive, and the vehicle determines whether to perform the unlocking operation based on the iris image;

[0169] In this embodiment of the application, the triggering module is further configured to: trigger the acquisition of current information collected by the proximity sensor and head movement information of the wearer of the head-mounted device collected by the inertial sensor when the unlocking operation is performed.

[0170] In one embodiment of this application, the head-mounted device is equipped with a bone conduction speaker. Based on this, the fatigue recognition device 600 provided in this embodiment further includes:

[0171] The control module is used to control the bone conduction speaker to output reminder information when the wearer is in a state of fatigue.

[0172] This application embodiment also provides a head-mounted device 700, including as follows: Figure 6 The fatigue detection device 600, proximity sensor, and inertial sensor shown are illustrated.

[0173] Or, such as Figure 7 As shown, it includes a memory 710, a processor 720, the proximity sensor 730, and the inertial sensor 740;

[0174] The proximity sensor 730 emits signals toward the human eye and is used to collect current information. The inertial sensor 740 is used to collect head movement information of the wearer of the head-mounted device. The memory 710 is used to store computer instructions. The processor 720 is used to retrieve the computer instructions from the memory 710 to execute the fatigue recognition method as described in any of the above method embodiments.

[0175] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the fatigue recognition method according to any one of the above method embodiments.

[0176] This application may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application.

[0177] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0178] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0179] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0180] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0181] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0182] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0183] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be well known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0184] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this application is defined by the appended claims.

Claims

1. A fatigue identification method, characterized in that, Applied to head-mounted devices, the head-mounted devices are equipped with proximity sensors and inertial sensors, including: The current information collected by the proximity sensor and the head movement information of the wearer of the head-mounted device collected by the inertial sensor are acquired, and the signal emission direction of the proximity sensor is directed toward the human eye; Based on the current information, the wearer's blinking frequency is determined; Based on the head movement information, the wearer's yawning frequency is determined; The wearer's fatigue state is determined based on the yawning frequency and the blinking frequency; The step of determining the wearer's yawning frequency based on the head movement information includes: The head-mounted device is equipped with a microphone to acquire sound information collected by the microphone; Based on the head movement information, it is determined whether the wearer has opened their mouth wide for a period of time, and based on the sound information, it is determined whether the wearer's yawning frequency has been determined.

2. The method according to claim 1, characterized in that, The step of determining the wearer's fatigue state based on the yawning frequency and the blinking frequency includes: Based on the head movement information, determine the wearer's head-down frequency; The wearer's fatigue state is determined based on the frequency of head tilting, the frequency of yawning, and the frequency of blinking.

3. The method according to claim 1, characterized in that, The head-mounted device is equipped with a positioning sensor. Before acquiring the current information collected by the proximity sensor and the head movement information of the wearer of the head-mounted device collected by the inertial sensor, the method further includes: The driving status of the wearer is determined based on the positioning sensor, and the driving status includes driving a vehicle and not driving a vehicle; When the driving state is driving a vehicle, the step of acquiring the current information collected by the proximity sensor and the head movement information of the wearer of the head-mounted device collected by the inertial sensor is triggered.

4. The method according to claim 3, characterized in that, When the driving state is driving a vehicle, the step of triggering the acquisition of current information collected by the proximity sensor and head movement information of the wearer of the head-mounted device collected by the inertial sensor includes: When the driving state is driving a vehicle, determine the wearer's continuous driving time; If the continuous driving time is less than or equal to a preset time, the step of acquiring the current information collected by the proximity sensor and the head movement information of the wearer of the head-mounted device collected by the inertial sensor is triggered. The method further includes: If the continuous driving duration exceeds a preset duration, it is determined that the wearer is in a state of fatigue.

5. The method according to claim 1, characterized in that, Determining the wearer's fatigue state based on the yawning frequency and the blinking frequency includes: The wearer is determined to be in a state of fatigue if at least one of the following conditions is met: The first item is that the yawning frequency is greater than the second preset frequency, where the second preset frequency is the maximum value of the yawning frequency when the person is in a fatigued state (not fatigued). The second item is that the blinking frequency is greater than a third preset frequency, where the third preset frequency is the maximum blinking frequency in a non-fatigue state.

6. The method according to claim 1, characterized in that, The head-mounted device is equipped with an iris camera. Before acquiring the current information collected by the proximity sensor and the head movement information of the wearer of the head-mounted device collected by the inertial sensor, the method further includes: Acquire the iris image of the wearer captured by the iris camera; The iris image is sent to the vehicle that the wearer is about to drive, and the vehicle determines whether to perform the unlocking operation based on the iris image; When the unlocking operation is performed, the acquisition of current information collected by the proximity sensor and head movement information of the wearer of the head-mounted device collected by the inertial sensor is triggered.

7. The method according to any one of claims 1-6, characterized in that, The head-mounted device is equipped with a bone conduction speaker, and the method further includes: When the wearer is in a state of fatigue, the bone conduction speaker is controlled to output a reminder message.

8. A fatigue detection device, characterized in that, Applied to head-mounted devices, the head-mounted devices are equipped with proximity sensors and inertial sensors, including: The acquisition module is used to acquire the current information collected by the proximity sensor and the head movement information of the wearer of the head-mounted device collected by the inertial sensor, wherein the signal emission direction of the proximity sensor is towards the human eye; The first determining module is used to determine the blinking frequency of the wearer based on the current information; The second determining module is used to determine, based on the head movement information, whether the wearer has opened their mouth wide and maintained it for a period of time, and thus determine the wearer's yawning frequency; The third determining module is used to determine the wearer's fatigue state based on the yawning frequency and the blinking frequency.

9. A head-mounted device, characterized in that, The head-mounted device includes the fatigue detection device, proximity sensor, and inertial sensor as described in claim 8; Alternatively, the head-mounted device may include a memory, a processor, the proximity sensor, and the inertial sensor; Wherein, the signal emission direction of the proximity sensor is toward the human eye, the proximity sensor is used to collect current information, the inertial sensor is used to collect head movement information of the wearer of the head-mounted device, the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the fatigue recognition method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the fatigue recognition method according to any one of claims 1-7.

Images