Fatigue detection method and device for head-mounted equipment
By acquiring facial data from wearers of head-mounted devices and using various fatigue detection models to analyze eye opening and closing, blinking frequency, and mouth opening and closing, the problem of insufficient accuracy in fatigue detection of existing head-mounted devices is solved, enabling timely fatigue alerts.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-11
- Publication Date
- 2026-05-12
AI Technical Summary
Existing head-mounted devices lack accuracy in detecting user fatigue and are unable to promptly detect and remind users to rest.
By acquiring the wearer's facial data, including eye opening and closing, blinking frequency, and mouth opening and closing, a pre-set fatigue detection model is used for comprehensive analysis, outputting fatigue detection results and prompting the user.
It improves the accuracy of fatigue detection and can promptly remind users to take a break.
Smart Images
Figure CN122004749A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of head-mounted device technology, and more specifically, to a fatigue detection method for a head-mounted device, a fatigue detection device for a head-mounted device, and a head-mounted device. Background Technology
[0002] Currently, fatigue detection for users can be performed using head-mounted devices such as Virtual Reality (VR) devices and Augmented Reality (AR) devices. Typically, head-mounted devices can first detect the user's eye data and combine it with a fatigue threshold to determine whether the wearer is in a fatigued state. However, this method cannot accurately detect whether the wearer is in a fatigued state, resulting in the inability to detect and remind users who are fatigued in a timely manner. Summary of the Invention
[0003] The purpose of this disclosure is to provide a new technical solution for fatigue detection in head-mounted devices.
[0004] According to a first aspect of the present disclosure, a fatigue detection method for a head-mounted device is provided, the method comprising:
[0005] Acquire facial data of the wearer of the head-mounted device; wherein the facial data includes at least one of eye opening and closing, eye blinking frequency, and mouth opening and closing.
[0006] The fatigue detection results of the wearer are obtained based on the preset fatigue detection model and the facial data;
[0007] If the fatigue detection result indicates that the wearer is in a state of fatigue, a prompt message indicating that the wearer is in a state of fatigue will be output.
[0008] The preset fatigue detection model includes a first fatigue detection model, a second fatigue detection model, and a third fatigue detection model. The first fatigue detection model reflects the mapping relationship between eye opening and closing degree and fatigue detection results. The second fatigue detection model reflects the mapping relationship between eye blinking frequency and fatigue detection results. The third fatigue detection model reflects the mapping relationship between mouth opening and closing degree and fatigue detection results.
[0009] Optionally, the head-mounted device includes a camera device.
[0010] The acquisition of facial data of the wearer of the head-mounted device includes:
[0011] The camera device captures a facial image of the wearer.
[0012] The facial image is identified to obtain the wearer's facial data.
[0013] Optionally, the camera device includes at least a first camera, a second camera, a third camera, and a fourth camera, wherein the first camera is located at a first position on the head-mounted device, the second camera is located at a second position on the head-mounted device, the third camera is located at a third position on the head-mounted device, and the fourth camera is located at a fourth position on the head-mounted device, and the first position, the second position, the third position, and the fourth position are different positions of the head-mounted device.
[0014] Optionally, the eye opening degree includes the left eye opening degree and the right eye opening degree, the eye blinking frequency includes the left eye blinking frequency and the right eye blinking frequency, and the mouth opening degree includes the left corner of the mouth opening degree and the right corner of the mouth opening degree.
[0015] The process of identifying the facial image and obtaining the wearer's facial data includes:
[0016] Identify the first facial image captured by the first camera to obtain the opening degree of the left eye and the blinking frequency of the left eye;
[0017] The second facial image captured by the second camera is identified to obtain the opening degree of the right eye and the blinking frequency of the right eye.
[0018] Identify the third facial image captured by the third camera to obtain the degree of opening and closing of the left corner of the mouth; and,
[0019] The fourth facial image captured by the fourth camera is identified to obtain the degree of opening and closing of the right corner of the mouth.
[0020] Optionally, the facial data includes eye opening and closing.
[0021] The step of obtaining the wearer's fatigue detection result based on the preset fatigue detection model and the facial data includes:
[0022] Based on the eye opening degree, a vector value for a first feature vector reflecting the fatigue detection result is determined; wherein, the first feature vector includes at least one of the following features: current eye opening degree, rate of change of eye opening degree, cumulative change of eye opening degree, average eye opening degree, standard deviation of eye opening degree, maximum eye opening degree, minimum eye opening degree, and fluctuation of eye opening degree.
[0023] The fatigue detection result of the wearer is obtained based on the vector value and the first fatigue detection model.
[0024] Optionally, the facial data includes the frequency of eye blinks.
[0025] The step of obtaining the wearer's fatigue detection result based on the preset fatigue detection model and the facial data includes:
[0026] Based on the blink frequency, a vector value for a second feature vector reflecting the fatigue detection result is determined; wherein the second feature vector includes at least one of the current blink frequency and the average blink interval.
[0027] The fatigue detection results of the wearer are obtained based on the vector values and the second fatigue detection model.
[0028] Optionally, the facial data includes mouth opening degree.
[0029] The step of obtaining the wearer's fatigue detection result based on the preset fatigue detection model and the facial data includes:
[0030] Based on the mouth opening degree, a vector value for a third feature vector reflecting the fatigue detection result is determined; wherein, the third feature vector includes at least one of the following features: current mouth opening degree, mouth opening degree change rate, cumulative mouth opening degree change, average mouth opening degree, mouth opening degree standard deviation, maximum mouth opening degree, minimum mouth opening degree, and maximum mouth opening degree.
[0031] The fatigue detection results of the wearer are obtained based on the vector values and the third fatigue detection model.
[0032] Optionally, the facial data includes eye opening and closing degree, blinking frequency, and mouth opening and closing degree.
[0033] The step of obtaining the wearer's fatigue detection result based on the preset fatigue detection model and the facial data includes:
[0034] Based on the first fatigue detection model and the eye opening / closing degree, the first fatigue detection result of the wearer is obtained;
[0035] Based on the second fatigue detection model and the blinking frequency of the eyes, the second fatigue detection result of the wearer is obtained;
[0036] Based on the third fatigue detection model and the mouth opening degree, the third fatigue detection result of the wearer is obtained;
[0037] The fatigue detection result is obtained based on the first fatigue detection result, the second fatigue detection result, and the third fatigue detection result.
[0038] Optionally, the method further includes:
[0039] Obtain a first training sample set, a second training sample set, and a third training sample set; wherein, each first training sample in the first training sample set includes an eye opening and closing degree sample and a fatigue detection label for the eye opening and closing degree sample; each second training sample in the second training sample set includes an eye blinking frequency sample and a fatigue detection label for the eye blinking frequency sample; and each third training sample in the third training sample set includes a mouth opening and closing degree sample and a fatigue detection label for the mouth opening and closing degree sample.
[0040] Based on the first training sample set, the first fatigue detection model to be trained is trained to obtain the trained first fatigue detection model.
[0041] Based on the second training sample set, the second fatigue detection model to be trained is trained to obtain the trained second fatigue detection model; and...
[0042] Based on the third training sample set, the third fatigue detection model to be trained is trained to obtain the trained third fatigue detection model.
[0043] According to a second aspect of the present disclosure, a fatigue detection device for a head-mounted device is provided, the device comprising:
[0044] The acquisition module is used to acquire facial data of the wearer of the head-mounted device; wherein, the facial data includes eye opening and closing degree, eye blinking frequency and mouth opening and closing degree;
[0045] The fatigue detection module is used to obtain the fatigue detection result of the wearer based on the preset fatigue detection model and the facial data;
[0046] The prompting module is used to output a prompt message indicating that the wearer is in a state of fatigue when the fatigue detection result indicates that the wearer is in a state of fatigue.
[0047] The preset fatigue detection model includes a first fatigue detection model, a second fatigue detection model, and a third fatigue detection model. The first fatigue detection model reflects the mapping relationship between eye opening and closing degree and fatigue detection results. The second fatigue detection model reflects the mapping relationship between eye blinking frequency and fatigue detection results. The third fatigue detection model reflects the mapping relationship between mouth opening and closing degree and fatigue detection results.
[0048] One beneficial effect of this disclosure is that the head-mounted device can obtain the wearer's fatigue detection result based on the wearer's eye opening degree and a first fatigue detection model reflecting the mapping relationship between eye opening degree and fatigue detection result, and / or based on the wearer's eye blinking frequency and a second fatigue detection model reflecting the mapping relationship between eye blinking frequency and fatigue detection result, and / or based on the wearer's mouth opening degree and a third fatigue detection model reflecting the mapping relationship between mouth opening degree and fatigue detection result. This can effectively improve the accuracy of fatigue detection, so that when the fatigue detection result indicates that the user is in a fatigued state, the user can be promptly reminded to rest.
[0049] Other features and advantages of this specification will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description
[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of this specification and, together with their description, serve to explain the principles of this specification.
[0051] Figure 1 This is a schematic diagram of the hardware configuration of a head-mounted device according to an embodiment of the present disclosure;
[0052] Figure 2 This is a schematic flowchart of a fatigue detection method for a head-mounted device according to an embodiment of the present disclosure;
[0053] Figure 3 This is a schematic diagram of a user wearing a head-mounted device according to an embodiment of the present disclosure;
[0054] Figure 4a This is one of the schematic diagrams of eye opening and closing according to an embodiment of the present disclosure;
[0055] Figure 4b This is a second schematic diagram of eye opening and closing according to an embodiment of the present disclosure;
[0056] Figure 5a This is one of the schematic diagrams illustrating the blinking frequency of the eye according to an embodiment of the present disclosure;
[0057] Figure 5b This is a second schematic diagram of the blinking frequency of the eye according to an embodiment of the present disclosure;
[0058] Figure 6a This is one of the schematic diagrams of mouth opening and closing according to an embodiment of the present disclosure;
[0059] Figure 6b This is a second schematic diagram of the mouth opening degree according to an embodiment of the present disclosure.
[0060] Figure 7This is a schematic block diagram of a fatigue detection device for a head-mounted device according to an embodiment of the present disclosure;
[0061] Figure 8 This is a schematic block diagram of a head-mounted device according to an embodiment of the present disclosure. Detailed Implementation
[0062] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the embodiments of the present disclosure.
[0063] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.
[0064] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.
[0065] In all the examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.
[0066] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0067] <Hardware Configuration>
[0068] Figure 1 This is a block diagram of the hardware configuration of a head-mounted device 1000 according to an embodiment of the present disclosure.
[0069] The head-mounted device 1000 is, for example, but not limited to, an AR device or a VR device; this embodiment does not limit it to such devices.
[0070] like Figure 1As shown, the head-mounted device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a microphone 1700, and a speaker 1800, etc. The processor 1100 may include, but is not limited to, a central processing unit (CPU), a microprocessor (MCU), etc. The memory 1200 may include, for example, ROM (Read-Only Memory), RAM (Random Access Memory), non-volatile memory such as a hard disk, etc. The interface device 1300 may include, for example, various bus interfaces, such as serial bus interfaces (including USB interfaces), parallel bus interfaces, etc. The communication device 1400 may be capable of wired or wireless communication, for example. The display device 1500 includes a display component and a light source component. The input device 1600 may include, for example, a touch screen, a keyboard, a gamepad, etc. The microphone 1700 can be used to input voice information. The speaker 1800 can be used to output voice information.
[0071] In this embodiment, the memory 1200 of the head-mounted device 1000 is used to store instructions that control the processor 1100 to operate in order to implement or support the implementation of a fatigue detection method for the head-mounted device according to any embodiment. Those skilled in the art can design instructions based on the solutions disclosed in this specification. How the instructions control the processor to operate is well known in the art and will not be described in detail here.
[0072] Those skilled in the art should understand that, although in Figure 1 The present specification shows a number of devices of the head-mounted device 1000; however, the head-mounted device 1000 of the embodiments described herein may involve only some of the devices, or may include other devices, which is not limited herein.
[0073] Figure 1 The head-mounted device 1000 shown is for illustrative purposes only and is not intended to limit this specification, its application, or its use.
[0074] Hereinafter, various embodiments and examples according to the present disclosure will be described with reference to the accompanying drawings.
[0075] <Method Implementation>
[0076] Figure 2 This disclosure illustrates a fatigue detection method for a head-mounted device according to an embodiment of the present disclosure, such as... Figure 2 As shown, the fatigue detection method for the head-mounted device in this embodiment may include the following steps S2100 to S2300:
[0077] Step S2100: Obtain facial data of the wearer of the head-mounted device.
[0078] The facial data includes at least one of eye opening and closing, eye blinking frequency, and mouth opening and closing.
[0079] Generally speaking, eye opening degree can be understood as the degree to which the eyes open and close, including the opening degree of the left eye and the opening degree of the right eye.
[0080] Generally speaking, blinking frequency can be understood as the number of times the eyes blink, which includes the blinking frequency of the left eye and the blinking frequency of the right eye.
[0081] Generally speaking, mouth opening degree can be understood as the degree to which the mouth is open, which includes the opening degree of the left corner of the mouth and the opening degree of the right corner of the mouth.
[0082] In layman's terms, if a user is fatigued, their blinking frequency will increase and the blinking speed will decrease. At the same time, the user will also yawn, and when yawning, their mouth will open wide and remain in a certain position. Therefore, whether a user is fatigued can be determined by the degree of eye opening and closing, the frequency of eye blinking, and the degree of mouth opening and closing. This method is more comprehensive, accurate, and timely.
[0083] In this embodiment, the head-mounted device includes a camera device. Step S2100, which involves acquiring the facial data of the wearer of the head-mounted device, may further include: acquiring a facial image of the wearer captured by the camera device; recognizing the facial image to obtain the facial data of the wearer.
[0084] Reference Figure 3 Taking a head-mounted VR device as an example, the camera system of the head-mounted device includes at least a first camera 1, a second camera 2, a third camera 3, and a fourth camera 4. The first camera 1 is located in the first position on the head-mounted device, specifically on the left side of the device's outer shell, primarily for capturing the user's left eye. The second camera 2 is located in the second position on the head-mounted device, specifically on the right side of the device's outer shell, primarily for capturing the user's right eye. The third camera 3 is located in the third position on the head-mounted device, specifically on the lower left side of the device's outer shell, primarily for capturing the left half of the user's face. The fourth camera 4 is located in the fourth position on the head-mounted device, specifically on the lower right side of the device's outer shell, primarily for capturing the lower right half of the user's face. In other words, the first, second, third, and fourth positions mentioned above represent different locations on the head-mounted device. Figure 3 The black lines in the diagram represent the area that the camera can capture. In this embodiment, the head-mounted device, by using at least four cameras, can capture the user's facial expressions for more accurate fatigue detection.
[0085] The facial images typically include a first facial image captured by a first camera, a second facial image captured by a second camera, a third facial image captured by a third camera, and a fourth facial image captured by a fourth camera.
[0086] In an optional embodiment, the above-mentioned identification of the facial image to obtain the wearer's facial data may further include: identifying a first facial image captured by the first camera to obtain the left eye opening degree and the left eye blinking frequency; identifying a second facial image captured by the second camera to obtain the right eye opening degree and the right eye blinking frequency; identifying a third facial image captured by the third camera to obtain the wearer's left corner of mouth opening degree; and identifying a fourth facial image captured by the fourth camera to obtain the wearer's right corner of mouth opening degree.
[0087] Among them, eye opening and closing can include eye opening and closing at different times. That is, eye opening and closing can be understood as an eye opening and closing sequence, specifically including the left eye opening and closing sequence and the right eye opening and closing sequence.
[0088] Reference Figure 3 When a user wears a head-mounted device, such as a VR device, the head-mounted device can automatically activate the camera device, with the first camera 1, the second camera 2, the third camera 3, and the fourth camera 4 simultaneously taking pictures at time intervals T.
[0089] The following explanation uses the example of obtaining the left eye opening degree based on a first facial image captured by a first camera 1, and obtaining the right eye opening degree based on a second facial image captured by a second camera 2. If the user is wearing a head-mounted device, the first camera 1 captures a first facial image, and the second camera 2 captures a second facial image. The head-mounted device recognizes the captured first facial image to obtain the user's left eye opening degree α0, α1, α2, α3…αt for each time period T, and recognizes the captured second facial image to obtain the user's right eye opening degree α0', α1', α2', α3'…αt' for each time period T. (Referring to…) Figure 4a α0 represents the initial opening degree of the left eye, as referenced. Figure 4b α0' represents the initial opening degree of the right eye. Where αt represents the current opening degree of the left eye, and αt' represents the current opening degree of the right eye.
[0090] Among them, the blinking frequency is usually the blinking frequency of the eyes at different times. That is, the blinking frequency can be understood as a blinking frequency sequence, specifically including the blinking frequency sequence of the left eye and the blinking frequency sequence of the right eye.
[0091] Reference Figure 5a If the user wears a head-mounted device, the first camera 1 captures a first facial image. The head-mounted device then identifies the first facial image to obtain the angle β0 between the upper and lower eyelids of the left eye and the angle β0' between the upper eyelid of the left eye and the lowest point of the eyeball. Typically, the ratio of β0' / β0 is used as the proportion of the left eyeball covered by the upper eyelid. When the proportion of the left eyeball covered by the upper eyelid is 80%, it can be determined that the left eye is blinking.
[0092] Reference Figure 5b If the user wears a head-mounted device, the first camera 2 captures a second facial image. The head-mounted device then identifies the second facial image to obtain the angle γ0 between the upper and lower eyelids of the right eye and the angle γ0' between the upper eyelid of the right eye and the lowest point of the eyeball. Typically, the ratio of γ0' / γ0 is used as the proportion of the eyeball covered by the upper eyelid of the right eye. When the proportion of the eyeball covered by the upper eyelid of the right eye is 80%, it can be determined that the right eye is blinking.
[0093] Typically, within a time interval T, the head-mounted device, based on the first facial image captured by the first camera 1, identifies the number of times the left eye blinks from 80% to 80% of its eyeball covered by the upper eyelid, and then back to 80% of its eyeball covered by the upper eyelid. This number is the blink frequency of the left eye within that time interval T. Correspondingly, within a time interval T, the head-mounted device, based on the first facial image captured by the first camera 2, identifies the number of times the right eye blinks from 80% to 80% of its eyeball covered by the upper eyelid, and then back to 80% of its eyeball covered by the upper eyelid. This number is the blink frequency of the right eye within that time interval T.
[0094] The following explanation uses the example of obtaining the blink frequency of the left eye from a first facial image captured by a first camera 1, and obtaining the blink frequency of the right eye from a second facial image captured by a second camera 2. If the user is wearing a head-mounted device, the first camera 1 captures the first facial image, and the second camera 2 captures the second facial image. The head-mounted device recognizes the captured first facial image to obtain the user's left eye blink frequencies L1, L2, L3...Lt within each time period T, and recognizes the captured second facial image to obtain the user's right eye blink frequencies R1, R2, R3...Rt within each time period T. Here, Lt is the current blink frequency of the left eye, and Rt is the current blink frequency of the right eye.
[0095] Among them, mouth opening degree can include mouth opening degree at different times. That is, mouth opening degree can be understood as a mouth opening degree sequence, specifically including the left corner of the mouth opening degree sequence and the right corner of the mouth opening degree sequence.
[0096] The following explanation uses the example of obtaining the left corner of the mouth opening based on a third facial image captured by the third camera 3, and the right corner of the mouth opening based on a fourth facial image captured by the fourth camera 4. If the user is wearing a head-mounted device, the third camera 3 captures a third facial image, and the fourth camera 4 captures a fourth facial image. The head-mounted device will recognize the captured third facial image and obtain the user's left corner of the mouth opening θ0 for each time period T. 、 θ1, θ2, θ3...θn, and the fourth facial image captured by recognition, to obtain the opening degree θ0' of the right corner of the user's mouth for each time period T. 、θ1', θ2', θ3'...θn'. Where θ0 is the initial opening degree of the left corner of the mouth, and θ0' is the initial opening degree of the right corner of the mouth. (Refer to...) Figure 6a θ1 represents the opening degree of the left corner of the mouth within a time interval T, and θ1' represents the corresponding opening degree of the right corner of the mouth. When the mouth opens to a certain extent, refer to... Figure 6b θn represents the current opening degree of the left corner of the mouth, and θn' represents the current opening degree of the right corner of the mouth. The difference is determined by comparison. Figure 6b When the mouth shown is opened to a certain extent, the degree of opening of its corners is less than... Figure 6a The degree of mouth opening shown.
[0097] Step S2200: Obtain the fatigue detection result of the wearer based on the preset fatigue detection model and the facial data.
[0098] The preset fatigue detection model includes a first fatigue detection model, a second fatigue detection model, and a third fatigue detection model. The first fatigue detection model reflects the mapping relationship between eye opening and closing degree and fatigue detection results. The second fatigue detection model reflects the mapping relationship between eye blinking frequency and fatigue detection results. The third fatigue detection model reflects the mapping relationship between mouth opening and closing degree and fatigue detection results.
[0099] In an optional embodiment, the facial data includes eye opening and closing. Step S2200, based on a preset fatigue detection model and the facial data, to obtain the wearer's fatigue detection result, may further include the following steps S2210a to S2220a:
[0100] Step S2210a: Determine the vector value of the first feature vector reflecting the fatigue detection result based on the eye opening degree.
[0101] The first feature vector includes the current eye opening degree, the rate of change of eye opening degree, the cumulative change of eye opening degree, the average eye opening degree, the standard deviation of eye opening degree, the maximum eye opening degree, the minimum eye opening degree, and the fluctuation of eye opening degree.
[0102] For example, the first feature vector X = (x1, x2, x3, x4, x5, x7, x8), where x1 is the current eye opening degree, x2 is the rate of change of eye opening degree, x3 is the cumulative change of eye opening degree, x4 is the average eye opening degree, x5 is the standard deviation of eye opening degree, x6 is the maximum eye opening degree, x7 is the minimum eye opening degree, and x8 is the fluctuation of eye opening degree.
[0103] Step S2220a: Obtain the fatigue detection result of the wearer based on the vector value and the first fatigue detection model.
[0104] The fatigue detection result includes whether the wearer is in a fatigued state or not. For example, a fatigue detection result of "1" indicates that the wearer is in a fatigued state, while a fatigue detection result of "0" indicates that the wearer is not in a fatigued state.
[0105] For example, taking the first feature vector X = (x1, x2, x3, x4, x5, x7, x8) as an example, x1 represents the current eye opening degree α. t and α′ t x2 represents the rate of change in eye opening and closing as Δαt=αt-α(t-1) and Δαt′=αt′-α(t-1′), and x3 represents the cumulative rate of change in eye opening and closing as... and x4 represents the average eye opening and closing angles as mean(α0, α1……αt) and mean(α0′, α1′……αt′), x5 represents the standard deviation of eye opening and closing angles as std(α0, α1……αt) and std(α0′, α1′……αt′), x6 represents the maximum eye opening and closing angles as max(α0, α1……αt) and max(α0′, α1′……αt′), x7 represents the minimum eye opening and closing angles as min(α0, α1……αt) and min(α0′, α1′……αt′), and x8 represents the fluctuation of eye opening and closing angles as max(α0, α1……αt)-min(α0, α1……αt) and max(α0′, α1′……αt′)-min(α0′, α1′……αt′).
[0106] In step S2220a, the vector value of the first feature vector is substituted into the first fatigue detection model to obtain the fatigue detection result of the wearer.
[0107] Based on the above steps S2210a to S2220a, it can perform fatigue detection on the wearer based on eye opening and closing degree and the first fatigue detection model, which can effectively improve the accuracy of fatigue detection.
[0108] In an optional embodiment, the facial data includes the blinking frequency of the eyes. Step S2200, based on a preset fatigue detection model and the facial data, obtaining the fatigue detection result of the wearer, may further include the following steps S2210b to S2220b:
[0109] Step S2210b: Determine the vector value of the second feature vector that reflects the fatigue detection result based on the blinking frequency of the eyes.
[0110] The second feature vector includes at least one of the current blink frequency and the average blink interval.
[0111] Typically, the average blink interval is the ratio of the duration of the fatigue test to the total blink frequency of the eyes during the fatigue test.
[0112] Of course, the second feature vector may also include the maximum continuous non-blinking time, which is the longest non-blinking time found among all non-blinking time periods in this fatigue detection. The non-blinking time period is the time interval between two adjacent blinking events.
[0113] Step S2220b: Obtain the fatigue detection result of the wearer based on the vector value and the second fatigue detection result.
[0114] The fatigue detection result includes whether the wearer is in a fatigued state or not. For example, a fatigue detection result of "1" indicates that the wearer is in a fatigued state, while a fatigue detection result of "0" indicates that the wearer is not in a fatigued state.
[0115] In step S2220b, the vector value of the second feature vector is substituted into the second fatigue detection model to obtain the fatigue detection result of the wearer.
[0116] Based on the above steps S2210b to S2220b, it can perform fatigue detection on the wearer based on the blinking frequency and the second fatigue detection model, which can effectively improve the accuracy of fatigue detection.
[0117] In an optional embodiment, the facial data includes mouth opening degree. Step S2200, based on a preset fatigue detection model and the facial data, obtaining the wearer's fatigue detection result, may further include the following steps S2210c to S2220c:
[0118] Step S2210c: Determine the vector value of the third feature vector that reflects the fatigue detection result based on the mouth opening degree.
[0119] The third feature vector includes the current mouth opening degree, the rate of change of mouth opening degree, the cumulative change of mouth opening degree, the average mouth opening degree, the standard deviation of mouth opening degree, the maximum mouth opening degree, the minimum mouth opening degree, and the maximum mouth opening degree.
[0120] For example, the third feature vector X = (x1, x2, x3, x4, x5, x7, x8), where x1 is the current mouth opening degree, x2 is the rate of change of mouth opening degree, x3 is the cumulative change of mouth opening degree, x4 is the average mouth opening degree, x5 is the standard deviation of mouth opening degree, x6 is the maximum mouth opening degree, x7 is the minimum mouth opening degree, and x8 is the maximum mouth opening degree.
[0121] Step S2220c: Obtain the fatigue detection result of the wearer based on the vector value and the third fatigue detection result.
[0122] The fatigue detection result includes whether the wearer is in a fatigued state or not. For example, a fatigue detection result of "1" indicates that the wearer is in a fatigued state, while a fatigue detection result of "0" indicates that the wearer is not in a fatigued state.
[0123] For example, taking the third feature vector X = (x1, x2, x3, x4, x5, x7, x8) as an example, x1 represents the current mouth opening degree as θn and θn', x2 represents the rate of change of mouth opening degree as Δθn = θn - θ(n-1) and Δθn' = θn' - θ(n-1'), and x3 represents the cumulative change of mouth opening degree as... and x4 represents the average mouth opening as mean(θ0, θ1......θn) and mean(θ0′, θ1′......θn′), x5 represents the standard deviation of mouth opening as std(θ0, θ1......θn) and std(θ0′, θ1′......θn′), x6 represents the maximum mouth opening as max(θ0, θ1......θn) and max(θ0′, θ1′......θn′), x7 represents the minimum mouth opening as min(θ0, θ1......θn) and min(θ0′, θ1′......θn′), and x8 represents the fluctuation of mouth opening as max(θ0, θ1......θn)-min(θ0, θ1......θn) and max(θ0′, θ1′......θn′)-min(θ0′, θ1′......θn′).
[0124] In step S2220c, the vector value of the third feature vector is substituted into the third fatigue detection model to obtain the fatigue detection result of the wearer.
[0125] Based on the above steps S2210c to S2220c, it can perform fatigue detection on the wearer based on mouth opening degree and the third fatigue detection model, which can effectively improve the accuracy of fatigue detection.
[0126] In an optional embodiment, the facial data includes eye opening and closing degree, blinking frequency, and mouth opening and closing degree. Step S2200, based on a preset fatigue detection model and the facial data, obtaining the wearer's fatigue detection result, may further include the following steps S2210d to S2240d:
[0127] Step S2210d: Based on the first fatigue detection model and the eye opening / closing degree, obtain the first fatigue detection result of the wearer.
[0128] In step S2210d, the head-mounted device substitutes the vector value of the first feature vector into the first fatigue detection model in order to obtain the wearer's first fatigue detection result.
[0129] Step S2220d: Based on the second fatigue detection model and the blinking frequency of the eyes, obtain the second fatigue detection result of the wearer.
[0130] In step S2220d, the head-mounted device substitutes the vector value of the second feature vector into the second fatigue detection model to obtain the wearer's second fatigue detection result.
[0131] Step S2230d: Based on the third fatigue detection model and the mouth opening degree, obtain the third fatigue detection result of the wearer.
[0132] In step S2230d, the head-mounted device substitutes the vector value of the aforementioned third feature vector into the third fatigue detection model in order to obtain the wearer's third fatigue detection result.
[0133] Step S2240d: Obtain the fatigue detection result based on the first fatigue detection result, the second fatigue detection result, and the third fatigue detection result.
[0134] In step S2240d, the head-mounted device can obtain the wearer's fatigue detection results based on the first fatigue detection result, the second fatigue detection result, and the third fatigue detection result.
[0135] Based on the above steps S2210d to S2240d, the fatigue detection of the wearer is carried out by combining the eye opening and closing degree, the blinking frequency of the eyes and the mouth opening and closing degree, which can effectively improve the accuracy of fatigue detection.
[0136] Step S2300: If the fatigue detection result indicates that the wearer is in a fatigued state, output a prompt message indicating that the wearer is in a fatigued state.
[0137] The prompts can be text or voice prompts; this embodiment does not limit the types of prompts.
[0138] In this embodiment, when the fatigue detection result indicates that the wearer is in a fatigued state, the head-mounted device can output a prompt message indicating that the wearer is in a fatigued state, so as to prompt the wearer to rest in time based on the prompt message.
[0139] Through the embodiments of the present disclosure, the head-mounted device can obtain the fatigue detection result of the wearer according to the eye opening degree of the wearer and the first fatigue detection model reflecting the mapping relationship between the eye opening degree and the fatigue detection result, and / or according to the eye blinking frequency of the wearer and the second fatigue detection model reflecting the mapping relationship between the eye blinking frequency and the fatigue detection result, and / or according to the mouth opening degree of the wearer and the third fatigue detection model reflecting the mapping relationship between the mouth opening degree and the fatigue detection result, which can effectively improve the accuracy of fatigue detection, so as to timely remind the user to pay attention to rest when the fatigue detection result is in a fatigue state.
[0140] In one embodiment, before the head-mounted device leaves the factory, the first fatigue detection model, the second fatigue detection model and the third fatigue detection model can be obtained through pre-training. Here, the fatigue detection method of the head-mounted device according to the embodiments of the present disclosure further includes the following steps S3100 to step S3400:
[0141] Step S3100, obtain the first training sample set, the second training sample set and the third training sample set.
[0142] Each first training sample in the first training sample set includes an eye opening degree sample and a fatigue detection label of the eye opening degree sample.
[0143] Generally, the features in each eye opening degree sample form the above first feature vector. For example, the features of the eye opening degree sample are the first feature vector X = (x1, x2, x3, x4, x5, x7, x8), where x1 is the current eye opening degree, x2 is the eye opening degree change rate, x3 is the cumulative eye opening degree change, x4 is the average eye opening degree, x5 is the standard deviation of the eye opening degree, x6 is the maximum eye opening degree, x7 is the minimum eye opening degree, and x8 is the eye opening degree fluctuation.
[0144] Generally, the fatigue detection label of the eye opening degree sample can be 0 or 1. Generally speaking, the more fatigued a person is, the smaller the eye opening degree. Threshold0 and Threshold0’ are the thresholds for judging whether the user is fatigued. When (αt < Threshold0) ∧ (αt’ < Threshold0’), it is judged that the user is in a fatigued state, that is, the fatigue detection label of the corresponding eye opening degree sample is 1. Otherwise, the fatigue detection label of the corresponding eye opening degree sample is 0.
[0145] Each second training sample in the second training sample set includes an eye blinking frequency sample and a fatigue detection label of the eye blinking frequency sample.
[0146] Generally, the features in each eye blinking frequency sample form the above second feature vector.
[0147] Generally, the fatigue detection label of the eye blink frequency sample can be 0 or 1. Threshold1 and Threshold1’ are the thresholds for judging whether the user is fatigued. When (αt>Threshold1)∧(αt’>Threshold1’) is satisfied, it is judged that the user is in a fatigued state, that is, the fatigue detection label of the corresponding eye blink frequency sample is 1. Otherwise, the fatigue detection label of the corresponding eye blink frequency sample is 0.
[0148] Among them, each third training sample in the third training sample set includes a mouth opening and closing degree sample and the fatigue detection label of the mouth opening and closing degree sample.
[0149] Generally, the features in each mouth opening and closing degree sample form the above-mentioned second feature vector. For example, the features of the mouth opening and closing degree sample, that is, the second feature vector X=(x1, x2, x3, x4, x5, x7, x8), where x1 is the current mouth opening and closing degree, x2 is the change rate of the mouth opening and closing degree, x3 is the cumulative change of the mouth opening and closing degree, x4 is the average mouth opening and closing degree, x5 is the standard deviation of the mouth opening and closing degree, x6 is the maximum mouth opening and closing degree, x7 is the minimum mouth opening and closing degree, and x8 is the maximum mouth opening and closing degree.
[0150] Generally, the fatigue detection label of the mouth opening and closing degree sample can be 0 or 1. Threshold2 and Threshold2’ are the thresholds for judging whether the user is fatigued. When (θn<Threshold2)∧(θn’<Threshold2’) is satisfied, it is judged that the user is in a fatigued state, that is, the fatigue detection label of the corresponding mouth opening and closing degree sample is 1. Otherwise, the fatigue detection label of the corresponding mouth opening and closing degree sample is 0.
[0151] Step S3200: Train the to-be-trained first fatigue detection model according to the first training sample set to obtain the trained first fatigue detection model.
[0152] Step S3300: Train the to-be-trained second fatigue detection model according to the second training sample set to obtain the trained second fatigue detection model.
[0153] Step S3400: Train the to-be-trained third fatigue detection model according to the third training sample set to obtain the trained third fatigue detection model.
[0154] It should be noted that the above-mentioned first fatigue detection model, second fatigue detection model, and third fatigue detection model can all be understood as binary classification models. When training, a logistic regression algorithm can be used as the binary classification algorithm, and a random forest algorithm can be used to improve the accuracy and robustness of model training.
[0155] According to this embodiment, a first fatigue detection model based on eye opening and closing degree, a second fatigue detection model based on eye blinking frequency, and a third fatigue detection model based on mouth opening and closing degree can be constructed respectively, so as to perform fatigue detection on the wearer based on the first fatigue detection model, the second fatigue detection model, and the third fatigue detection model respectively.
[0156] <Device Embodiment>
[0157] Figure 7 This is a schematic diagram of a fatigue detection device for a head-mounted device according to one embodiment, with reference to... Figure 7 As shown, the fatigue detection device 700 of the head-mounted device includes an acquisition module 710, a fatigue detection module 720, and a prompting module 730.
[0158] The acquisition module 710 is used to acquire facial data of the wearer of the head-mounted device; wherein, the facial data includes eye opening and closing degree, eye blinking frequency and mouth opening and closing degree;
[0159] The fatigue detection module 720 is used to obtain the fatigue detection result of the wearer based on the preset fatigue detection model and the facial data;
[0160] The prompting module 730 is used to output a prompt message indicating that the wearer is in a fatigued state when the fatigue detection result indicates that the wearer is in a fatigued state.
[0161] The preset fatigue detection model includes a first fatigue detection model, a second fatigue detection model, and a third fatigue detection model. The first fatigue detection model reflects the mapping relationship between eye opening and closing degree and fatigue detection results. The second fatigue detection model reflects the mapping relationship between eye blinking frequency and fatigue detection results. The third fatigue detection model reflects the mapping relationship between mouth opening and closing degree and fatigue detection results.
[0162] In one embodiment, the head-mounted device includes a camera device and an acquisition module 710, specifically used to acquire a facial image of the wearer captured by the camera device; identify the facial image, and obtain facial data of the wearer.
[0163] In one embodiment, the camera device includes at least a first camera, a second camera, a third camera, and a fourth camera. The first camera is located at a first position on the head-mounted device, the second camera is located at a second position on the head-mounted device, the third camera is located at a third position on the head-mounted device, and the fourth camera is located at a fourth position on the head-mounted device. The first position, the second position, the third position, and the fourth position are different positions of the head-mounted device.
[0164] In one embodiment, the eye opening degree includes the left eye opening degree and the right eye opening degree, the eye blinking frequency includes the left eye blinking frequency and the right eye blinking frequency, and the mouth opening degree includes the left corner of the mouth opening degree and the right corner of the mouth opening degree. The acquisition module 710 is specifically used to: obtain the left eye blinking frequency; identify the second facial image captured by the second camera to obtain the right eye opening degree and the right eye blinking frequency; identify the third facial image captured by the third camera to obtain the left corner of the mouth opening degree; and identify the fourth facial image captured by the fourth camera to obtain the right corner of the mouth opening degree.
[0165] In one embodiment, the facial data includes eye opening and closing. The fatigue detection module 720 is specifically used to determine a vector value of a first feature vector reflecting the fatigue detection result based on the eye opening and closing. The first feature vector includes at least one of the following features: current eye opening and closing, rate of change of eye opening and closing, cumulative change of eye opening and closing, average eye opening and closing, standard deviation of eye opening and closing, maximum eye opening and closing, minimum eye opening and closing, and fluctuation of eye opening and closing. The fatigue detection result of the wearer is obtained based on the vector value and the first fatigue detection model.
[0166] In one embodiment, the facial data includes eye blinking frequency. The fatigue detection module 720 is specifically used to determine the vector value of a second feature vector reflecting the fatigue detection result based on the eye blinking frequency. The second feature vector includes at least one of the current eye blinking frequency and the average blinking interval. The fatigue detection result of the wearer is obtained based on the vector value and the second fatigue detection model.
[0167] In one embodiment, the facial data includes mouth opening degree. The fatigue detection module 720 is specifically used to determine the vector value of a third feature vector reflecting the fatigue detection result based on the mouth opening degree. The third feature vector includes at least one of the following features: current mouth opening degree, mouth opening degree change rate, cumulative mouth opening degree change, average mouth opening degree, mouth opening degree standard deviation, maximum mouth opening degree, minimum mouth opening degree, and maximum mouth opening degree. The fatigue detection result of the wearer is obtained based on the vector value and the third fatigue detection model.
[0168] In one embodiment, the facial data includes eye opening and closing degree, eye blinking frequency, and mouth opening and closing degree. The fatigue detection module 720 is specifically used to obtain a first fatigue detection result of the wearer based on the first fatigue detection model and the eye opening and closing degree; to obtain a second fatigue detection result of the wearer based on the second fatigue detection model and the eye blinking frequency; to obtain a third fatigue detection result of the wearer based on the third fatigue detection model and the mouth opening and closing degree; and to obtain the fatigue detection result based on the first fatigue detection result, the second fatigue detection result, and the third fatigue detection result.
[0169] In one embodiment, the device 700 further includes a training module (not shown). The training module is configured to acquire a first training sample set, a second training sample set, and a third training sample set; wherein each first training sample in the first training sample set includes an eye opening / closing sample and a fatigue detection label for the eye opening / closing sample; each second training sample in the second training sample set includes an eye blinking frequency sample and a fatigue detection label for the eye blinking frequency sample; and each third training sample in the third training sample set includes a mouth opening / closing sample and a fatigue detection label for the mouth opening / closing sample; the module trains a first fatigue detection model to be trained based on the first training sample set to obtain a trained first fatigue detection model; trains a second fatigue detection model to be trained based on the second training sample set to obtain a trained second fatigue detection model; and trains a third fatigue detection model to be trained based on the third training sample set to obtain a trained third fatigue detection model.
[0170] According to embodiments of this disclosure, the head-mounted device can obtain the wearer's fatigue detection results based on the wearer's eye opening and closing degree and a first fatigue detection model reflecting the mapping relationship between eye opening and closing degree and fatigue detection results, and / or based on the wearer's eye blinking frequency and a second fatigue detection model reflecting the mapping relationship between eye blinking frequency and fatigue detection results, and / or based on the wearer's mouth opening and closing degree and a third fatigue detection model reflecting the mapping relationship between mouth opening and closing degree and fatigue detection results. This can effectively improve the accuracy of fatigue detection, so that when the fatigue detection result indicates that the user is in a fatigued state, the device can promptly remind the user to take a rest.
[0171] <Equipment Example>
[0172] Figure 8 This is a schematic diagram of the hardware structure of a head-mounted device according to one embodiment. For example... Figure 8 As shown, the head-mounted device 800 includes a processor 810 and a memory 820.
[0173] The memory 820 can be used to store executable computer instructions.
[0174] The processor 810 can be used to execute the fatigue detection method for a head-mounted device according to the method embodiments of this disclosure, under the control of the executable computer instructions.
[0175] The head-mounted device 800 can be as follows: Figure 1 The head-mounted device 1000 shown may also be a device with other hardware structures, which is not limited here.
[0176] In another embodiment, the head-mounted device 800 may include the fatigue detection device 700 of the head-mounted device described above.
[0177] In one embodiment, each module of the fatigue detection device 700 of the above-mentioned head-mounted device can be implemented by the processor 810 running computer instructions stored in the memory 820.
[0178] Computer-readable storage media
[0179] This disclosure also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, perform the fatigue detection method for a head-mounted device provided in this disclosure.
[0180] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.
[0181] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.
[0182] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.
[0183] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.
[0184] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.
[0185] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0186] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0187] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are equivalent.
[0188] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.
Claims
1. A fatigue detection method for a head-mounted device, characterized in that, The method includes: Acquire facial data of the wearer of the head-mounted device; wherein the facial data includes at least one of eye opening and closing, eye blinking frequency, and mouth opening and closing. The fatigue detection results of the wearer are obtained based on the preset fatigue detection model and the facial data; If the fatigue detection result indicates that the wearer is in a state of fatigue, a prompt message indicating that the wearer is in a state of fatigue will be output. The preset fatigue detection model includes a first fatigue detection model, a second fatigue detection model, and a third fatigue detection model. The first fatigue detection model reflects the mapping relationship between eye opening and closing degree and fatigue detection results. The second fatigue detection model reflects the mapping relationship between eye blinking frequency and fatigue detection results. The third fatigue detection model reflects the mapping relationship between mouth opening and closing degree and fatigue detection results.
2. The method according to claim 1, characterized in that, The head-mounted device includes a camera unit. The acquisition of facial data of the wearer of the head-mounted device includes: The camera device captures a facial image of the wearer. The facial image is identified to obtain the wearer's facial data.
3. The method according to claim 2, characterized in that, The camera device includes at least a first camera, a second camera, a third camera, and a fourth camera. The first camera is located at a first position on the head-mounted device, the second camera is located at a second position on the head-mounted device, the third camera is located at a third position on the head-mounted device, and the fourth camera is located at a fourth position on the head-mounted device. The first position, the second position, the third position, and the fourth position are different positions on the head-mounted device.
4. The method according to claim 3, characterized in that, The eye opening degree includes the opening degree of the left eye and the opening degree of the right eye; the eye blinking frequency includes the blinking frequency of the left eye and the blinking frequency of the right eye; the mouth opening degree includes the opening degree of the left corner of the mouth and the opening degree of the right corner of the mouth. The process of identifying the facial image and obtaining the wearer's facial data includes: Identify the first facial image captured by the first camera to obtain the opening degree of the left eye and the blinking frequency of the left eye; The second facial image captured by the second camera is identified to obtain the opening degree of the right eye and the blinking frequency of the right eye. Identify the third facial image captured by the third camera to obtain the degree of opening and closing of the left corner of the mouth; and, The fourth facial image captured by the fourth camera is identified to obtain the degree of opening and closing of the right corner of the mouth.
5. The method according to claim 1, characterized in that, The facial data includes eye opening and closing. The step of obtaining the wearer's fatigue detection result based on the preset fatigue detection model and the facial data includes: Based on the eye opening degree, a vector value for a first feature vector reflecting the fatigue detection result is determined; wherein, the first feature vector includes at least one of the following features: current eye opening degree, rate of change of eye opening degree, cumulative change of eye opening degree, average eye opening degree, standard deviation of eye opening degree, maximum eye opening degree, minimum eye opening degree, and fluctuation of eye opening degree. The fatigue detection result of the wearer is obtained based on the vector value and the first fatigue detection model.
6. The method according to claim 1, characterized in that, The facial data includes the frequency of eye blinking. The step of obtaining the wearer's fatigue detection result based on the preset fatigue detection model and the facial data includes: Based on the blink frequency, a vector value for a second feature vector reflecting the fatigue detection result is determined; wherein the second feature vector includes at least one of the current blink frequency and the average blink interval. The fatigue detection results of the wearer are obtained based on the vector values and the second fatigue detection model.
7. The method according to claim 1, characterized in that, The facial data includes the degree of mouth opening. The step of obtaining the wearer's fatigue detection result based on the preset fatigue detection model and the facial data includes: Based on the mouth opening degree, a vector value for a third feature vector reflecting the fatigue detection result is determined; wherein, the third feature vector includes at least one of the following features: current mouth opening degree, mouth opening degree change rate, cumulative mouth opening degree change, average mouth opening degree, mouth opening degree standard deviation, maximum mouth opening degree, minimum mouth opening degree, and maximum mouth opening degree. The fatigue detection results of the wearer are obtained based on the vector values and the third fatigue detection model.
8. The method according to claim 1, characterized in that, The facial data includes eye opening and closing, blinking frequency, and mouth opening and closing. The step of obtaining the wearer's fatigue detection result based on the preset fatigue detection model and the facial data includes: Based on the first fatigue detection model and the eye opening / closing degree, the first fatigue detection result of the wearer is obtained; Based on the second fatigue detection model and the blinking frequency of the eyes, the second fatigue detection result of the wearer is obtained; Based on the third fatigue detection model and the mouth opening degree, the third fatigue detection result of the wearer is obtained; The fatigue detection result is obtained based on the first fatigue detection result, the second fatigue detection result, and the third fatigue detection result.
9. The method according to claim 1, characterized in that, The method further includes: Obtain a first training sample set, a second training sample set, and a third training sample set; wherein, each first training sample in the first training sample set includes an eye opening and closing degree sample and a fatigue detection label for the eye opening and closing degree sample; each second training sample in the second training sample set includes an eye blinking frequency sample and a fatigue detection label for the eye blinking frequency sample; and each third training sample in the third training sample set includes a mouth opening and closing degree sample and a fatigue detection label for the mouth opening and closing degree sample. Based on the first training sample set, the first fatigue detection model to be trained is trained to obtain the trained first fatigue detection model. Based on the second training sample set, the second fatigue detection model to be trained is trained to obtain the trained second fatigue detection model; and... Based on the third training sample set, the third fatigue detection model to be trained is trained to obtain the trained third fatigue detection model.
10. A fatigue detection device for a head-mounted device, characterized in that, The device includes: The acquisition module is used to acquire facial data of the wearer of the head-mounted device; wherein, the facial data includes eye opening and closing degree, eye blinking frequency and mouth opening and closing degree; The fatigue detection module is used to obtain the fatigue detection result of the wearer based on the preset fatigue detection model and the facial data; The prompting module is used to output a prompt message indicating that the wearer is in a state of fatigue when the fatigue detection result indicates that the wearer is in a state of fatigue. The preset fatigue detection model includes a first fatigue detection model, a second fatigue detection model, and a third fatigue detection model. The first fatigue detection model reflects the mapping relationship between eye opening and closing degree and fatigue detection results. The second fatigue detection model reflects the mapping relationship between eye blinking frequency and fatigue detection results. The third fatigue detection model reflects the mapping relationship between mouth opening and closing degree and fatigue detection results.