Eye behavior monitoring methods, devices, and media

Abstract

The application discloses an eye use behavior monitoring method, device and medium, and relates to the technical field of head-mounted devices. The method is applied to a head-mounted device, the head-mounted device is provided with an eye use behavior data collection unit and an eye image collection unit, and the method comprises the following steps: detecting whether a wearer of the head-mounted device continuously exists in an unhealthy eye use behavior within a first preset time length by using eye use behavior data collected by the eye use behavior data collection unit; in the case that the wearer continuously exists in the unhealthy eye use behavior within the first preset time length, detecting whether the wearer continuously exists in an electronic screen watching behavior within a second preset time length by using an eye reflection image collected by the eye image collection unit; and outputting a reminding information in the case that the wearer continuously exists in the electronic screen watching behavior within the second preset time length. The method can effectively monitor the eye use behavior of the wearer.

Description

Technical Field

[0001] This application relates to the field of head-mounted display technology, and more specifically, to a method, device, and medium for monitoring eye behavior. Background Technology

[0002] With the widespread use of electronic products and increasing academic pressure, the rate of myopia among teenagers is rising globally. Myopia in teenagers not only directly affects their physical and mental health but also poses a significant public health challenge. Therefore, how to monitor teenagers' eye-use behavior to achieve myopia prevention and control has become an urgent technical problem to be solved. Summary of the Invention

[0003] One objective of this application is to provide a new technical solution for monitoring eye use behavior.

[0004] According to a first aspect of this application, a method for monitoring eye use behavior is provided, the method being applied to a head-mounted device, the head-mounted device being equipped with an eye use behavior data acquisition unit and an eye image acquisition unit, the method comprising: The eye behavior data collected by the eye behavior data collection unit is used to detect whether the wearer of the head-mounted device continuously exhibits poor eye behavior within a first preset time period. If unhealthy eye-use behavior persists within the first preset time period, the eye reflection image acquired by the eye image acquisition unit is used to detect whether the wearer continues to view electronic screens within the second preset time period. If the user continues to view the electronic screen within the second preset time period, a reminder message will be output.

[0005] Optionally, the eye-use behavior data acquisition unit includes at least one of an inertial measurement unit, a first distance detection unit, and an ambient light detection unit. The step of detecting whether the wearer of the head-mounted device continuously exhibits poor eye-use behavior within a first preset time period based on the eye-use behavior data acquired by the eye-use behavior data acquisition unit includes: If, within a first preset duration, at least one of the following conditions is met: the wearer is in a poor posture state based on the inertial measurement data collected by the inertial measurement unit; the wearer is in a dynamic head-down eye-use state based on the inertial measurement data; the actual distance between the wearer and the object being gazed at is less than a first preset distance based on the first distance detection unit; and the actual ambient brightness is less than a preset ambient brightness based on the ambient light brightness detection unit, then it is determined that poor eye-use behavior persists within the first preset duration.

[0006] Optionally, the step of detecting whether the wearer continuously views an electronic screen within a second preset time period using the eye reflection image acquired by the eye image acquisition unit includes: Within a second preset duration, a judgment operation is performed according to a first set time interval to obtain a judgment result. The judgment operation includes: waking up the eye image acquisition unit, controlling the eye image acquisition unit to continuously acquire multiple frames of eye reflection images and then controlling the eye image acquisition unit to go into sleep mode, determining whether there are any associated features of an electronic screen in the multiple frames of eye reflection images, and determining that the judgment result is that there is an act of watching an electronic screen if there are associated features of an electronic screen. If the result of each of the aforementioned judgment operations is that there is a viewing behavior on an electronic screen, it is determined that the viewing behavior on an electronic screen continues to exist within a second preset time period. If at least one judgment operation results in no screen viewing behavior, it is determined that there is no screen viewing behavior within the second preset time period or that screen viewing behavior has not continued within the second preset time period. The screen viewing behavior data collected by the screen viewing behavior data collection unit is repeated to detect whether the wearer of the head-mounted device has continued to have poor screen viewing behavior within the first preset time period.

[0007] Optionally, after outputting the reminder information, the method further includes: If the wearer exhibits corrective behavior, the eye behavior data collected by the eye behavior data collection unit is repeated to detect whether the wearer of the head-mounted device continuously exhibits poor eye behavior within a first preset time period. If the eye behavior data collected by the eye behavior data collection unit determines that the wearer does not exhibit poor eye behavior, it is determined that the wearer exhibits corrective behavior. If the wearer does not perform any corrective actions, at least one of the following is executed: outputting the reminder information at a second set time interval, sending an uncorrected event to a terminal device communicatively connected to the head-mounted device, and sending the cumulative duration of viewing the electronic screen to the terminal device.

[0008] Optionally, before detecting whether the wearer of the head-mounted device continuously exhibits poor eye-use behavior within a first preset time period, the method further includes: Detect whether the head-mounted device is being worn; In this case, the eye-use behavior data acquisition unit is activated; If not, the eye behavior data acquisition unit is put into a sleep state.

[0009] Optionally, the head-mounted device is provided with a second distance detection unit, and the detection of whether the head-mounted device is being worn includes: If the inertial sensing data collected by the inertial measurement unit determines that the head-mounted device is stationary within a third preset time period, and the actual distance between the head-mounted device and the wearer's face detected by the second distance detection unit is greater than a second preset distance, then the head-mounted device is determined to be in a non-wearing state. If the actual distance detected by the second distance detection unit is less than or equal to the second preset distance, and / or if the inertial sensing data collected by the inertial measurement unit determines that the head-mounted device is in a non-stationary state within a third preset time period, then the head-mounted device is determined to be in a wearing state.

[0010] Optionally, the method further includes: If the wearer does not have or does not continuously engage in any unhealthy eye-use behavior within the first preset time period, a non-electronic screen unhealthy eye-use behavior log is generated. The non-electronic screen unhealthy eye-use behavior log includes non-electronic screen unhealthy eye-use behavior events, the time of occurrence of the non-electronic screen unhealthy eye-use behavior events, and the cumulative duration of the non-electronic screen unhealthy eye-use behavior events. Based on the non-electronic screen poor eye use behavior log within the fourth preset time period, generate a non-electronic screen poor eye use behavior analysis report and / or eye use suggestions.

[0011] Optionally, determining whether there are associated features of an electronic screen in the multi-frame eye reflection images includes: For any frame of the multi-frame eye reflection image, determine whether there are any associated features of the electronic screen in the eye reflection image, wherein the associated features include at least a light spot that conforms to the shape of the electronic screen; If, in a predetermined proportion of the multi-frame eye reflection images, there are associated features of an electronic screen, then it is determined that there are associated features of an electronic screen in the multi-frame eye reflection images.

[0012] According to a second aspect of this application, an eye behavior monitoring device is provided, the device being applied to a head-mounted device, the head-mounted device being provided with an eye behavior data acquisition unit and an eye image acquisition unit, the device comprising: The first detection module is used to detect whether the wearer of the head-mounted device continuously exhibits poor eye-use behavior within a first preset time period by using eye-use behavior data collected by the eye-use behavior data collection unit. The second detection module is used to detect whether the wearer has been continuously viewing an electronic screen for a second preset time period when there is continuous poor eye use behavior within the first preset time period, by using the eye reflection image acquired by the eye image acquisition unit. The output module is used to output a reminder message when there is continuous viewing of the electronic screen within the second preset time period.

[0013] According to a third aspect of this application, a head-mounted device is provided, the head-mounted device comprising the means as described in the second aspect; Alternatively, the head-mounted device may include an eye behavior data acquisition unit, an eye image acquisition unit, a memory, and a processor. The eye behavior data acquisition unit is used to acquire eye behavior data, the eye image acquisition unit is used to acquire eye reflection images, the memory is used to store computer instructions, and the processor is used to retrieve the computer instructions from the memory to perform the method as described in any one of the first aspects.

[0014] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of the first aspects.

[0015] This embodiment provides a method for monitoring eye use behavior. The method is applied to a head-mounted device, which includes an eye use behavior data acquisition unit and an eye image acquisition unit. The method includes: detecting whether the wearer of the head-mounted device continuously engages in unhealthy eye use behavior within a first preset time period using eye use behavior data acquired by the eye use behavior data acquisition unit; detecting whether the wearer continuously engages in screen viewing behavior within a second preset time period using eye reflection images acquired by the eye image acquisition unit if unhealthy eye use behavior is continuously observed within the second preset time period; and outputting a reminder message if screen viewing behavior is continuously observed within the second preset time period. This method can monitor the events causing unhealthy eye use behavior when it is detected that the wearer is continuously engaging in such behavior for a certain period. When the event causing the unhealthy eye use behavior is screen viewing, the unhealthy eye use behavior is determined to be a high-risk screen-related unhealthy eye use behavior leading to myopia, and a reminder message is output to remind the wearer to stop the high-risk screen-related unhealthy eye use behavior. This allows for effective monitoring of the wearer's eye use behavior. Furthermore, this method does not capture images of the environment, thus avoiding the recording of images in the environment that involve the privacy of others, thereby reducing the limitations on the use cases of head-mounted devices.

[0016] Other features and advantages of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present application and, together with their description, serve to explain the principles of the present application.

[0018] Figure 1 This is a flowchart illustrating an eye behavior monitoring method provided in an embodiment of this application. Figure 1 ; Figure 2 This is a flowchart illustrating an eye behavior monitoring method provided in an embodiment of this application. Figure 2 ; Figure 3 This is a schematic diagram of the structure of an eye behavior monitoring device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a head-mounted device provided in an embodiment of this application. Detailed Implementation

[0019] Various exemplary embodiments of this application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this application.

[0020] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0021] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0022] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0023] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0024] This application provides a method for monitoring eye use behavior. This method is applied to a head-mounted device and can monitor the events causing the undesirable eye use behavior when it is detected that the wearer is continuously engaging in such behavior over a certain period. When the detected event causing the undesirable eye use behavior is viewing an electronic screen, the method determines that the undesirable eye use behavior is a high-risk behavior that could lead to myopia and outputs a reminder message to prompt the wearer to stop the high-risk undesirable eye use behavior.

[0025] like Figure 1 As shown, the eye behavior monitoring method provided in this application includes the following steps S1100 to S1300.

[0026] Step S1100: Using eye behavior data collected by the eye behavior data collection unit, detect whether the wearer of the head-mounted device continuously exhibits poor eye behavior within a first preset time period.

[0027] In one embodiment of this application, the eye behavior data acquisition unit includes at least one of an inertial measurement unit, a first distance detection unit, and an ambient light intensity detection unit. Based on this, the above step S1100 is specifically implemented through the following step S1110.

[0028] Step S1110: If, within a first preset duration, the following conditions are met continuously: the wearer is determined to be in an unhealthy posture state based on the inertial measurement data collected by the inertial measurement unit; the wearer is determined to be in a dynamic head-down eye-use state based on the inertial measurement data; the actual distance between the wearer and the object being gazed at is less than a first preset distance; and the actual ambient brightness is less than a preset ambient brightness, then it is determined that unhealthy eye-use behavior continues to exist within the first preset duration.

[0029] In this embodiment, the head-mounted device includes an inertial measurement unit, a first distance detection unit, and an ambient light detection unit. The inertial measurement unit collects inertial measurement data from the head-mounted device; specifically, this data refers to the inertial measurement data of the wearer's head. The first distance detection unit collects the distance between the wearer and an object in the environment. That is, the distance collection direction of the first distance detection unit is opposite to the wearer's gaze direction, aligned with the wearer's line of sight. In one example, the first distance detection unit may be located on the front side of the head-mounted device's frame, and is a time-of-flight (ToF) distance sensor. The ambient light detection unit collects the actual ambient light of the environment in which the wearer is located. In one example, the ambient light detection unit is an ambient light sensor (ALS).

[0030] The inertial measurement unit (IMU) includes an accelerometer and a gyroscope. Based on this, the IMU collects inertial measurement data including three-axis acceleration data and three-axis angular velocity data. Regarding the three-axis acceleration data, if the acceleration data matches that of a walking state, the wearer is determined to be walking; if the acceleration data matches that of a stationary state, the wearer is determined to be stationary. Regarding the three-axis angular velocity data, if the angular velocity data matches that of a pre-set head-down angle, the wearer is determined to be in a head-down position. If the wearer is both walking and in a head-down position, the wearer is determined to be in a dynamic head-down eye-use state. If the wearer is both stationary and in a head-down position, the wearer is determined to be in an unhealthy posture state. It is understandable that both dynamic head-down eye-use and unhealthy posture states are considered poor eye-use behaviors.

[0031] For the first distance detection unit, the distance detected by the first distance detection unit is recorded as the actual distance between the wearer and the object being gazed at. The first preset distance is the safe eye-use distance, and the value range of the first preset distance is 30~40cm. If the actual distance between the wearer and the object being gazed at is less than the first preset distance, it indicates that the wearer is not using their eyes safely due to close-range eye use, and it is determined that the wearer has poor eye-use behavior.

[0032] For the ambient light detection unit, the ambient light level detected by the unit is used as the actual ambient light level. The preset ambient light level is a recommended level for healthy eye use, and its value ranges from 100 to 2000 Lux. If the actual ambient light level is lower than the preset level, it indicates that the ambient light is insufficient, and it is determined that the wearer has engaged in improper eye-use behavior.

[0033] Based on the above, if at any given moment the following conditions are met: the wearer is in an unhealthy posture state as determined by inertial measurement data collected by the inertial measurement unit; the wearer is in a dynamic head-down eye-use state as determined by inertial measurement data; the actual distance between the wearer and the object being gazed at is less than a first preset distance as determined by the first distance detection unit; and the actual ambient brightness is less than a preset ambient brightness as determined by the ambient light brightness detection unit, then the wearer is determined to have engaged in unhealthy eye-use behavior. Furthermore, the duration of the wearer's unhealthy eye-use behavior is accumulated. If the duration reaches a first preset duration, then the unhealthy eye-use behavior is determined to have persisted for the first preset duration.

[0034] In one example, the first preset duration is 5 minutes. Corresponding to step S1100 above, if no undesirable eye-use behavior is observed within the first preset duration, it is determined that the wearer is using their eyes safely. Step S1100 is then repeated to detect whether the wearer continues to exhibit undesirable eye-use behavior within the first preset duration. Furthermore, if undesirable eye-use behavior is initially identified within the first preset duration but is subsequently interrupted, it is determined that undesirable eye-use behavior is not continuously observed within the first preset duration. Step S1100 is then repeated to detect whether the wearer continues to exhibit undesirable eye-use behavior within the first preset duration.

[0035] For step S1100 above, step S1200 is triggered only if undesirable eye-use behavior persists for a first preset duration, in order to determine whether the wearer's undesirable eye-use behavior is caused by viewing an electronic screen. This implements an anti-shake mechanism that effectively avoids misidentifying brief actions such as looking down to pick something up as undesirable eye-use behavior, thereby preventing the frequent triggering of step S1200.

[0036] If, based on step S1100 above, it is determined that the wearer of the head-mounted device continuously exhibits poor eye-use behavior within a first preset time period, the following step S1200 is executed to determine whether the wearer's poor eye-use behavior is caused by viewing an electronic screen.

[0037] Step S1200: If there is continuous poor eye-use behavior within the first preset time period, the wearer is detected by using the eye reflection image acquired by the eye image acquisition unit to determine whether the wearer is continuously viewing an electronic screen within the second preset time period.

[0038] In this embodiment, the eye image acquisition unit is used to acquire images emitted from the corneal surface to obtain an eye reflection image. That is, the image acquisition direction of the eye image acquisition unit is towards the wearer's eyes. In one example, the eye image acquisition unit can be placed on the side of the head-mounted device's frame facing the wearer's eyes. Because the image acquisition direction of the eye image acquisition unit is towards the wearer's eyes, the eye behavior monitoring method provided in this application does not acquire images of the environment. This avoids recording images in the environment that involve the privacy of others, thereby reducing the limitations on the use scenarios of the head-mounted device. For example, in the case of a smart glasses device for teenagers, since it does not acquire images of the environment, the head-mounted device will not be prohibited from being brought into the school.

[0039] In one embodiment of this application, the above step S1200 is specifically implemented by the following steps S1210 to S1230.

[0040] Step S1210: Perform a judgment operation within the second preset time period according to the first set time interval to obtain a judgment result.

[0041] The judgment operation is implemented through the following steps S1211 to S1214.

[0042] Step S1211: Wake up the eye image acquisition unit.

[0043] In this embodiment, the eye image acquisition unit is only awakened when a judgment operation is performed once. That is, the eye image acquisition unit remains in sleep mode unless necessary. This reduces the power consumption of the head-mounted device and extends its standby time. For small-sized head-mounted devices, reducing power consumption to extend standby time is crucial.

[0044] Step S1212: After controlling the eye image acquisition unit to continuously acquire multiple frames of eye reflection images, control the eye image acquisition unit to go into sleep mode.

[0045] In one example, the number of frames corresponding to multiple eye reflection images is specifically within the range of 15 to 30 frames. After waking up the eye image acquisition unit, the unit is controlled to continuously acquire multiple frames of eye reflection images, which are used to obtain the judgment result of the corresponding judgment operation. After controlling the eye image acquisition unit to continuously acquire multiple frames of eye reflection images, the unit is then put into sleep mode.

[0046] Step S1213: Determine whether there are associated features of the electronic screen in the multi-frame eye images.

[0047] In one embodiment of this application, the above step S1213 is specifically implemented through the following steps S1213-1 and S1213-2.

[0048] Step S1213-1: For any frame in the multi-frame eye reflection image, determine whether there are associated features of the electronic screen in the eye reflection image.

[0049] Among them, the associated features include at least a light spot that conforms to the shape of the electronic screen.

[0050] In this embodiment, for any frame of the eye reflection image, the corneal region is first extracted to obtain a corneal region image. Further, the corneal region image undergoes preprocessing such as distortion correction and contrast enhancement to obtain a processed corneal region image. Next, for the processed corneal region image, a spot extraction algorithm is used to determine whether a spot exists in the eye reflection image. If no spot exists, it is determined that there are no associated features of an electronic screen in the eye reflection image. If a spot exists, an edge detection algorithm is used to find closed rectangular and / or trapezoidal spots with clear corners and straight edges that conform to the shape of an electronic screen. If a closed rectangular and / or trapezoidal spot with clear corners and straight edges that conforms to the shape of an electronic screen can be found, it is determined that there are associated features of an electronic screen in the eye reflection image. It should be noted that due to viewing angle and other factors, the electronic screen may appear trapezoidal in the eye reflection image formed by the eye.

[0051] Of course, other related characteristics of electronic screens can also include the spectrum, such as the spectral characteristics that match those of an electronic screen. Specifically, different types of electronic screens have different spectra. For example, for LCD screens, white light mainly comes from blue LEDs exciting yellow phosphors, so the spectrum has a distinct peak in the blue light region and a broad peak in the green-yellow region. OLED screens, on the other hand, emit light from individual RGB organic light-emitting diodes, and their spectrum consists of three relatively narrow peaks that combine to form white light.

[0052] Step S1213-2: If there are associated features of an electronic screen in the eye reflection images of a preset proportion in the multi-frame eye reflection images, determine that there are associated features of an electronic screen in the multi-frame eye reflection images.

[0053] The preset ratio is, for example, fifty percent. If the preset ratio of eye reflection images in the multi-frame eye reflection images contains associated features of an electronic screen, then it is determined that the multi-frame eye reflection images contain associated features of an electronic screen.

[0054] Due to issues such as the image acquisition quality of the eye image acquisition unit, it is impossible to accurately determine whether there are associated features of an electronic screen in a single frame of eye reflection image. Therefore, in steps S1213-1 and S1213-2, multiple frames of eye reflection images are used to comprehensively determine whether there are associated features of an electronic screen in multiple frames of eye reflection images. This overcomes the problem of being unable to accurately determine whether there are associated features of an electronic screen in a single frame of eye reflection image due to issues such as the image acquisition quality of the eye image acquisition unit.

[0055] Step S1220: If the result of each judgment operation is that there is a viewing behavior on the electronic screen, it is determined that the viewing behavior on the electronic screen continues to exist within the second preset time period.

[0056] In one example, the second preset duration is 5 minutes.

[0057] If, within the second preset time period, the result of each judgment operation performed at the first preset time interval is that the behavior of viewing an electronic screen exists, then it is determined that the behavior of viewing an electronic screen continues within the second preset time period. At this time, it is determined that the wearer is engaging in high-risk poor eye-use behavior related to electronic screens, and the following step S1300 is executed.

[0058] Step S1230: If the result of at least one judgment operation is that there is no viewing of electronic screens, determine that there is no viewing of electronic screens within the second preset time period or that there is no continuous viewing of electronic screens within the second preset time period, and repeatedly detect whether the wearer of the head-mounted device has continuous poor eye behavior by collecting eye behavior data through the eye behavior data acquisition unit.

[0059] Corresponding to step S1220 above, if the result of at least one judgment operation is that there is no behavior of viewing an electronic screen, it means that the wearer briefly viewed an electronic screen or did not view an electronic screen at all. Poor eye use behavior is caused by behaviors other than viewing an electronic screen, such as doing homework at close range or reading in dim light. In this case, step S1100 above is repeated.

[0060] In one embodiment of this application, corresponding to step S1200 above, the eye behavior monitoring method provided by this application further includes the following steps S1240 and S1250.

[0061] Step S1240: If the wearer does not have or does not continuously engage in any undesirable eye-use behavior within the first preset time period, a non-electronic screen undesirable eye-use behavior log is generated. The non-electronic screen undesirable eye-use behavior log includes non-electronic screen undesirable eye-use behavior events, the time of occurrence of non-electronic screen undesirable eye-use behavior events, and the cumulative duration of non-electronic screen undesirable eye-use behavior events.

[0062] Step S1250: Based on the non-electronic screen poor eye use behavior log within the fourth preset time period, generate a non-electronic screen poor eye use behavior analysis report and / or eye use suggestions.

[0063] In one example, the fourth preset duration is one week. In one embodiment of this application, step S1250 can be implemented by calling a large AI model.

[0064] In this embodiment, step S1240 generates a log of poor eye use behavior outside of electronic screens, enabling timely and detailed recording of such behavior. Furthermore, step S1250 allows for analysis of the log and / or provision of eye use suggestions based on it.

[0065] Step S1300: If the behavior of viewing the electronic screen continues within the second preset time period, a reminder message is output.

[0066] If the user continues to view electronic screens for a second preset duration, it indicates that the user is engaging in high-risk screen-related eye-use behavior. At this point, a reminder message is output to intervene in the user's high-risk screen-related eye-use behavior. In one example, the reminder message includes a vibration and / or a voice message such as "Please rest your eyes."

[0067] This embodiment provides a method for monitoring eye use behavior. The method is applied to a head-mounted device, which includes an eye use behavior data acquisition unit and an eye image acquisition unit. The method includes: detecting whether the wearer of the head-mounted device continuously engages in unhealthy eye use behavior within a first preset time period using eye use behavior data acquired by the eye use behavior data acquisition unit; detecting whether the wearer continuously engages in screen viewing behavior within a second preset time period using eye reflection images acquired by the eye image acquisition unit if unhealthy eye use behavior is continuously observed within the second preset time period; and outputting a reminder message if screen viewing behavior is continuously observed within the second preset time period. This method can monitor the events causing unhealthy eye use behavior when it is detected that the wearer is continuously engaging in such behavior for a certain period. When the event causing the unhealthy eye use behavior is screen viewing, the unhealthy eye use behavior is determined to be a high-risk screen-related unhealthy eye use behavior leading to myopia, and a reminder message is output to remind the wearer to stop the high-risk screen-related unhealthy eye use behavior. This allows for effective monitoring of the wearer's eye use behavior. Furthermore, this method does not capture images of the environment, thus avoiding the recording of images in the environment that involve the privacy of others, thereby reducing the limitations on the use cases of head-mounted devices.

[0068] In one embodiment of this application, after the head-mounted device outputs a reminder message, the wearer may have the possibility of correcting their eye-use behavior. Therefore, the eye-use behavior monitoring method provided in this application further includes the following steps S1400 and S1500 after step S1300.

[0069] Step S1400: If the wearer exhibits corrective behavior, the eye behavior data collected by the eye behavior data acquisition unit is repeatedly used to detect whether the wearer of the head-mounted device continues to exhibit poor eye behavior within a first preset time period.

[0070] In cases where eye behavior data collected by the eye behavior data collection unit confirms that the wearer does not have any poor eye behavior, it is determined that the wearer has corrective behaviors.

[0071] In this embodiment, after a certain period of time following the output of the prompt message, such as 30 seconds, eye-use behavior data is re-acquired to determine whether the wearer has any improper eye-use behavior. If no improper eye-use behavior is found, it indicates that the wearer corrected their eye-use behavior after receiving the prompt message. At this time, step S1100 is repeated to conduct a new round of eye-use behavior monitoring.

[0072] In step S1500, if the wearer does not perform any corrective behavior, at least one of the following is executed: outputting a reminder message at a second set time interval, sending an uncorrected event to a terminal device that is communicatively connected to the head-mounted device, and sending the cumulative duration of viewing the electronic screen to the terminal device.

[0073] Corresponding to step S1400 above, if the wearer does not engage in corrective behavior, a reminder message is output at a second preset time interval, for example, every 30 seconds, to frequently remind the wearer of high-risk screen-related eye-use behaviors. And / or, an uncorrected event is sent to a terminal device connected to the head-mounted device, allowing the person holding the terminal device (usually a parent) to manually remind the wearer. And / or, the cumulative duration of screen viewing is sent to the terminal device, allowing the person holding the terminal device to understand the duration of the wearer's high-risk screen-related eye-use behaviors.

[0074] In one embodiment of this application, in order to reduce the power consumption of the head-mounted device and extend its standby time, the eye behavior monitoring method provided in this application further includes the following steps S1120 to S1140 before the above step S1100.

[0075] Step S1120: Monitor whether the head-mounted device is being worn.

[0076] In one embodiment of this application, a contact sensor is provided on the head-mounted device. If the contact sensor detects a contact, the head-mounted device is determined to be in a wearing state. Otherwise, the head-mounted device is determined to be in a non-wearing state.

[0077] In another embodiment of this application, the head-mounted device is provided with a second distance detection unit. In this case, the above step S1120 is specifically implemented by the following steps S1121 and S1122.

[0078] Step S1121: If the inertial sensing data collected by the inertial measurement unit determines that the head-mounted device is stationary within a third preset time period, and the actual distance between the head-mounted device and the wearer's face detected by the second distance detection unit is greater than the second preset distance, then the head-mounted device is determined to be in a non-wearing state.

[0079] In this embodiment, the second distance detection device is used to detect the distance between the head-mounted device and the wearer's face. That is, the distance acquisition direction of the second distance detection unit is towards the side of the wearer wearing the head-mounted device. The second preset distance is the maximum distance between the wearer and the second distance detection device when the wearer is wearing the head-mounted device. In one example, the second preset distance is 3cm. Furthermore, the third preset duration is exemplarily 10 minutes.

[0080] Understandably, if the actual distance detected by the second distance detection unit is greater than the second preset distance, it is determined that the wearer is not wearing the head-mounted device. Simultaneously, if the inertial sensing data collected by the inertial measurement unit determines that the head-mounted device is stationary within a third preset time period, it indicates that the head-mounted device has been stationary for an extended period. At this point, it is further determined that the wearer is not wearing the head-mounted device. Thus, through the aforementioned step S1121, a dual judgment is achieved, accurately determining the wearing status of the head-mounted device.

[0081] Step S1122: If the actual distance detected by the second distance detection unit is less than or equal to the second preset distance, and / or the inertial sensing data collected by the inertial measurement unit determines that the head-mounted device is in a non-stationary state within the third preset time period, then the head-mounted device is determined to be in a wearing state.

[0082] Step S1130: If so, wake up the eye behavior data acquisition unit.

[0083] Step S1140: If not, control the eye behavior data acquisition unit to be in sleep mode.

[0084] After determining whether the head-mounted device is being worn based on step S1120, if the head-mounted device is being worn, the eye behavior data acquisition unit is woken up to collect eye behavior data; otherwise, the eye behavior data acquisition unit is put into sleep mode. This avoids unnecessary wake-ups of the eye behavior data acquisition unit, thereby reducing the power consumption of the head-mounted device and extending its standby time.

[0085] Based on the above embodiments, this application also provides another method for monitoring eye use behavior, such as... Figure 2 As shown, the process includes steps S2001 to S2010.

[0086] Step S2001: Detect whether the head-mounted device is being worn. If yes, proceed to step S2002 below; if no, proceed to step S2003 below. Step S2002: Wake up the eye behavior data acquisition unit, and then execute the following step S2004; Step S2003: Control the eye behavior data acquisition unit to be in sleep mode.

[0087] Step S2004: Using the eye behavior data collected by the eye behavior data collection unit, detect whether the wearer of the head-mounted device continuously exhibits poor eye behavior within a first preset time period. If yes, execute the following step S2005; otherwise, repeat the above step S2004. Step S2005: Using the eye reflection image acquired by the eye image acquisition unit, detect whether the wearer continuously views the electronic screen within the second preset time period. If yes, execute the following step S2008. If no, repeat the above step S2004, as well as the following steps S2006 and S2007. Step S2006: Generate a log of poor eye use behavior on non-electronic screens; Step S2007: Based on the non-electronic screen poor eye use behavior log within the fourth preset time period, generate a non-electronic screen poor eye use behavior analysis report and / or eye use suggestions; Step S2008: Output a reminder message, then proceed to step S2009; Step S2009: Detect whether the wearer has any corrective behavior. If yes, repeat step S2004 above. If no, proceed to step S2011 below.

[0088] Step S2010 involves executing at least one of the following: outputting a reminder message at a second set time interval, sending an uncorrected event to a terminal device connected to the head-mounted device, and sending the cumulative duration of viewing the electronic screen to the terminal device.

[0089] like Figure 3 As shown, this application also provides an eye behavior monitoring device 300, which is applied to a head-mounted device. The head-mounted device is equipped with an eye behavior data acquisition unit and an eye image acquisition unit. The device 300 includes: The first detection module 310 is used to detect whether the wearer of the head-mounted device continuously exhibits poor eye-use behavior within a first preset time period by using eye-use behavior data collected by the eye-use behavior data collection unit. The second detection module 320 is used to detect whether the wearer has been continuously viewing an electronic screen for a second preset time period when there is continuous poor eye behavior within the first preset time period, by using the eye reflection image acquired by the eye image acquisition unit. The output module 330 is used to output a reminder message when there is continuous viewing of the electronic screen within the second preset time period.

[0090] In one embodiment of this application, the eye-use behavior data acquisition unit includes at least one of an inertial measurement unit, a first distance detection unit, and an ambient light intensity detection unit. The first detection module 310 is specifically used for: If, within a first preset duration, at least one of the following conditions is met: the wearer is in a poor posture state based on the inertial measurement data collected by the inertial measurement unit; the wearer is in a dynamic head-down eye-use state based on the inertial measurement data; the actual distance between the wearer and the object being gazed at is less than a first preset distance based on the first distance detection unit; and the actual ambient brightness is less than a preset ambient brightness based on the ambient light brightness detection unit, then it is determined that poor eye-use behavior persists within the first preset duration.

[0091] In one embodiment of this application, the second detection module 320 is specifically used for: Within a second preset duration, a judgment operation is performed according to a first set time interval to obtain a judgment result. The judgment operation includes: waking up the eye image acquisition unit, controlling the eye image acquisition unit to continuously acquire multiple frames of eye reflection images and then controlling the eye image acquisition unit to go into sleep mode, determining whether there are any associated features of an electronic screen in the multiple frames of eye reflection images, and determining that the judgment result is that there is an act of watching an electronic screen if there are associated features of an electronic screen. If the result of each of the aforementioned judgment operations is that there is a viewing behavior on an electronic screen, it is determined that the viewing behavior on an electronic screen continues to exist within a second preset time period. If at least one judgment operation results in no screen viewing behavior, it is determined that there is no screen viewing behavior within the second preset time period or that screen viewing behavior has not continued within the second preset time period. The screen viewing behavior data collected by the screen viewing behavior data collection unit is repeated to detect whether the wearer of the head-mounted device has continued to have poor screen viewing behavior within the first preset time period.

[0092] In one embodiment of this application, the eye behavior monitoring device 300 provided in this application further includes: The correction detection module is used to, when the wearer exhibits corrective behavior, repeat the eye behavior data collected by the eye behavior data acquisition unit to detect whether the wearer of the head-mounted device continuously exhibits poor eye behavior within a first preset time period. If the eye behavior data collected by the eye behavior data acquisition unit determines that the wearer does not exhibit poor eye behavior, then the wearer is determined to have corrective behavior. If the wearer does not perform any corrective actions, at least one of the following is executed: outputting the reminder information at a second set time interval, sending an uncorrected event to a terminal device communicatively connected to the head-mounted device, and sending the cumulative duration of viewing the electronic screen to the terminal device.

[0093] In one embodiment of this application, the eye behavior monitoring device 300 provided in this application further includes: A wear detection module is used to detect whether the head-mounted device is being worn. In this case, the eye-use behavior data acquisition unit is activated; If not, the eye behavior data acquisition unit is put into a sleep state.

[0094] In one embodiment of this application, the head-mounted device is provided with a second distance detection unit and a wearing detection module, specifically used to: determine that the head-mounted device is in a non-wearing state when the inertial sensing data collected by the inertial measurement unit determines that the head-mounted device is stationary within a third preset time period, and the actual distance between the head-mounted device and the wearer's face detected by the second distance detection unit is greater than a second preset distance; If the actual distance detected by the second distance detection unit is less than or equal to the second preset distance, and / or if the inertial sensing data collected by the inertial measurement unit determines that the head-mounted device is in a non-stationary state within a third preset time period, then the head-mounted device is determined to be in a wearing state.

[0095] In one embodiment of this application, the eye behavior monitoring device 300 provided in this application further includes: The analysis module is used to generate a non-electronic screen poor eye use behavior log when the wearer does not have or does not continuously have poor eye use behavior within the first preset time period. The non-electronic screen poor eye use behavior log includes non-electronic screen poor eye use behavior events, the time when the non-electronic screen poor eye use behavior events occur, and the cumulative duration of the non-electronic screen poor eye use behavior events. Based on the non-electronic screen poor eye use behavior log within the fourth preset time period, generate a non-electronic screen poor eye use behavior analysis report and / or eye use suggestions.

[0096] In one embodiment of this application, the second detection module 320 is specifically used for: For any frame of the multi-frame eye reflection image, determine whether there are any associated features of the electronic screen in the eye reflection image, wherein the associated features include at least a light spot that conforms to the shape of the electronic screen; If, in a predetermined proportion of the multi-frame eye reflection images, there are associated features of an electronic screen, then it is determined that there are associated features of an electronic screen in the multi-frame eye reflection images.

[0097] This application also provides a head-mounted device, which includes any of the eye behavior monitoring devices 300 provided in the above-described device embodiments.

[0098] This application also provides another head-mounted device 400, such as Figure 4 As shown, the head-mounted device 400 includes an eye behavior data acquisition unit 410, an eye image acquisition unit 420, a memory 430, and a processor 440. The eye behavior data acquisition unit 410 is used to acquire eye behavior data, the eye image acquisition unit 420 is used to acquire eye reflection images, the memory 430 is used to store computer instructions, and the processor 440 is used to retrieve the computer instructions from the memory 430 to execute the method as described in any of the above method embodiments.

[0099] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of the above-described method embodiments.

[0100] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0101] This application may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement any of the methods in the foregoing embodiments of this application.

[0102] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media may include, for example, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), compact disc-read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any combination thereof. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0103] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include one or more of copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to computer-readable storage media in the respective computing / processing device.

[0104] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object programs written in any combination of one or more programming languages. These programming languages ​​may include object-oriented programming languages ​​(e.g., Smalltalk, C++, etc.) and conventional procedural programming languages ​​(e.g., the "C" language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network (e.g., a local area network or a wide area network), or it may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays, or programmable logic arrays, are personalized using state information from the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of the embodiments of this application.

[0105] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0106] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0107] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It should be noted that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are all equivalent.

[0109] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this application is defined by the appended claims.

Claims

1. A method for monitoring eye use behavior, characterized in that, The method is applied to a head-mounted device, the head-mounted device being equipped with an eye behavior data acquisition unit and an eye image acquisition unit, the method comprising: The eye behavior data collected by the eye behavior data collection unit is used to detect whether the wearer of the head-mounted device continuously exhibits poor eye behavior within a first preset time period. If unhealthy eye-use behavior persists within the first preset time period, the eye reflection image acquired by the eye image acquisition unit is used to detect whether the wearer continues to view electronic screens within the second preset time period. If the user continues to view the electronic screen within the second preset time period, a reminder message will be output.

2. The method according to claim 1, characterized in that, The eye-use behavior data acquisition unit includes at least one of an inertial measurement unit, a first distance detection unit, and an ambient light intensity detection unit. The eye-use behavior data acquired by the eye-use behavior data acquisition unit is used to detect whether the wearer of the head-mounted device continuously exhibits poor eye-use behavior within a first preset time period, including: If, within a first preset duration, at least one of the following conditions is met: the wearer is in a poor posture state based on the inertial measurement data collected by the inertial measurement unit; the wearer is in a dynamic head-down eye-use state based on the inertial measurement data; the actual distance between the wearer and the object being gazed at is less than a first preset distance based on the first distance detection unit; and the actual ambient brightness is less than a preset ambient brightness based on the ambient light brightness detection unit, then it is determined that poor eye-use behavior persists within the first preset duration.

3. The method according to claim 1, characterized in that, The detection of whether the wearer continuously views an electronic screen within a second preset time period using the eye reflection image acquired by the eye image acquisition unit includes: Within a second preset duration, a judgment operation is performed according to a first set time interval to obtain a judgment result. The judgment operation includes: waking up the eye image acquisition unit, controlling the eye image acquisition unit to continuously acquire multiple frames of eye reflection images and then controlling the eye image acquisition unit to go into sleep mode, determining whether there are any associated features of an electronic screen in the multiple frames of eye reflection images, and determining that the judgment result is that there is an act of watching an electronic screen if there are associated features of an electronic screen. If the result of each of the aforementioned judgment operations is that there is a viewing behavior on an electronic screen, it is determined that the viewing behavior on an electronic screen continues to exist within a second preset time period. If at least one judgment operation results in no screen viewing behavior, it is determined that there is no screen viewing behavior within the second preset time period or that screen viewing behavior has not continued within the second preset time period. The screen viewing behavior data collected by the screen viewing behavior data collection unit is repeated to detect whether the wearer of the head-mounted device has continued to have poor screen viewing behavior within the first preset time period.

4. The method according to claim 1, characterized in that, The method further includes, after the output reminder information: If the wearer exhibits corrective behavior, the eye behavior data collected by the eye behavior data collection unit is repeated to detect whether the wearer of the head-mounted device continuously exhibits poor eye behavior within a first preset time period. If the eye behavior data collected by the eye behavior data collection unit determines that the wearer does not exhibit poor eye behavior, it is determined that the wearer exhibits corrective behavior. If the wearer does not perform any corrective actions, at least one of the following is executed: outputting the reminder information at a second set time interval, sending an uncorrected event to a terminal device communicatively connected to the head-mounted device, and sending the cumulative duration of viewing the electronic screen to the terminal device.

5. The method according to claim 1, characterized in that, Before detecting whether the wearer of the head-mounted device continuously exhibits poor eye-use behavior within a first preset time period, the method further includes: Detect whether the head-mounted device is being worn; In this case, the eye-use behavior data acquisition unit is activated; If not, the eye behavior data acquisition unit is controlled to be in a sleep state.

6. The method according to claim 5, characterized in that, The head-mounted device is equipped with a second distance detection unit, and the detection of whether the head-mounted device is being worn includes: If the inertial sensing data collected by the inertial measurement unit determines that the head-mounted device is stationary within a third preset time period, and the actual distance between the head-mounted device and the wearer's face detected by the second distance detection unit is greater than a second preset distance, then the head-mounted device is determined to be in a non-wearing state. If the actual distance detected by the second distance detection unit is less than or equal to the second preset distance, and / or if the inertial sensing data collected by the inertial measurement unit determines that the head-mounted device is in a non-stationary state within a third preset time period, then the head-mounted device is determined to be in a wearing state.

7. The method according to claim 1, characterized in that, The method further includes: If the wearer does not have or does not continuously engage in any unhealthy eye-use behavior within the first preset time period, a non-electronic screen unhealthy eye-use behavior log is generated. The non-electronic screen unhealthy eye-use behavior log includes non-electronic screen unhealthy eye-use behavior events, the time of occurrence of the non-electronic screen unhealthy eye-use behavior events, and the cumulative duration of the non-electronic screen unhealthy eye-use behavior events. Based on the non-electronic screen poor eye use behavior log within the fourth preset time period, generate a non-electronic screen poor eye use behavior analysis report and / or eye use suggestions.

8. The method according to claim 3, characterized in that, Determining whether there are electronic screen-related features in the multi-frame eye reflection images includes: For any frame of the multi-frame eye reflection image, determine whether there are any associated features of the electronic screen in the eye reflection image, wherein the associated features include at least a light spot that conforms to the shape of the electronic screen; If, in a predetermined proportion of the multi-frame eye reflection images, there are associated features of an electronic screen, then it is determined that there are associated features of an electronic screen in the multi-frame eye reflection images.

9. A head-mounted device, characterized in that, The head-mounted device includes an eye behavior data acquisition unit, an eye image acquisition unit, a memory, and a processor. The eye behavior data acquisition unit is used to acquire eye behavior data, the eye image acquisition unit is used to acquire eye reflection images, the memory is used to store computer instructions, and the processor is used to retrieve the computer instructions from the memory to execute the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images