State sensing method, wearable device, storage medium and program product

Abstract

The invention discloses a state sensing method, wearable equipment, a storage medium and a program product, and relates to the technical field of wearable equipment, the state sensing method is applied to the wearable equipment, and the method comprises the following steps: determining whether a user fixation point is in a content display range of the wearable equipment through a preset eye movement tracking module; under the condition that the user fixation point is in the content display range, monitoring the fixation time of the user fixation point; determining whether the attention of the user is concentrated in a content display range or not according to whether the watching time exceeds a preset time threshold value or not; and outputting external prompt information under the condition that the attention of the user is focused on the content display range. According to the method and the device, the user state is sensed in real time, and the external prompt information which can be sensed by surrounding people is output, so that the external interaction use convenience of the wearable equipment is improved.

Description

Technical Field

[0001] This application relates to the field of wearable device technology, and more particularly to state sensing methods, wearable devices, storage media, and program products. Background Technology

[0002] In recent years, with the advancement of hardware and software algorithms, wearable devices have become more lightweight and discreet in appearance, and the difference between them and ordinary glasses has been decreasing. This allows users to use wearable devices continuously without attracting attention, thus seamlessly integrating them into social, work, and life scenarios.

[0003] However, due to the discreet nature of wearable devices, it may be difficult for outsiders to judge the wearer's current state of attention. This can lead to misunderstandings or communication breakdowns in social situations, such as when pedestrians meet or when customers and sales associates interact.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a state-aware method, wearable device, storage medium, and program product, aiming to solve the technical problem of how to improve the ease of use of wearable devices for external interaction.

[0006] To achieve the above objectives, this application proposes a state-aware method, which includes: The system uses a preset eye-tracking module to determine whether the user's gaze point is within the content display range of the wearable device. When the user's gaze point is within the content display area, monitor the gaze duration of the user's gaze point; Based on whether the gaze duration exceeds a preset time threshold, it is determined whether the user's attention is focused on the content display area; When the user's attention is focused on the content display area, an external prompt message is output.

[0007] In one embodiment, the eye-tracking module includes an infrared light source and a camera. The step of determining whether the user's gaze point is within the content display range of the wearable device using a preset eye-tracking module includes: The infrared light source generates infrared light that illuminates the user's eyes, and the camera captures an image of the eyes. The gaze direction is determined using a preset pupil-corneal reflex method and the eye image. Based on the gaze direction and the content display range, determine whether the user's gaze point is within the content display range.

[0008] In one embodiment, the external notification information includes a first invisible light signal and a visible light signal, and the step of outputting the external notification information includes: A first invisible light signal is emitted to other wearable devices via a pre-set invisible light emitter, wherein the first invisible light signal includes an infrared light signal; and / or, Generate a light command, and control the indicator light of the wearable device to emit a visible light signal according to the light command.

[0009] In one embodiment, the method further includes: When the user's attention is focused on the content display area, the user's real-time location is determined according to a preset positioning system; If the real-time location is in a preset dangerous location, the display content of the wearable device is turned off, and / or the real-time location is uploaded to a preset cloud network.

[0010] In one embodiment, the method further includes: When the user's attention is focused on the content display area, a second invisible light signal is received in real time through a preset invisible light sensor; Based on the frequency band of the second invisible light signal, determine whether the second invisible light signal is an emergency contact signal; If the second invisible light signal is an emergency contact signal, an internal alarm message is output.

[0011] In one embodiment, the method further includes: While the user's attention is focused on the content display area, a video stream is captured in real time; The video stream is used to detect whether a hand image appears using a pre-trained machine learning model. When a hand image appears in the video stream, the hand action is obtained based on the motion trajectory of the hand key points in the hand image in consecutive frames of the video stream. If the hand gesture is a preset waving gesture, an internal alarm message is output.

[0012] In one embodiment, the method further includes: In response to a triggered sharing command, determine the content to be shared and generate a sharing signal based on the content to be shared; The sharing signal is sent to a preset sharing user, wherein the sharing user is determined based on the sharing instruction.

[0013] Furthermore, to achieve the above objectives, this application also proposes a state sensing device, which includes: An eye-tracking module is used to determine whether the user's gaze point is within the content display range of the wearable device through a preset eye-tracking module. A gaze duration monitoring module is used to monitor the gaze duration of the user's gaze point when the user's gaze point is within the content display range; An attention judgment model is used to determine whether the user's attention is focused on the content display area based on whether the gaze time exceeds a preset time threshold. The information output module is used to output external prompts when the user's attention is focused on the content display area.

[0014] In addition, to achieve the above objectives, this application also proposes a wearable device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the state-aware method as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the state-aware method as described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the state-aware method as described above.

[0017] The technical solutions proposed in this application, including one or more, have at least the following technical effects: First, by using a preset eye-tracking module, it determines whether the user's gaze point is within the content display range of the wearable device. By capturing the user's gaze focus in real time, it directly distinguishes whether the user is observing the external real environment or focusing on the virtual content provided by the device, providing a basis for subsequent attention judgment. Second, when the user's gaze point is within the content display range, it monitors the user's gaze duration and determines whether the user's attention is focused on the content display range based on whether the gaze duration exceeds a preset time threshold. By quantifying the user's continuous gaze behavior, it provides an objective basis for attention judgment, avoiding misjudgments caused by brief scanning or unintentional glances, and improving the accuracy of user state perception. Third, when the user's attention is focused on the content display range, it outputs external prompts to transform the internal, invisible user attention state into a visible and perceptible signal to the external environment. Thus, people around can clearly know that the user is focused on virtual content based on these external prompts and take appropriate actions such as avoiding them, not disturbing them, or finding non-contact methods to wake them up, thereby improving the ease of use of the wearable device for external interaction. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the state perception method of this application. Figure 2 An interactive flowchart of the state-aware method provided in Embodiment 1 of this application; Figure 3 A schematic diagram of the external environment perception process provided in Embodiment 2 of this application; Figure 4 This is a schematic diagram of the module structure of the state sensing device according to an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the state-aware method in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] It should be noted that in the description of this application and the appended claims, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0024] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0025] The main solution of this application embodiment is: to determine whether the user's gaze point is within the content display range of the wearable device through a preset eye-tracking module; when the user's gaze point is within the content display range, to monitor the gaze time of the user's gaze point; to determine whether the user's attention is focused on the content display range based on whether the gaze time exceeds a preset time threshold; and to output external prompt information when the user's attention is focused on the content display range.

[0026] In recent years, with the advancement of hardware and software algorithms, wearable devices have become more lightweight and discreet in appearance, and the difference between them and ordinary glasses has been decreasing. This allows users to use wearable devices continuously without attracting attention, thus seamlessly integrating them into social, work, and life scenarios.

[0027] However, due to the discreet nature of wearable devices, it may be difficult for outsiders to judge the wearer's current state of attention. This can lead to misunderstandings or communication breakdowns in social situations, such as when pedestrians meet or when customers and sales associates interact.

[0028] To address the aforementioned issues, this application provides a state perception method for wearable devices. This method utilizes a pre-set eye-tracking module to determine whether the user's gaze is within the content display area of ​​the wearable device. By capturing the user's gaze focus in real time, it directly distinguishes whether the user is observing the external real environment or focusing on the virtual content provided by the device, providing a basis for subsequent attention judgment. Furthermore, when the user's gaze is within the content display area, the method monitors the user's gaze duration and determines whether the user's attention is concentrated within the content display area based on whether the gaze duration exceeds a preset time threshold. By quantifying the user's continuous gaze behavior, it provides an objective basis for attention judgment, avoiding misjudgments caused by brief glances or unintentional glances, thus improving the accuracy of user state perception. Finally, when the user's attention is concentrated within the content display area, it outputs external prompts, transforming the internal, invisible user attention state into a visible and perceptible signal to the external environment. This allows those around the user to clearly understand that the user is focused on virtual content and proactively take appropriate actions such as avoiding them, not disturbing them, or seeking non-contact methods to wake them up, thereby improving the ease of use of the wearable device for external interaction.

[0029] The wearable devices in this application embodiment may include, but are not limited to, wearable devices such as: Mixed Reality (MR) devices (e.g., MR glasses or MR helmets), Augmented Reality (AR) devices (e.g., AR glasses or AR helmets), Virtual Reality (VR) devices (e.g., VR glasses or VR helmets), Extended Reality (XR) devices, or some combination thereof. In this embodiment, for ease of description, the wearable device will be used as the execution subject in the following description.

[0030] Based on this, embodiments of this application provide a state-awareness method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the state perception method of this application.

[0031] In this embodiment, the state-aware method includes steps S10 to S40: Step S10: Determine whether the user's gaze point is within the content display range of the wearable device using a preset eye-tracking module; An eye-tracking module is a hardware component integrated into wearable devices (such as smart glasses). It typically includes an infrared light source and an infrared camera. It can illuminate the user's eyeball with an infrared light source and capture the reflection image of the eyeball surface with a camera. Based on the reflection image, it analyzes features such as pupil position and corneal reflection point to calculate the orientation and line of sight of the eyeball in three-dimensional space in real time.

[0032] The user gaze point refers to the target location where the user's eyes are focused, as determined by the eye-tracking module.

[0033] The content display area refers to the spatial range occupied by the display area of ​​a wearable device used to present information, which is usually represented by a virtual display interface.

[0034] In one feasible implementation, the eye-tracking module includes an infrared light source and a camera, and step S10 includes: Step S11: Infrared light is generated by using an infrared light source to illuminate the user's eyes, and an image of the eyes is captured by a camera; An infrared light source is a device that emits infrared light of a specific wavelength. The infrared light emitted is invisible light and is used to illuminate the user's eyes, and is reflected after contact with the eyes.

[0035] An eye image is a two-dimensional digital image captured by a camera that contains the structure of the eyeball illuminated by infrared light; the image contains at least two key features: a dark pupil area and a bright corneal reflector.

[0036] Step S12: Determine the gaze direction using a preset pupil-corneal reflex method and an eye image; The Pupil-Center Corneal Reflection (PCCR) method is an eye-tracking algorithm based on pupil and corneal reflection features. By analyzing the position and intensity changes of reflected infrared light on the pupil and cornea, it can calculate the direction of eye movement and the position of the fixation point.

[0037] The direction of gaze refers to the orientation of the optical axis of the eye in space. When the head remains still and the eyes are looking straight ahead, the optical axis of the eye usually points straight ahead. However, when the eyes move, the direction of the optical axis changes. For example, when the eyes turn to the left, the optical axis points to the left; when the eyes turn upward, the optical axis points upward.

[0038] For example, infrared light is emitted from an infrared light source to illuminate the user's eyes and generate a corneal reflection point; then, an eye image is captured in real time by a camera, and the captured eye image is preprocessed with noise reduction, grayscale conversion, etc., and the position coordinates of the pupil center and the corneal reflection point in the preprocessed eye image are determined by an image processing algorithm; then, based on the position coordinates of the pupil and the corneal reflection point, the angle between the pupil and the corneal reflection point is determined, and the user's gaze direction is determined based on this angle.

[0039] Step S13: Determine whether the user's gaze point is within the content display area based on the gaze direction and the content display range.

[0040] For example, the gaze direction can be converted into a two-dimensional coordinate point in the coordinate system corresponding to the current content display range according to the preset mapping function of the gaze direction and coordinate point in the wearable device, so as to obtain the gaze point coordinates; then, the gaze point coordinates are compared with the boundary conditions of the content display orientation, and a status flag is output according to the comparison result, indicating that the user's gaze point is within the content display range or the user's gaze point is not within the content display range.

[0041] In this embodiment, eye-tracking technology is used to capture the user's attention in real time, clearly distinguishing whether the user is immersed in virtual content or focused on the external environment, gaining a deeper understanding of the user's behavior and intentions, and adopting different external interaction strategies based on the user's attention, so as to improve the ease of use of wearable devices for external interaction.

[0042] Step S20: When the user's gaze point is within the content display area, monitor the user's gaze duration. Fixation time refers to the length of time a user's gaze remains within the content display area, as recorded by an internal timer on the device. It can be calculated based on the continuous cumulative value of the timer or the difference in timestamps.

[0043] Step S30: Determine whether the user's attention is focused on the content display area based on whether the gaze time exceeds a preset time threshold; The time threshold is a pre-set time value used to determine whether a user's attention is focused, and is used to distinguish between focused gaze and casual skipping.

[0044] For example, when the user's gaze is detected to be within the content display area, a high-precision timer is immediately started or resumed to record the gaze time until the user's gaze moves out of the display area. During the entire recording process, the current gaze time is read from the timer in real time or periodically and compared with a preset time threshold. If the gaze time is greater than or equal to the preset time threshold, it can be determined that the user's attention is focused on the content display area of ​​the wearable device, triggering an external prompt; otherwise, if the gaze time is less than the preset time threshold, it is determined that the user's attention is not focused.

[0045] Understandably, by monitoring the user's gaze duration and comparing it with a preset time threshold, it is possible to effectively distinguish between conscious viewing and unconscious, reflexive glances, reduce misjudgments of the user's attention state, provide more accurate external interaction prompts, and improve the ease of use of external interactions.

[0046] Step S40: When the user's attention is focused on the content display area, output external prompt information.

[0047] External notification information refers to any form of signal generated and emitted by a wearable device that is intended to be perceived by the external environment (including other users or other devices), including but not limited to: light signals, sound, text display, etc.

[0048] For example, when it is detected that the user's attention is focused on the content display area, a specific physical signal (external prompt information) is generated according to the preset prompt rules, and the generated physical signal is continuously emitted through an actuator (such as a light-emitting diode, speaker, Bluetooth module, etc.); at the same time, the user's attention concentration state is monitored, and the output of external prompt information ends when the user's attention is no longer focused on the content display area.

[0049] In one feasible implementation, the external prompting information includes a first invisible light signal and a visible light signal, and the step of outputting the external prompting information in step S40 includes: Step S41: Transmit a first invisible light signal to other wearable devices via a preset invisible light emitter, wherein the first invisible light signal includes an infrared light signal; and / or, An invisible light emitter is a component pre-integrated into a wearable device for emitting electromagnetic waves in the invisible spectrum; it communicates with other wearable devices by emitting invisible light.

[0050] Invisible light signals refer to electromagnetic wave signals with specific encoding and modulation, whose wavelengths are within the invisible light range, such as infrared light with wavelengths from 780 nanometers to 1 millimeter. In order to distinguish between the invisible light signals emitted and received by this wearable device, the invisible light signal emitted by the invisible light transmitter is called the first invisible light signal, and the invisible light signal received by the invisible light sensor is called the second invisible light signal.

[0051] For example, when it is detected that the user's attention is focused on the content display area of ​​the wearable device, that is, the user is not paying attention to the external environment, an infrared light signal (first invisible light signal) is sent to the outside at a low frequency through a preset invisible light emitter; then, when the invisible light sensor on another user's wearable device receives the infrared light signal from that user's wearable device, it can be determined that the user's attention is not in the external environment.

[0052] Step S42: Generate a light command and control the indicator light of the wearable device to emit a visible light signal according to the light command.

[0053] Lighting commands are control signals used to specify the visible light signals emitted by the indicator lights of wearable devices. They typically include parameters such as light color, flashing mode (e.g., constant light, breathing, fast flashing), and duration.

[0054] Indicator lights are components on wearable devices that emit visible light signals. They are typically made of LEDs (Light Emitting Diodes) or other light-emitting elements; they can emit different colors or flashing patterns of light according to light commands.

[0055] Visible light signals are light signals that can be directly perceived by the human eye and are used to provide intuitive prompts or warnings to other users or other wearable devices.

[0056] For example, when it is detected that the user's attention is focused on the content display area of ​​the wearable device, that is, the user is not paying attention to the external environment, a light instruction containing light control parameters is generated; then, the light instruction is synchronously transmitted to the indicator light through the communication line or wireless communication module (such as Bluetooth) inside the wearable device, and the current, voltage and other parameters of the light-emitting element (such as LED) of the indicator light are adjusted according to the light control parameters contained in the light instruction, so that it emits the corresponding visible light signal.

[0057] In this embodiment, by outputting external prompts containing both visible and invisible light signals, other users, whether wearing the corresponding wearable device or not, can perceive the current user's attention state and take appropriate communication methods based on the user's attention state, which helps to improve the ease of use of wearable devices for external interaction.

[0058] In one feasible implementation, the state-aware method further includes: Step S43: When the user's attention is focused on the content display area, determine the user's real-time location according to the preset positioning system; A positioning system refers to a system used in wearable devices to determine the user's location information, including but not limited to GPS (Global Positioning System) and BeiDou system.

[0059] Real-time location refers to the specific geographical location of a user at a certain moment, as determined by a positioning system, and can be represented by latitude and longitude coordinates obtained by the positioning system.

[0060] Step S44: If the real-time location is in a preset dangerous location, turn off the display content of the wearable device, and / or upload the real-time location to a preset cloud network.

[0061] Dangerous locations refer to pre-defined geographical locations that may pose safety risks, such as roadsides, motor vehicle lanes, and construction areas.

[0062] Cloud networks refer to remote server networks used for storing and sharing data, such as connected vehicles, real-time maps, or other Internet of Things platforms.

[0063] For example, when it is detected that the user's attention is focused on the content display area of ​​the wearable device, that is, the user is not paying attention to the external environment, the GPS system is used to obtain the coordinate information of the user's real-time location and compare it with the preset danger location to determine whether the user is currently close to or in danger; then, if the user's real-time location is in the preset danger location, corresponding rectification measures are implemented. For example, when the user is crossing the road or walking in the motor vehicle lane, the virtual display content in the user's wearable device is turned off directly, and the user's real-time location is uploaded to the real-time map in the cloud. A "Caution pedestrian" warning icon is generated based on the user's location to remind vehicles entering the road section to slow down in advance. Then, after the user moves away from the danger location, the corresponding virtual content can be re-displayed.

[0064] Understandably, by using a positioning system to determine in real time whether a user is in a dangerous location, potential dangers can be detected in a timely manner, enabling early warnings. Furthermore, upon detecting danger, the virtual display content can be forcibly shut down, allowing the user's attention to quickly return to the real environment. At the same time, the user's real-time location can be uploaded to the cloud network to provide real-time warnings to other users. Together, these measures reduce the occurrence of accidents caused by users not paying attention to their external environment while wearing wearable devices, thereby improving the safety of wearing device use.

[0065] For example, please refer to Figure 2 , Figure 2 An interactive flowchart of a state-awareness method is provided. It can determine the user's gaze direction (S101) using a preset eye-tracking module, and determine whether the user's gaze point is within the content display range based on the gaze direction and the content display range of the wearable device (S102). If the user's gaze point is within the content display range, the user's gaze duration is further monitored (S103), and it is determined whether the user's gaze duration exceeds a time threshold (S104). If the gaze exceeds the time threshold, it indicates that the user is currently focused on virtual content, and an external prompt message can be output (S105) so that others around can perceive that the user is currently immersed in the virtual display content of the wearable device. The method for outputting the external prompt message can include emitting invisible light signals (S106) or controlling the indicator light on the wearable device to emit visible light signals (S107).

[0066] This embodiment provides a state perception method that quantifies attention state through eye tracking, gaze duration monitoring, and time threshold comparison. This allows for accurate determination of whether the user's attention is focused on the content display area of ​​the wearable device, improving the accuracy and reliability of the wearable device's perception of the user's state. Furthermore, when the user's attention is detected to be focused on virtual content, external prompts such as visible and invisible light are output. This externalizes the user's internal and unknowable cognitive state into intuitive social language, enabling those around the user to clearly understand that the user is in an immersive state and thus proactively adjust their behavior (such as choosing not to disturb or actively avoiding the user). This reduces behavioral misunderstandings or communication breakdowns and improves the ease of use of the wearable device for external interaction.

[0067] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. In addition, the state perception method further includes: Step A10: When the user's attention is focused on the content display area, the second invisible light signal is received in real time through a preset invisible light sensor. An invisible light sensor is a photoelectric detection element that is sensitive to specific invisible light bands (such as infrared light), typically a photodiode or phototransistor.

[0068] The second invisible light signal refers to an invisible light carrier emitted by an external source (such as other wearable devices) and received by an invisible light sensor.

[0069] Step A20: Determine whether the second invisible light signal is an emergency contact signal based on the frequency band of the second invisible light signal; A frequency band refers to the center wavelength of an invisible light signal carrier or the spectral range it occupies; for example, a signal with a wavelength of 940nm can be preset as a normal communication frequency band, and a signal with a wavelength of 850nm can be preset as an emergency contact frequency band.

[0070] For example, when it is detected that the user's attention is focused on the content display area of ​​the wearable device, that is, the user is focused on the virtual content, the invisible light sensor is activated to perceive the invisible light signals in the surrounding environment in real time; when a second invisible light signal emitted by an external source is received, the received light signal is converted into an analog electrical signal, and then amplified and filtered to remove environmental noise; then, the analog electrical signal is converted into a digital signal by an analog-to-digital converter, the digital signal is subjected to a fast Fourier transform to determine its dominant frequency band, and the dominant frequency band is compared with a preset emergency contact frequency band. If the dominant frequency band is within the preset emergency contact frequency band, the second invisible light signal can be determined to be an emergency contact signal.

[0071] Step A30: If the second invisible light signal is an emergency contact signal, output internal alarm information.

[0072] Internal alarm information refers to the reminder information generated when an emergency contact signal is detected. It can remind users through pop-up displays, sounds, vibrations, etc. on wearable devices.

[0073] Understandably, by using a second invisible light sensor, the device can proactively detect communication signals from other wearable devices in the external environment while the user is focused on virtual content, and output alarm information to remind the user. This helps to draw the user's attention away from the virtual content and towards the external environment, thereby enabling proactive perception of the external environment and improving the ease of use of wearable devices for external interaction.

[0074] In one feasible implementation, the state-aware method further includes: Step B10: Capture the video stream in real time while the user's attention is focused on the content display area; A video stream is a continuous sequence of images of the scene in front of a wearable device, captured by a camera module. It is usually presented in the form of frames.

[0075] Step B20: Detect whether a hand image appears in the video stream using a pre-trained machine learning model; Pre-trained machine learning models refer to algorithmic models that have been trained and optimized with a large amount of data. They are capable of analyzing and recognizing input data (such as images or videos) and detecting hand images in video streams.

[0076] A hand image refers to a pixel region in a single or multiple frames of a video stream that is identified by a machine learning model as containing a hand structure.

[0077] For example, when it is detected that the user's attention is focused on the content display area of ​​the wearable device, that is, the user is focused on the virtual content, the video stream is captured in real time through the camera and input into the pre-trained machine learning model. This model has been trained on a large number of hand images and can accurately identify hand images in the video stream. Then, the machine learning model analyzes the input video stream data and outputs detection results, such as whether a hand image is detected and the position of the hand in the video frame.

[0078] Step B30: When a hand image appears in the video stream, the hand action is obtained based on the motion trajectory of the hand key points in the hand image in consecutive frames of the video stream. Key hand points refer to the set of two-dimensional or three-dimensional coordinate points representing the positions of various joints in the hand, extracted from hand images through machine learning models. These include feature points such as the wrist, the base of each finger, the middle of the finger, and the fingertip.

[0079] Motion trajectory refers to the path formed by the positional changes of key points of the hand in consecutive frames. It can be represented by a parametric curve to represent the dynamic characteristics of the hand.

[0080] Hand gestures refer to specific hand gestures or actions identified by analyzing the movement trajectories corresponding to key points on the hand, such as waving, clicking, and swiping.

[0081] For example, in consecutive frames of a video stream, the same set of hand key points in the detected hand images are tracked to ensure that the same physical key points are correctly associated in each frame; then, for each hand key point, its coordinates in consecutive frames are connected to obtain the motion trajectory corresponding to each hand key point; then, each motion trajectory is analyzed, for example, to determine whether the motion trajectories of all key points remain synchronized, or whether only the motion trajectory corresponding to the index finger changes, and the hand action is determined based on the analysis results.

[0082] Understandably, by capturing video streams in real time and actively observing interactive elements (such as hand movements) in the external environment, it is possible to identify interactive attempts by other users in the external environment who are not wearing the wearable device, thereby achieving proactive perception of the external environment and improving the ease of use of wearable devices for external interaction.

[0083] Step B40: If the hand gesture is a preset waving gesture, output an internal alarm message.

[0084] For example, if a waving motion is detected, it means that there is another user in front of the user trying to contact them. The user can be alerted by a pop-up window or sound that "someone is trying to communicate with you", so as to wake the user from the state of being immersed in virtual content in a timely manner.

[0085] For example, please refer to Figure 3 , Figure 3A flowchart illustrating an external environment perception process is provided. When a user is detected focusing on virtual content, the external environment can be perceived by receiving invisible light signals (S108) or capturing video streams (S112). After receiving an invisible light signal, its frequency band can be analyzed to determine its signal frequency band (S109). Based on whether the signal frequency band matches a preset emergency signal frequency band, it can be determined whether the invisible light signal is an emergency contact signal (S110). If the invisible light signal is an emergency contact signal, an internal alarm message is output (S111) to interrupt the user's immersion and remind the user that there is a communication signal in the external environment. After capturing a video stream, a pre-trained machine learning model can be used to extract hand images from the video stream and analyze the changes in key hand points in the hand images across consecutive video frames to determine the hand movements of other people in the current field of vision (S113). It can then be determined whether the hand movement is a waving gesture (S114). If so, an internal alarm message is output (S111) to interrupt the user's immersion and remind the user that other people in the external environment are attempting to communicate.

[0086] In this embodiment, by receiving invisible light signals or recognizing hand movements in front of the device when the user is focused on virtual content, it can determine whether there are other users trying to communicate in the external environment, thus achieving proactive perception of the external environment; at the same time, by outputting internal alarm information, it can remind the user to pay attention to the external environment, thereby achieving focused state wake-up without touching the user's body, improving the ease of use of wearable devices for external interaction.

[0087] In one feasible implementation, the state-aware method further includes: Step E10: In response to the triggered sharing instruction, determine the content to be shared, and generate a sharing signal based on the content to be shared; A sharing command is a command triggered by a user in a certain way (such as gestures, voice, button clicks, etc.) to instruct the device to perform a content sharing operation.

[0088] A shared signal is a data transmission signal that follows a communication protocol (such as Bluetooth, Wi-Fi, etc.) and is used to transmit content to be shared.

[0089] For example, a user can trigger a sharing operation by using the voice command "share this document"; then, in response to the received voice command, the wearable device determines the document displayed within the content display range as the content to be shared; then, it retrieves the document from the device storage and encodes it into a format suitable for transmission, generating a sharing signal, or sets the document (the content to be shared) to a sharing mode, generates an access link, encodes the access link into a format suitable for transmission, and generates a sharing signal.

[0090] Step E20: Send a sharing signal to the preset sharing user, wherein the sharing user is determined based on the sharing instruction.

[0091] A shared user refers to a target object that has an identifier that can be uniquely identified in a communication protocol.

[0092] For example, after receiving a sharing instruction, the wearable device extracts user information (such as nickname, name, etc.), determines the sharing user and its corresponding unique identifier based on the user information, encodes the unique identifier and the corresponding content to be shared into a sharing signal, and then sends the sharing signal to the corresponding sharing user. If the sharing instruction does not specify the sharing user, the sharing signal containing the content to be shared can be broadcast to all nearby users wearing the wearable device.

[0093] In this embodiment, the sharing operation allows users to easily share what they see on the wearable device with other users in real time without having to speak, thus improving the ease of use of the wearable device.

[0094] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the state perception method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0095] This application also provides a state sensing device, please refer to... Figure 4 The state sensing device includes: The eye-tracking module 10 is used to determine whether the user's gaze point is within the content display range of the wearable device through a preset eye-tracking module; The gaze duration monitoring module 20 is used to monitor the gaze duration of the user's gaze point when the user's gaze point is within the content display range; The attention judgment module 30 is used to determine whether the user's attention is focused on the content display area based on whether the gaze time exceeds a preset time threshold. The information output module 40 is used to output external prompts when the user's attention is focused on the content display area.

[0096] Optionally, the eye-tracking module 10 is also used for: Infrared light is generated by an infrared light source to illuminate the user's eyes, and an image of the eyes is captured by a camera. The gaze direction is determined using a preset pupil-corneal reflex method and an eye image; Based on the direction of gaze and the content display area, determine whether the user's gaze point is within the content display area.

[0097] Optionally, the information output module 40 is also used for: A first invisible light signal is emitted to other wearable devices via a pre-set invisible light emitter, wherein the first invisible light signal includes an infrared light signal; and / or, Generate light commands and, based on these commands, control the indicator lights on the wearable device to emit visible light signals.

[0098] Optionally, the information output module 40 is also used for: When the user's attention is focused on the content display area, the user's real-time location is determined according to the preset positioning system; If the real-time location is in a preset dangerous location, turn off the display content of the wearable device, and / or upload the real-time location to a preset cloud network.

[0099] Optionally, the information output module 40 is also used for: When the user's attention is focused on the content display area, a second invisible light signal is received in real time through a preset invisible light sensor; Based on the frequency band of the second invisible light signal, determine whether the second invisible light signal is an emergency contact signal; If the second invisible light signal is an emergency contact signal, output internal alarm information.

[0100] Optionally, the information output module 40 is used for: Real-time video streaming is captured when the user's attention is focused on the content display area; The pre-trained machine learning model is used to detect whether hand images appear in the video stream. When a hand image appears in a video stream, the hand motion is obtained based on the motion trajectory of the hand key points in the hand image in consecutive frames of the video stream. If the hand gesture is a preset waving gesture, output an internal alarm message.

[0101] Optionally, the state-aware device further includes: a sharing module, used for: In response to a triggered sharing command, determine the content to be shared and generate a sharing signal based on the content; Send a sharing signal to a preset sharing user, where the sharing user is determined based on the sharing instruction.

[0102] The state sensing device provided in this application, employing the state sensing method described in the above embodiments, can solve the technical problem of how to improve the ease of use of wearable devices for external interaction. Compared with the prior art, the beneficial effects of the state sensing device provided in this application are the same as those of the state sensing method described in the above embodiments, and other technical features in the state sensing device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0103] This application provides a wearable device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the state awareness method in the first embodiment described above.

[0104] The following is for reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing wearable devices according to embodiments of this application. Wearable devices in embodiments of this application may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The wearable device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0105] like Figure 5 As shown, the wearable device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the wearable device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the wearable device to communicate wirelessly or wiredly with other devices to exchange data. While wearable devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0106] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0107] The wearable device provided in this application, employing the state perception method described in the above embodiments, can solve the technical problem of how to improve the ease of use of the wearable device for external interaction. Compared with the prior art, the beneficial effects of the wearable device provided in this application are the same as those of the state perception method provided in the above embodiments, and other technical features of the wearable device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0108] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0109] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0110] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the state-aware method described above.

[0111] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0112] The aforementioned computer-readable storage medium may be included in the wearable device; or it may exist independently and not assembled into the wearable device.

[0113] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a wearable device, cause the wearable device to: determine whether the user's gaze point is within the content display area of ​​the wearable device via a preset eye-tracking module; monitor the user's gaze duration when the user's gaze point is within the content display area; determine whether the user's attention is focused on the content display area based on whether the gaze duration exceeds a preset time threshold; and output external prompt information when the user's attention is focused on the content display area.

[0114] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0116] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0117] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described state-awareness method, thereby solving the technical problem of how to improve the ease of use of wearable devices for external interaction. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the state-awareness method provided in the above embodiments, and will not be repeated here.

[0118] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the state-aware method described above.

[0119] The computer program product provided in this application can solve the technical problem of how to improve the ease of use of wearable devices for external interaction. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the state awareness method provided in the above embodiments, and will not be repeated here.

[0120] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A state-aware method, characterized in that, The state-awareness method is applied to wearable devices, and the method includes: The system uses a preset eye-tracking module to determine whether the user's gaze point is within the content display range of the wearable device. When the user's gaze point is within the content display area, monitor the gaze duration of the user's gaze point; Based on whether the gaze duration exceeds a preset time threshold, it is determined whether the user's attention is focused on the content display area; When the user's attention is focused on the content display area, an external prompt message is output.

2. The state-aware method as described in claim 1, characterized in that, The eye-tracking module includes an infrared light source and a camera. The step of determining whether the user's gaze point is within the content display range of the wearable device using the preset eye-tracking module includes: The infrared light source generates infrared light that illuminates the user's eyes, and the camera captures an image of the eyes. The gaze direction is determined using a preset pupil-corneal reflex method and the eye image. Based on the gaze direction and the content display range, determine whether the user's gaze point is within the content display range.

3. The state perception method as described in claim 1, characterized in that, The external notification information includes a first invisible light signal and a visible light signal, and the step of outputting the external notification information includes: A first invisible light signal is emitted to other wearable devices via a pre-set invisible light emitter, wherein the first invisible light signal includes an infrared light signal; and / or, Generate a light command, and control the indicator light of the wearable device to emit a visible light signal according to the light command.

4. The state-aware method as described in claim 1, characterized in that, The method further includes: When the user's attention is focused on the content display area, the user's real-time location is determined according to a preset positioning system; If the real-time location is in a preset dangerous location, the display content of the wearable device is turned off, and / or the real-time location is uploaded to a preset cloud network.

5. The state perception method as described in claim 1, characterized in that, The method further includes: When the user's attention is focused on the content display area, a second invisible light signal is received in real time through a preset invisible light sensor; Based on the frequency band of the second invisible light signal, determine whether the second invisible light signal is an emergency contact signal; If the second invisible light signal is an emergency contact signal, an internal alarm message is output.

6. The state-aware method as described in claim 1, characterized in that, The method further includes: While the user's attention is focused on the content display area, a video stream is captured in real time; The video stream is used to detect whether a hand image appears using a pre-trained machine learning model. When a hand image appears in the video stream, the hand action is obtained based on the motion trajectory of the hand key points in the hand image in consecutive frames of the video stream. If the hand gesture is a preset waving gesture, an internal alarm message is output.

7. The state-aware method as described in claim 1, characterized in that, The method further includes: In response to a triggered sharing command, determine the content to be shared and generate a sharing signal based on the content to be shared; The sharing signal is sent to a preset sharing user, wherein the sharing user is determined based on the sharing instruction.

8. A wearable device, characterized in that, The wearable device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the state-aware method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the state-aware method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the state-aware method as described in any one of claims 1 to 7.

Images