Unlocking method, device, equipment, medium and product of head-mounted display device

Abstract

The application discloses a kind of unlocking method, device, equipment, medium and product of head-mounted display device, it is related to information security technical field, comprising: when detecting that user exists unlocking intention, at least two target objects that user wears head-mounted display device successively gazes at are determined, target object is the object in the real physical environment;Determine the object semantic feature of each target object and the relative spatial relationship feature between objects;Determine the eye movement dynamics feature when user gazes at each target object;According to the object semantic feature, the relative spatial relationship feature and the eye movement dynamics feature generate real-time feature set and match with pre-stored template feature set;Matching is successful then unlocking.The application is unlocked by object in real physical environment in combination with the eye movement behavior characteristics unique to user, without user actively performing any operation, improves the overall security and use convenience of head-mounted display device unlocking.

Description

Technical Field

[0001] This invention relates to the field of information security technology, and in particular to a method, apparatus, device, medium, and product for unlocking a head-mounted display device. Background Technology

[0002] As a new generation of human-computer interaction terminals, the unlocking method of head-mounted display devices directly affects the user experience and device security. Currently, the mainstream unlocking methods for head-mounted display devices mainly include the following categories: 1. Traditional Password / Pattern Unlock. Users unlock their devices by entering a password or drawing a pattern on a virtual keyboard using a controller or gestures. However, this method is inefficient, the input process is easily spied on, and its security is insufficient.

[0003] II. Unlocking based on gaze point information. This type of solution collects the user's gaze point information and compares it with a pre-stored unlock code to determine whether to unlock. However, unlock codes are usually virtual patterns or number sequences with limited combinations, and the gaze trajectory is easily observed and copied. Furthermore, the user needs to actively gaze at specific virtual elements, which disrupts the sense of immersion.

[0004] III. Static Biometric Recognition. This type of solution authenticates a user's identity by collecting static biometric features such as iris, fingerprint, or face. However, static biometric features are easily copied and forged, and the recognition rate is unstable in scenarios such as when the user is wearing a mask, glasses, or in low light conditions.

[0005] IV. Eye-tracking based pattern password verification. This type of solution displays a pattern unlocking design on a head-mounted display, and the user completes the password input by tracking their eyes. However, this type of solution still relies on virtual elements displayed on the screen, has limited password combination space, the trajectory is easily observed and copied, and requires the user to actively perform the unlocking operation. Summary of the Invention

[0006] This invention provides a method, apparatus, device, medium, and product for unlocking a head-mounted display device, in order to solve the technical problem that the unlocking security of head-mounted display devices is insufficient and requires users to actively perform the unlocking operation in the prior art.

[0007] According to one aspect of the present invention, a method for unlocking a head-mounted display device is provided, the method comprising: If the system detects that the user intends to unlock the head-mounted display device, it determines at least two target objects that the user is sequentially looking at while wearing the head-mounted display device; wherein, the target objects are objects in the real physical environment in which the head-mounted display device is located. Determine the semantic features of each target object, and determine the relative spatial relationship features between each target object; Determine the eye movement dynamics characteristics corresponding to the user's sequential gaze at each of the target objects; wherein, the eye movement dynamics characteristics include at least one of gaze duration characteristics, saccade speed characteristics, microsaccade characteristics, and pupillary change characteristics; A real-time feature set is generated based on the object semantic features, the relative spatial relationship features, and the eye-tracking dynamics features, and the real-time feature set is matched with a pre-stored template feature set; If the pairing is successful, the head-mounted display device is unlocked.

[0008] According to another aspect of the present invention, an unlocking device for a head-mounted display device is provided, the device comprising: The target object determination module is used to determine at least two target objects that the user is sequentially looking at while wearing the head-mounted display device when the system detects that the user intends to unlock the head-mounted display device; wherein the target objects are objects in the real physical environment in which the head-mounted display device is located. The first feature determination module is used to determine the object semantic features of each of the target objects, and to determine the relative spatial relationship features between the target objects. The second feature determination module determines the eye movement dynamics features corresponding to the user's sequential gaze at each of the target objects; wherein, the eye movement dynamics features include at least one of gaze duration features, saccade speed features, microsaccade features, and pupillary change features; The matching module is used to generate a real-time feature set based on the semantic features of the object, the relative spatial relationship features, and the eye-tracking dynamics features, and to match the real-time feature set with a pre-stored template feature set; An unlocking module is used to unlock the head-mounted display device if a match is successful.

[0009] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method described in any one of the present invention.

[0010] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the method described in any one of the present invention.

[0011] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method of any one of the present invention.

[0012] The beneficial effects of this invention are as follows: First, this invention extends the cryptographic carrier from traditional virtual numbers or patterns to objects in a real physical environment, requiring the user to gaze at at least two target objects. Simply knowing the object types is insufficient for verification—the system simultaneously requires a joint match of object semantic features, relative spatial relationship features between objects, and user eye-tracking dynamics features. Eye-tracking dynamics features are inherent behavioral biometrics of the user, possessing individual specificity and being difficult to imitate. Even after knowing the object types and spatial relationships, attackers cannot simulate the target user's eye-tracking characteristics, thus failing verification. Compared to existing schemes that rely solely on virtual passwords or static biometrics, security is significantly improved.

[0013] Secondly, this invention automatically triggers the verification process by detecting whether the user intends to unlock, without requiring any active user action. During the verification process, the user only needs to look at objects in their real-world environment naturally, and the system completes implicit authentication. Compared to existing technologies that require the user to actively perform an unlocking operation, interrupting the usage flow, this invention integrates the unlocking process into the user's natural behavior, maintaining the continuity of use of the head-mounted display device.

[0014] Third, this invention does not rely on fixed patterns or a limited character set on a virtual interface. Instead, it uses a sequence of objects in a real physical environment, the three-dimensional spatial relationships between objects, and the user's own eye-tracking dynamics to construct the password. Since the types of objects and their spatial relationships in the real environment are almost infinite, and different users have individual differences in their eye-tracking characteristics, the password combination space theoretically approaches infinity, effectively overcoming the limitation of finite password combinations in existing technologies.

[0015] Fourth, this invention uses dynamic eye-tracking behavior features for identity authentication, without relying on the complete acquisition of static biometric features such as iris texture, facial structure, or fingerprint patterns. Therefore, eye tracking can still function normally in scenarios where existing biometric solutions are prone to failure, such as when users are wearing masks, glasses, or in low-light conditions, thus ensuring the environmental adaptability of the unlocking process.

[0016] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of an unlocking method for a head-mounted display device provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of a gaze target determination method provided in Embodiment 2 of the present invention; Figure 3 This is a flowchart of a method for determining the semantic features of an object provided in Embodiment 3 of the present invention; Figure 4 This is a flowchart of a method for determining relative spatial relationship features provided in Embodiment 4 of the present invention; Figure 5 A flowchart of a matching verification method provided in Embodiment 5 of the present invention; Figure 6 This is a flowchart of a template feature set update method provided in Embodiment Six of the present invention; Figure 7 A flowchart of a stress detection method provided in Embodiment 7 of the present invention; Figure 8 This is a schematic diagram of the structure of an unlocking device for a head-mounted display device provided in Embodiment 8 of the present invention; Figure 9 This is a schematic diagram of the structure of an electronic device that implements the unlocking method of the head-mounted display device according to an embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0020] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] Example 1 Figure 1 This is a flowchart illustrating a method for unlocking a head-mounted display device according to Embodiment 1 of the present invention. This embodiment is applicable to scenarios where a user needs to unlock a head-mounted display device for normal use. The method can be executed by an unlocking device for the head-mounted display device, which can be implemented in hardware and / or software, such as by the processor of the head-mounted display device. Figure 1 As shown, the method includes: S101. If it is detected that the user intends to unlock the head-mounted display device, determine at least two target objects that the user is looking at sequentially while wearing the head-mounted display device.

[0022] The intent to unlock the head-mounted display device refers to the system's determination of the user's need to use the device and unlock it after the user puts the device on, by detecting the device's wearing status and the user's eye condition. This intent is a prerequisite for triggering the subsequent unlocking process, unlike accidental triggering when the user has no need to use the device.

[0023] The target object is an object in the real physical environment in which the head-mounted display device is located. The real physical environment refers to the three-dimensional physical space in which the head-mounted display device is currently located, which is different from the virtual interface or virtual elements displayed on the device screen. Objects in the real physical environment have objectively existing three-dimensional spatial positions and geometric properties.

[0024] In one implementation, the processor of the head-mounted display device (hereinafter referred to as the processor) detects the wearing status of the head-mounted display device through an inertial measurement unit. When the processor detects that the device is being worn by a user, it activates an eye-tracking module to detect whether the user's eyes are open and looking forward. When the processor detects that the device is being worn and the user's eyes are open and looking forward, it confirms that the user intends to unlock the device and automatically initiates the unlocking process. The user naturally gazes at at least two target objects in their surrounding real physical environment, such as a computer, keyboard, and water glass on a desk. The processor collects the user's gaze point data in real time through the eye-tracking module and simultaneously acquires an image of the environment in front of them through an image acquisition module, combining this with depth information to determine the spatial position of each object being gazed at. Based on the collected gaze point data, environmental image, and depth information, the processor determines the target objects that the user is gazing at in sequence.

[0025] S102. Determine the semantic features of each target object and the relative spatial relationship features between each target object.

[0026] Among them, object semantic features refer to the category attributes and content descriptions of target objects at the visual level, including the object's category label and the semantic feature vector used to characterize the object's visual appearance. The category label describes "what the object is," such as "computer," "keyboard," or "water cup"; the semantic feature vector quantifies the object's visual appearance, such as computable attributes like shape, texture, and color. Object semantic features are used during the unlocking verification process to identify and confirm whether the object the user is looking at corresponds to the object in the template feature set.

[0027] Relative spatial relationship features refer to the positional constraints between target objects in three-dimensional space, specifically including the relative distance and relative angle between any two target objects. These features are used during the unlocking verification process to confirm whether the spatial layout of the target objects matches the template feature set.

[0028] In one implementation, the processor acquires an environmental image in front of the head-mounted display device using a red-green-blue color space camera, identifies the target objects determined in S101 using a semantic segmentation model, obtains category labels for each target object, and extracts semantic feature vectors to characterize the visual appearance of the objects. The processor uses the category labels and semantic feature vectors as the semantic features of the objects.

[0029] The processor also acquires depth information of each target object using a time-of-flight depth camera, and combines this with pose information from the head-mounted display provided by the simultaneous localization and mapping (SOL) system to determine the spatial position of each target object in three-dimensional space. Based on the spatial position of each target object, the processor calculates the relative distance between any two target objects; simultaneously, it calculates the relative angle between any two target objects by combining this with the current line of sight from the head-mounted display. The processor uses the relative distance and relative angle as features of the relative spatial relationship.

[0030] S103. Determine the eye-tracking dynamics characteristics corresponding to the user's sequential gaze at each target object.

[0031] Eye-tracking dynamics (OMR) refers to the dynamic behavioral patterns of a user's eye movements in time and space while fixating on a target object. OMR reflects individual differences in a user's eye movement system, including neural control characteristics, muscle response characteristics, and visual attention strategies, and is individual-specific and difficult to imitate. OMR includes at least one of the following: fixation time characteristics, saccade speed characteristics, microsaccade characteristics, and pupillary change characteristics.

[0032] Fixation duration characteristics refer to the statistical distribution of the duration for which a user's gaze lingers on a single target object. The distribution patterns of fixation duration vary among different users and can be used to distinguish between them.

[0033] Salivation speed characteristics refer to the changes in eye movement speed when a user's gaze rapidly shifts from one target object to another. Parameters such as the peak speed and rate of change during saccades vary among different users.

[0034] Microsaccades are statistical characteristics of high-frequency, small-amplitude eye movements (microsaccades) that occur during fixation, including their frequency, average amplitude, and directional distribution. Microsaccades are involuntary eye movements, beyond the user's conscious control, and difficult to consciously imitate.

[0035] Pupil variability refers to the characteristics of pupil diameter changes over time when a user fixates on a target object. Changes in pupil diameter are related to the user's cognitive load, attentional state, and physiological responses; different users exhibit individual differences in pupil response patterns under the same visual stimulus.

[0036] In one implementation, the processor uses an eye-tracking module to collect eye movement data in real time as the user looks at each target object in sequence.

[0037] The processor obtains the fixation duration data when the user fixates on each target object from the eye-tracking data, analyzes the distribution characteristics of the fixation duration data, and uses the parameters describing the distribution characteristics as fixation duration features.

[0038] The processor obtains time-series data of the user's saccade speed as the gaze moves between various target objects from the eye-tracking data, analyzes the characteristics of the saccade speed change over time, and uses the parameters describing these characteristics as saccade speed features.

[0039] The processor identifies microsaccade events that occur during the user's gaze from eye-tracking data, and performs statistical analysis on the frequency, amplitude, and directional distribution of microsaccade events, using the statistical quantities obtained from the analysis as microsaccade features.

[0040] The processor obtains time-series data on the change of pupil diameter over time when the user gazes at various target objects from eye-tracking data, analyzes the change pattern of pupil diameter, and uses the parameters describing the change pattern as pupil change features.

[0041] The processor selects at least one feature from fixation time features, saccade speed features, microsaccade features, and pupil change features as the eye movement dynamics features corresponding to each target object.

[0042] S104. Generate a real-time feature set based on the semantic features of the object, the relative spatial relationship features, and the eye-tracking dynamics features, and match the real-time feature set with the pre-stored template feature set.

[0043] The real-time feature set refers to the data set extracted and generated in real time during the unlocking and verification process based on the target object currently being gazed at by the user, and used for comparison with the template feature set. The real-time feature set consists of three parts: semantic features of the object determined at the current moment, relative spatial relationship features, and eye-tracking dynamic features, reflecting the comprehensive information of the user's current gaze behavior in the three dimensions of semantics, space, and eye movement.

[0044] The template feature set refers to the data set pre-collected and stored during the registration phase, serving as the benchmark for unlock verification. The structure of the template feature set is the same as the real-time feature set, consisting of three parts: object semantic features, relative spatial relationship features, and eye-tracking dynamics features determined during registration. It reflects the comprehensive information of the user's gaze at each registered object in the semantic, spatial, and eye-tracking dimensions. After being generated during the registration phase, the template feature set is pre-stored in the head-mounted display device as a comparison benchmark for subsequent unlock verification.

[0045] In one implementation, during the registration phase, the processor guides the user to sequentially gaze at at least two registered objects in the registration environment. Following the same acquisition and extraction method as the real-time feature set, the processor determines the semantic features of each registered object, the relative spatial relationship features between the registered objects, and the eye-tracking dynamics features corresponding to the user's sequential gazing at each registered object. The processor generates a template feature set based on these three features and pre-stores the template feature set in the head-mounted display device.

[0046] During the unlocking verification phase, the processor combines the object semantic features, relative spatial relationship features, and eye-tracking dynamics features determined in steps S101 to S103 to generate a real-time feature set. The processor then matches this real-time feature set with a pre-stored template feature set: calculating the similarity between the real-time and template feature sets in the three dimensions of object semantic features, relative spatial relationship features, and eye-tracking dynamics features, and performing a comprehensive calculation of the similarity across these three dimensions to obtain the total similarity. If the total similarity reaches a preset threshold (e.g., 0.85), the match is considered successful, and the unlocking process begins; if the total similarity does not reach the preset threshold but is higher than the tolerance threshold (e.g., 0.60), the match is considered a failure, and the processor prompts the user to re-perform gaze verification; if the total similarity is lower than the tolerance threshold, the processor records the number of failures and performs a security lock when the number of failures reaches the upper limit.

[0047] S105. If pairing is successful, unlock the head-mounted display device.

[0048] A successful match means that the processor determines the total similarity between the real-time feature set and the template feature set reaches a preset threshold. This indicates that the current user's gaze behavior and the template feature set established during registration have a comprehensive similarity across semantic, spatial, and eye-tracking dimensions that meets the unlocking requirements. A successful match means that the current user's identity has been verified and they have the authority to unlock the device.

[0049] In one implementation, upon successful pairing, the processor generates an unlock command to release the head-mounted display device from its locked state, switching the device from a restricted operation mode to a normal usage mode. After unlocking, the processor displays a normal user interface on the head-mounted display device, allowing the user to access all device functions and applications normally. Optionally, the processor may also provide feedback to the user regarding successful unlocking through at least one of the following methods: display prompts, vibration feedback, or sound prompts. If pairing fails, the processor does not perform the unlocking operation, maintaining the device's locked state.

[0050] The beneficial effects of the embodiments of the present invention are as follows: First, this invention extends the password carrier from traditional virtual numbers or patterns to objects in a real physical environment, requiring the user to gaze at at least two target objects. Simply knowing the object types is insufficient for verification—the system simultaneously requires a joint match of object semantic features, relative spatial relationship features between objects, and user eye-tracking dynamics features. Eye-tracking dynamics features are inherent behavioral biometrics of the user, possessing individual specificity and being difficult to imitate. Even after knowing the object types and spatial relationships, attackers cannot simulate the target user's eye-tracking characteristics, thus failing verification. Compared to existing schemes that rely solely on virtual passwords or static biometrics, security is significantly improved.

[0051] Secondly, this embodiment of the invention automatically triggers the verification process by detecting whether the user intends to unlock, without requiring any active user action. During the verification process, the user only needs to look at objects in their real environment naturally to complete implicit authentication. Compared to existing technologies that require the user to actively perform an unlocking operation, interrupting the usage process, this embodiment of the invention integrates the unlocking process into the user's natural behavior, maintaining the continuity of use of the head-mounted display device.

[0052] Third, the embodiments of the present invention do not rely on fixed patterns or finite character sets on a virtual interface, but instead use sequences of objects in a real physical environment, three-dimensional spatial relationships between objects, and the user's own eye-tracking dynamics to construct the password. Since the types of objects and their spatial relationships in a real environment are almost infinitely varied, and different users have individual differences in their eye-tracking characteristics, the password combination space theoretically approaches infinity, effectively overcoming the limitation of finite password combinations in existing technologies.

[0053] Fourth, the embodiments of the present invention use dynamic eye-tracking behavior features for identity authentication, without relying on the complete acquisition of static biometric features such as iris texture, facial structure, or fingerprint patterns. Therefore, eye tracking can still work normally in scenarios where existing biometric identification schemes are prone to failure, such as when users are wearing masks, glasses, or in low-light conditions, thus ensuring the environmental adaptability of the unlocking process.

[0054] Optionally, fixation time characteristics are determined as follows: Acquire gaze duration data when a user gazes at a target object; fit the gaze duration data using a gamma distribution, and extract the shape and scale parameters of the gamma distribution as gaze duration features.

[0055] Fixation duration refers to the continuous length of time a user's gaze remains on a single target object. When a user fixates on an object, their gaze is not completely still, but rather maintains a fixation state within a certain range; the duration of this state is called fixation duration. Different users exhibit individual differences in the length and fluctuation pattern of their fixation duration when focusing on the same object.

[0056] The gamma distribution is a continuous probability distribution that is often used to describe data such as "waiting time" (e.g., the length of time required for an event to occur).

[0057] Shape and scale parameters are statistical quantities used to describe the shape and scale of a probability distribution. Shape parameters reflect the morphological characteristics of the data distribution (such as skewness), while scale parameters reflect the dispersion of the data distribution. By fitting a gamma distribution to gaze duration data, shape and scale parameters can be extracted to quantitatively describe the individual characteristics of a user's gaze behavior.

[0058] In one implementation, the processor acquires gaze duration data when the user gazes at various target objects. The processor fits the acquired gaze duration data using a gamma distribution to determine shape and scale parameters that best describe the distribution characteristics of the gaze duration data set.

[0059] Specifically, the probability density function of the gamma distribution is: ; in, Let the fixation duration be a random variable. For shape parameters, This is the scale parameter.

[0060] The processor uses the maximum likelihood estimation method to substitute the user's gaze duration data into the probability density function mentioned above, and fits the shape parameters that best describe the data set. and scale parameters The processor uses the fitted shape and scale parameters as gaze temporal features to represent the temporal distribution characteristics of the user's gaze behavior.

[0061] The above method fits the gaze duration data to a gamma distribution, transforming the raw time data into statistically significant shape and scale parameters, which can stably reflect individual differences in users' gaze behavior. Because the distribution characteristics of gaze duration are involuntary eye-movement behaviors, users find it difficult to consciously control and imitate, thus exhibiting good anti-spoofing capabilities and improving the reliability of identity verification.

[0062] Optionally, the saccade speed characteristic is determined in the following way: Obtain the time series of saccade speeds when a user gazes at a target object; extract statistical features from the saccade speed time series, and determine the saccade speed features based on the statistical features.

[0063] Salivation speed time series refers to the continuous data sequence formed by the change in eye movement speed over time as a user's gaze rapidly shifts from one fixation point to another. Salivation is the rapid eye movement between fixation points; the saccade speed rises rapidly from zero to a peak and then falls back to zero during saccades, forming a characteristic speed curve. The speed change pattern during saccades varies among different users.

[0064] Statistical characteristics include at least one of the following: average velocity, velocity standard deviation, peak velocity, average acceleration, and dominant frequency component.

[0065] In one implementation, the processor acquires continuous saccade speed data as the user's gaze shifts between various target objects via an eye-tracking module, forming a saccade speed time series, denoted as: ; in, For the time series of scanning speed, to This represents the scanning speed value at each sampling time.

[0066] Processor for saccadic speed time series Statistical analysis was performed to extract at least one of the following statistical characteristics as saccadic speed features: average speed, speed standard deviation, peak speed, average acceleration, and dominant frequency component. Specifically, average speed reflects the overall level of the user's saccadic speed; speed standard deviation reflects the degree of fluctuation in saccadic speed; peak speed reflects the user's eye muscle response ability; average acceleration reflects the dynamic characteristics of the saccadic initiation and braking phases; and the dominant frequency component reflects the periodic rhythmic characteristics of saccadic speed changes.

[0067] The processor uses the extracted statistical features as saccade speed features to represent the speed change characteristics of the user's gaze when it shifts rapidly.

[0068] The above method describes the user's saccadic behavior patterns by extracting various statistical features from the time series of saccadic speed, reflecting individual differences in dynamic gaze shifts from multiple quantitative dimensions. Since the time and frequency domain characteristics of saccadic speed reflect the inherent characteristics of the user's eye-tracking control system, they are difficult for others to observe and imitate, thus effectively distinguishing different users and enhancing the security of the unlocking process.

[0069] Optionally, the microsaccade features include at least one of microsaccade frequency, average microsaccade amplitude, and microsaccade directional distribution entropy.

[0070] Microsaccade frequency refers to the number of microsaccades that occur per unit of time while a user is fixating on a target object. Microsaccades are high-frequency, small-amplitude, involuntary eye movements that occur during fixation; they are non-involuntary eye movements and are not directly controlled by the user's conscious awareness. The microsaccade frequency varies in stability among different individuals under the same visual task.

[0071] The average amplitude of microsaccades refers to the average amplitude of each microsaccade during a user's fixation on a target object. Microsaccade amplitude reflects the amplitude characteristics of an individual's eye-tracking system during minute adjustments, and there are individual differences in the average amplitude of microsaccades between different individuals.

[0072] Microsaccade direction distribution entropy is a measure of the randomness or uncertainty of the distribution of microsaccade directions. The processor calculates the entropy value of the direction distribution by statistically analyzing the directions of each microsaccade. A higher entropy value indicates a more dispersed and random distribution of microsaccade directions; a lower entropy value indicates a more concentrated and regular distribution. The microsaccade direction distribution entropy varies among individuals when focusing on the same object.

[0073] In one implementation, the processor identifies micro-saccade events occurring when the user gazes at various target objects from eye-tracking data collected by the eye-tracking module, and calculates the frequency of micro-saccade occurrence as the micro-saccade frequency; it calculates the average amplitude of each micro-saccade as the average micro-saccade amplitude; and it performs statistical analysis on the direction of each micro-saccade, calculating the entropy value of the direction distribution as the micro-saccade direction distribution entropy. The processor selects at least one of the micro-saccade frequency, average micro-saccade amplitude, and micro-saccade direction distribution entropy as a micro-saccade feature.

[0074] The above method describes the user's microsaccade behavior pattern by extracting the frequency, amplitude, and directional distribution entropy of microsaccades. Since microsaccades are involuntary eye movements and not directly controlled by the user's conscious will, users find it difficult to consciously adjust or imitate their own microsaccade patterns, thus exhibiting strong anti-spoofing capabilities. Compared to parameters such as gaze point position or gaze trajectory, which can be intentionally controlled, microsaccade features can more realistically reflect the individual physiological differences of users, effectively preventing attackers from bypassing unlock verification by imitating or pre-programming gaze paths, further enhancing the security of the unlocking system.

[0075] Optionally, pupillary change characteristics are determined in the following ways: Acquire time-series data of pupil diameter changes when a user gazes at a target object; perform Fourier transform on the time-series data of pupil diameter changes to obtain frequency domain data, and extract the amplitude of at least one frequency component from the frequency domain data as a pupil change feature.

[0076] Among them, pupil diameter change time series data refers to the continuous change data sequence formed by the pupil diameter over time when a user is focusing on a target object. The size of the pupil diameter is not constant, but varies with the user's cognitive load, attention state, lighting conditions, and the activity of the autonomic nervous system, forming a time series signal with individual differences.

[0077] The amplitude of a frequency component refers to the intensity value of each frequency component after the original time-domain signal has been converted into a frequency-domain signal through a Fourier transform of the time-series data on pupil diameter changes. The larger the amplitude, the higher the proportion of that frequency component in the pupil diameter change.

[0078] In one implementation, the processor uses an eye-tracking module to collect data on the change in pupil diameter over time as the user gazes at various target objects, forming time-series data of pupil diameter change. The processor performs a Fourier transform on the time-series data of pupil diameter change, converting the time-domain signal into a frequency-domain signal, obtaining frequency-domain data. The processor extracts the amplitude of at least one frequency component from the frequency-domain data as a pupil change feature, for example, extracting the amplitudes of the first 10 frequency components to form a feature vector.

[0079] The above method uses Fourier transform to convert the time-domain variation signal of pupil diameter to the frequency domain for analysis, capturing the intensity distribution characteristics of different frequency components in the pupil diameter variation. This feature reflects the activity pattern of the user's autonomic nervous system, exhibiting significant individual variability and temporal stability, and is not easily affected by external factors such as ambient light. Compared with simply using time-domain statistics such as the mean pupil diameter, frequency domain features can more comprehensively describe the dynamic characteristics of pupil changes, enhancing the discriminative power of feature expression and thus improving the accuracy of unlocking verification.

[0080] Example 2 Figure 2 This is a flowchart of a gaze target determination method provided in Embodiment 2 of the present invention. It further optimizes and expands the implementation method of "determining at least two target objects that the user sequentially gazes at when wearing a head-mounted display device" in the above technical solution, and can be combined with the various optional implementation methods described above. For example... Figure 2 As shown, the method includes: S201. Obtain environmental images and environmental depth information in front of the head-mounted display device, and construct a three-dimensional spatial map based on the environmental images and environmental depth information.

[0081] In this context, environmental images refer to two-dimensional images acquired by a head-mounted display device through its image acquisition module, reflecting visual information about the actual physical environment in which the device is located. Environmental images include visual feature information such as the color, texture, and shape of various objects in the environment.

[0082] Environmental depth information refers to the depth data acquired by a head-mounted display device through its depth acquisition module, reflecting the distance information of various points in the real physical environment in which the device is located. Each pixel value in the environmental depth information represents the distance from the spatial point corresponding to that pixel to the depth camera, and is used to describe the three-dimensional structure of the environment.

[0083] A 3D spatial map is a digital model constructed by a processor based on environmental images and depth information, used to describe the 3D spatial structure of the real physical environment in which a head-mounted display device is located. The 3D spatial map contains information such as the 3D spatial position, size, and shape of each object in the environment, serving as the spatial index for subsequently determining the object the user is looking at. The 3D spatial map also contains the corresponding 3D bounding boxes for each object.

[0084] A 3D bounding box is the smallest rectangular prism used to approximate the 3D spatial extent of objects in a 3D map. 3D bounding boxes are axis-aligned, with their edges parallel to the coordinate axes of 3D space. They are defined by recording the minimum and maximum spatial coordinates of the objects. 3D bounding boxes are used for subsequent rapid intersection detection of line-of-sight rays and objects.

[0085] In one implementation, the processor acquires environmental images in front of the head-mounted display device using a red-green-blue color spatial camera and acquires corresponding environmental depth information using a time-of-flight depth camera. Based on the environmental images and depth information, the processor constructs a 3D spatial map of the real physical environment in which the head-mounted display device is located using simultaneous localization and mapping (SLAM) technology. The processor identifies objects in the environmental images using semantic segmentation technology and determines their position and size in 3D space based on their depth information, generating a corresponding 3D bounding box for each object in the 3D spatial map.

[0086] S202. Generate a gaze ray based on the eye-tracking data of the head-mounted display device, and determine whether there are intersections between the gaze ray and each 3D bounding box.

[0087] Among them, eye-tracking data refers to data collected by the processor through the eye-tracking module of the head-mounted display device, which reflects the user's eye movement status.

[0088] A gaze ray is a spatial ray that originates from the optical center of the image acquisition module on a head-mounted display device and extends along the user's current line of sight. The gaze ray is used to detect which object in three-dimensional space the user is currently looking at.

[0089] In one implementation, the processor acquires the user's eye-tracking data in real time through the eye-tracking module of the head-mounted display device, and determines the user's current gaze direction vector from the eye-tracking data. The processor uses the optical center of the image acquisition module on the head-mounted display device as the starting point of the gaze ray and the gaze direction vector as the direction to generate a gaze ray. The processor calculates the spatial geometric relationship between the gaze ray and the corresponding 3D bounding boxes of each object in the 3D spatial map, and determines whether there are any intersection points between the gaze ray and each 3D bounding box.

[0090] Specifically, let the optical center of the camera be... The unit vector of the line of sight is Then the line of sight ray is represented as: ; in, This represents the distance parameter along the line of sight. The processor performs intersection detection between the line of sight ray and the 3D bounding boxes of each object in the 3D spatial map, calculating whether there are intersection points between the line of sight ray and each 3D bounding box.

[0091] S203. If there is an intersection, the object with the intersection is identified as the target object that the user is looking at.

[0092] In one implementation, the processor makes a judgment based on the intersection detection results: if the gaze ray intersects with the 3D bounding box of an object, the processor determines that object as the target object currently being gazed at by the user; if the gaze ray intersects with multiple 3D bounding boxes simultaneously, the processor determines the object corresponding to the intersection closest to the origin of the gaze ray as the target object currently being gazed at by the user; if the gaze ray does not intersect with any 3D bounding boxes, the processor determines that the user is not currently gazing at any target object, and continues to acquire the user's eye-tracking data in real time through the eye-tracking module, generating new gaze rays for detection, until the target object being gazed at by the user is determined or the unlock verification times out.

[0093] The above steps determine the target object being gazed at by detecting the intersection of the gaze ray and the 3D bounding box. This simplifies the precise intersection calculation into a fast intersection detection, reducing computational complexity, ensuring real-time unlocking verification, and providing a reliable data foundation for subsequent feature extraction and matching. These steps fully utilize the standard RGB camera, depth camera, and eye-tracking module of the head-mounted display device, requiring no additional hardware and demonstrating good feasibility and cost advantages.

[0094] Example 3 Figure 3This is a flowchart of a method for determining the semantic features of objects according to Embodiment 3 of the present invention. It further optimizes and expands the implementation method of "determining the semantic features of each target object" in the above-described technical solutions, and can be combined with the various optional implementation methods described above. For example... Figure 3 As shown, the method includes: S301. Obtain an image of the environment in front of the head-mounted display device.

[0095] S302. Perform semantic segmentation on the image region where the target object is located in the environmental image to obtain the category label and semantic feature vector of the target object.

[0096] Semantic segmentation refers to the classification of each pixel in an image, labeling pixel regions belonging to the same object as the same category, thereby distinguishing different objects and their boundaries in an image. Semantic segmentation can accurately locate the image region where a target object is located in an environmental image, providing a precise pixel range for subsequent category recognition and feature extraction.

[0097] Category labels are category identifiers output by semantic segmentation models after classifying target object image regions. They describe "what the object is," such as "computer," "keyboard," or "water cup." Category labels are the result of discretizing and classifying objects from a semantic perspective.

[0098] A semantic feature vector is a high-dimensional numerical vector, such as a 1024-dimensional feature vector, extracted by a processor from an image region of a target object using a semantic segmentation model. It is used to quantify the visual appearance of the object. Semantic feature vectors numerically represent the visual features of an object from multiple dimensions, including shape, texture, and color, and are used to compare them with corresponding features in the template feature set during the unlocking verification process.

[0099] In one implementation, the processor acquires an environmental image in front of the head-mounted display device and determines the image region where the target object is located based on its position in the environmental image. The processor performs semantic segmentation on the image region containing the target object using a semantic segmentation model, identifies the object category corresponding to that region, obtains the category label for the target object, and extracts a high-dimensional semantic feature vector representing the visual appearance of the object using a feature extraction network within the semantic segmentation model. The processor uses both the category label and the semantic feature vector as the object's semantic features.

[0100] S303. Use category labels and semantic feature vectors as semantic features of objects.

[0101] In one implementation, the processor combines the category label and semantic feature vector together as the object semantic features of the target object, which are then used for subsequent unlock verification matching.

[0102] The above steps use semantic segmentation technology to perform pixel-level recognition of the image region of the target object, obtain category labels and semantic feature vectors, realize a comprehensive representation of the semantic information of the object from two dimensions: discrete category and continuous appearance, improve the accuracy and robustness of object recognition, and provide a more reliable feature basis for subsequent matching verification.

[0103] Example 4 Figure 4 This is a flowchart of a method for determining relative spatial relationship features according to Embodiment 4 of the present invention. It further optimizes and expands the implementation method of "determining the relative spatial relationship features between the target objects" in the above technical solution, and can be combined with the above optional implementation methods. For example... Figure 4 As shown, the method includes: S401. Obtain the object depth information of each target object and the pose information of the head-mounted display device, and determine the position information of each target object in three-dimensional space based on the object depth information and pose information.

[0104] Among them, object depth information refers to the distance data from each target object to the depth acquisition module. Object depth information reflects the position coordinates of the target object along the depth direction in three-dimensional space, and is one of the basic data for determining the three-dimensional spatial position of the object.

[0105] Pose information refers to the position and orientation data of a head-mounted display device in three-dimensional space. Position refers to the device's three-dimensional coordinates in the world coordinate system, and orientation refers to the device's rotation angles around each coordinate axis. Pose information is provided in real time by a simultaneous localization and mapping (SMR) system, used to transform the target object from the device coordinate system to the world coordinate system.

[0106] Position information refers to the three-dimensional spatial coordinates of a target object in the world coordinate system. Position information is the fundamental data for calculating the relative distance and relative angle between objects.

[0107] In one implementation, the processor acquires the object depth information of each target object through a time-of-flight depth camera and acquires the current pose information of the head-mounted display device in the world coordinate system through a simultaneous localization and mapping (SLAM) system. The processor performs coordinate transformation calculations based on the object depth and pose information, transforming each target object from the device coordinate system to the world coordinate system, and determining the three-dimensional spatial position information of each target object in the world coordinate system.

[0108] S402. Calculate the relative distance between any two target objects based on the position information of each target object.

[0109] Relative distance refers to the Euclidean distance between the three-dimensional spatial positions of any two target objects in the world coordinate system, used to describe how close or distant the two target objects are in space. Relative distance does not depend on the origin of the world coordinate system; it only reflects the relative positional relationship between the two objects.

[0110] In one implementation, the processor calculates the relative distance between any two target objects based on the determined three-dimensional spatial position information of each target object.

[0111] Specifically, let the target object be... The spatial location is Target object The spatial location is Then the target object With the target object relative distance between Represented as: ; The processor substitutes the three-dimensional spatial position information of each target object into the above formula and calculates the relative distance between any two target objects one by one.

[0112] S403. Calculate the relative angle between any two target objects based on the position information of each target object and the current viewing direction of the head-mounted display device.

[0113] Here, the current gaze direction refers to the unit direction vector of the user's gaze in the world coordinate system, which is obtained in real time by the eye-tracking module of the head-mounted display device.

[0114] A relative angle is the angle between the direction vector pointing from one target object to another, with one target object as its vertex, and the user's current line of sight. The relative angle reflects the spatial relationship between the target objects from the user's current viewing perspective.

[0115] In one implementation, the processor obtains the user's current gaze direction through an eye-tracking module, denoted as vector g. Based on the determined three-dimensional spatial position information of each target object, the processor calculates the direction vector pointing from one target object to another, using one target object as a vertex, and then calculates the angle between this direction vector and the current gaze direction, which is taken as the relative angle between the two target objects.

[0116] Specifically, let the target object be... The spatial location is Target object The spatial location is If the current viewing direction unit vector is g, then the target object... With the target object The relative angle between Represented as: ; in, For target object Pointing to target object directional vector, Let be the magnitude of the direction vector (i.e., the relative distance between two target objects), the numerator represent the dot product of the direction vector and the current viewing direction, and the denominator is used for normalization. The processor substitutes the three-dimensional spatial position information of each target object and the current viewing direction into the above formula to calculate the relative angle between any two target objects one by one.

[0117] S404. Relative distance and relative angle are used as characteristics of relative spatial relationships.

[0118] The above steps incorporate the positional information of each target object in three-dimensional space, the relative distance between any two target objects, and the relative angle calculated based on the current viewing direction into the spatial relationship feature, achieving a multi-dimensional quantitative description of the spatial layout between objects. Relative distance reflects the proximity of objects, while relative angle reflects the spatial distribution relationship of objects from the user's current viewing perspective. The combination of both effectively constrains the spatial attributes of the password. Since both relative distance and relative angle are relative quantities and do not depend on the absolute position of objects in the world coordinate system, even if the wearing position of the head-mounted display device changes, as long as the relative spatial layout between objects and the user's viewing perspective remain unchanged, this feature can still match correctly. When objects are moved or the user observes from different angles, the relative angle changes accordingly while the relative distance remains constant. This means that the spatial relationship feature implicitly contains the relative positional information between the user and the target objects, further enhancing the protection of the password unlocking function and improving the spatial robustness and security of the solution.

[0119] Example 5 Figure 5 This is a flowchart of a matching verification method provided in Embodiment 5 of the present invention. It further optimizes and expands the implementation method of "matching the real-time feature set with the pre-stored template feature set" in the above technical solution, and can be combined with the various optional implementation methods described above. For example... Figure 5 As shown, the method includes: S501. Determine the semantic similarity of objects between the real-time feature set and the template feature set.

[0120] S502. Determine the relative spatial relationship feature similarity between the real-time feature set and the template feature set.

[0121] S503. Determine the similarity of eye-tracking dynamic features between the real-time feature set and the template feature set.

[0122] Among them, the semantic feature similarity of objects refers to the degree of matching between the semantic features of each target object in the real-time feature set and the semantic features of the corresponding object in the template feature set. It is used to measure the consistency between the currently viewed object and the registered object in terms of category and visual appearance.

[0123] Relative spatial relationship feature similarity refers to the degree of matching between the spatial layout (including relative distance and relative angle) of each target object in the real-time feature set and the corresponding spatial layout in the template feature set. It is used to measure the consistency between the spatial arrangement of objects in the current environment and the registration.

[0124] Eye movement dynamics feature similarity refers to the degree of matching between the eye movement behavior features of a user when gazing at various target objects in the real-time feature set and the corresponding eye movement behavior features in the template feature set. It is used to measure the consistency between the current user's eye movement pattern and that of the registered user.

[0125] In one implementation, the processor calculates the similarity between the real-time feature set and the template feature set in three dimensions: (a) Semantic feature similarity of objects The processor calculates the cosine similarity between the semantic feature vectors of each target object in the real-time feature set and the semantic feature vectors of the corresponding objects in the template feature set. Let the semantic feature vector of the current target object be... The semantic feature vector of the corresponding object in the template feature set is Then the semantic feature similarity of objects Represented as: ; Here, the numerator represents the dot product of the two feature vectors, and the denominator represents the product of the magnitudes of the two feature vectors. The cosine similarity ranges from [-1, 1], with values ​​closer to 1 indicating greater similarity between the two vectors.

[0126] (ii) Similarity of relative spatial relationship features The processor calculates the similarity between the relative spatial relationship vectors between target objects in the real-time feature set and the corresponding relative spatial relationship vectors in the template feature set. Let the current spatial relationship be... If the template spatial relationship is Dt, then the relative spatial relationship feature similarity is... Calculated using Gaussian kernel function: ; in, This represents the Euclidean distance between the current spatial relation and the template spatial relation. This is the position tolerance parameter, used to control the rate at which similarity decreases with increasing distance. The Gaussian kernel function has a range of (0,1], and the closer the distance, the closer the similarity is to 1.

[0127] (III) Similarity of eye movement dynamics features The processor calculates the similarity between the eye-tracking dynamics features in the real-time feature set when the user gazes at each target object and the corresponding eye-tracking dynamics features in the template feature set. Let the current eye-tracking dynamics feature be... Template features are Then the similarity of eye movement dynamic features Calculated using Gaussian kernel function: ; in, This represents the square of the Euclidean distance between the current eye movement feature and the template eye movement feature. This is a physiological fluctuation tolerance parameter used to control the rate at which similarity decays as feature differences increase.

[0128] S504. Determine the total similarity based on the semantic feature similarity of objects, the relative spatial relationship feature similarity, and the eye-tracking dynamics feature similarity.

[0129] In one implementation, the processor determines the semantic similarity of the objects. Relative spatial relationship feature similarity Similarity to eye movement dynamics features The total similarity is calculated by weighted summation. Specifically, it is expressed as: ; in, , , These are the weight coefficients corresponding to the semantic feature similarity of objects, the relative spatial relationship feature similarity, and the eye-tracking dynamics feature similarity, respectively, satisfying... + + =1. The specific values ​​of each weighting coefficient can be configured according to the security level requirements of the actual application scenario. For example, in a preferred embodiment, The value is 0.3. The value is 0.2. The value is set to 0.5. The weighting coefficient corresponding to the eye-tracking dynamics feature similarity is... The highest value is selected to reflect the core role of eye-tracking dynamics in unlocking verification.

[0130] The processor uses the weighted summation result as the total similarity between the real-time feature set and the template feature set.

[0131] S505. If the total similarity reaches the preset threshold, the match is considered successful.

[0132] The above steps comprehensively evaluate three dimensions: object semantic features, relative spatial relationship features, and eye-tracking dynamics features. Inconsistency between any dimension and the template feature set leads to a decrease in overall similarity, significantly improving security when a single dimension is compromised. Simultaneously, spatial relationship similarity and eye-tracking dynamics similarity are calculated using a Gaussian kernel function, introducing a tolerance parameter to allow for slight shifts in object position and normal physiological fluctuations in eye movement features. This avoids false rejections due to minor environmental changes or user state fluctuations, ensuring robustness while maintaining security. Furthermore, by assigning the highest weight to eye-tracking dynamics features, even if an attacker knows the target object sequence and spatial layout, they cannot simulate the user's unique eye movement behavior patterns, effectively preventing password copying or imitation and significantly enhancing unlocking security.

[0133] Example 6 Figure 6 This is a flowchart of a template feature set update method provided in Embodiment Six of the present invention. It further optimizes and expands the implementation method following "matching the real-time feature set with the pre-stored template feature set" in the above technical solutions, and can be combined with the various optional implementation methods described above. For example... Figure 6 As shown, the method includes: S601. If the match is successful, obtain the template feature set before the update and the real-time feature set.

[0134] S602. Weight each feature value in the template feature set with the first weight to obtain the first weighted feature set, and weight each corresponding feature value in the real-time feature set with the second weight to obtain the second weighted feature set.

[0135] S603. Add the corresponding feature values ​​of the same dimension in the first weighted feature set and the second weighted feature set to obtain the updated template feature set.

[0136] Here, the first weight refers to the weight coefficient of the template feature set before the update during the template update process. The first weight is used to control the proportion of the old template in the new template after the update. The larger the value, the more the updated template tends to retain the features of the old template.

[0137] The first weighted feature set refers to the feature set obtained by multiplying each feature value in the template feature set before the update by the first weight.

[0138] The second weight refers to the weight coefficient of the real-time feature set during the template update process. The second weight controls the proportion of the newly acquired real-time feature set in the updated template; the larger the value, the more the updated template reflects the latest acquired data. The first weight is greater than the second weight, and the sum of the first and second weights is 1.

[0139] The second weighted feature set refers to the feature set obtained by multiplying each feature value in the real-time feature set by the second weight.

[0140] In one implementation, if a match is successful, the processor obtains the template feature set before the update. and the current real-time feature set The processor is given the highest weight. The first weighted feature set is obtained by weighting the feature values ​​in the template feature set. At the same time, with the second weight The corresponding feature values ​​in the real-time feature set are weighted to obtain the second weighted feature set. The processor adds the corresponding feature values ​​of the same dimension from the first weighted feature set to the second weighted feature set to obtain the updated template feature set. : ; in, ∈[0,1], +( )=1. In a preferred embodiment, The value is 0.9. The value is 0.1.

[0141] The above steps dynamically update the template feature set upon successful matching, allowing the template to drift slowly in response to user physiological changes (such as the gradual change in eye movement habits with age) and environmental changes (such as lighting conditions and slight object movements), maintaining the long-term stability and accuracy of unlock verification. The first weight is greater than the second weight, ensuring that the updated template feature set is based on the old template with fine-tuning based on real-time features. This avoids drastic changes in the template due to a single abnormal data collection, guaranteeing the template's stability and robustness. Simultaneously, after each successful verification, only the current features are fused with a smaller weight, making the template update process smooth and gradual, preventing the security risk of rapid contamination of the template by malicious samples.

[0142] Example 7 Figure 7 This is a flowchart of an anti-stress detection method provided in Embodiment 7 of the present invention. The implementation methods of the above technical solutions are further optimized and expanded, and can be combined with the various optional implementation methods described above. For example... Figure 7 As shown, the method includes: S701. During the process of the user sequentially looking at each target object, the user's eye movement behavior data is collected in real time.

[0143] S702. Based on eye movement data, determine whether there is a preset coercive eye movement pattern.

[0144] S703. If a coercive eye-tracking mode is present, enter the deception unlock mode and send a distress signal in the deception unlock mode.

[0145] Eye-tracking behavior data refers to data collected in real time by the processor through the eye-tracking module during the user's gaze at a target object, reflecting the user's eye movement state and change patterns. Eye-tracking behavior data includes, but is not limited to, parameters such as gaze point coordinates, pupil diameter, blink frequency, saccade pattern, and gaze duration, used to identify whether the user's eye-tracking behavior contains coercive features.

[0146] Stress eye-tracking patterns refer to covert distress signals emitted by users through specific combinations of eye movements when under duress. These patterns are pre-set by the user during registration. The covert nature of these patterns makes it difficult for bystanders to detect that the user is sending distress signals through eye movements.

[0147] Spoofing unlock mode refers to a fake unlock state that the processor enters after detecting a coerced eye-tracking pattern. In this mode, the processor superficially performs the unlock operation and presents the user with the normal device interface, but in reality, it is a pre-configured fake interface that hides the user's real applications and data. Spoofing unlock mode aims to mislead coercers into believing that the device has been unlocked normally, thereby preventing users from being further threatened if they refuse to unlock.

[0148] A distress signal is an emergency message containing the user's current location and on-site information that is sent out silently (without any sound, vibration, or display prompts) after the processor enters the deception unlock mode.

[0149] In one implementation, during the user's unlocking verification process of sequentially looking at each target object, the processor simultaneously collects the user's eye-tracking behavior data in real time via an eye-tracking module. The processor analyzes the collected eye-tracking behavior data to determine whether there are any behavioral characteristics that match a preset coercive eye-tracking pattern.

[0150] If the processor detects a coerced eye movement pattern in the user's eye movement data (such as a preset combination of "three rapid blinks accompanied by abnormal pupil constriction"), it determines that the user is currently under coercion. In this case, the processor does not execute the normal unlock verification process but immediately enters a deceptive unlock mode—the processor superficially performs the unlock operation and presents the user with a normal device interface, but in reality, it presents a pre-configured fake interface that hides all sensitive applications and data. Simultaneously, the processor silently sends a distress signal containing the current location information to preset emergency contacts or alarm platforms.

[0151] The above steps achieve a covert distress signal function by embedding coercion detection into the unlocking verification process without requiring additional user intervention. Because the coercion eye-tracking mode is user-preset and covert, the coercer cannot detect that the user is sending a distress signal, ensuring the user's safety. Entering the deceptive unlocking mode presents a fake interface, misleading the coercer into believing the device has been unlocked normally, preventing further coercion if the user refuses to unlock. The distress signal is sent silently, without any perceptible prompts, further ensuring the covert nature of the distress request. The entire anti-coercion process is completed through the user's natural eye-tracking behavior, requiring no additional active user action, thus balancing security and covertness.

[0152] In a preferred embodiment, the head-mounted display device includes an eye-tracking module, a red-green-blue color space camera, a time-of-flight depth camera, a simultaneous localization and mapping (SLAM) system, an inertial measurement unit, a processor, and a memory.

[0153] The eye-tracking module is used to collect the user's eye-tracking data. Preferably, the eye-tracking module uses a binocular infrared camera with a frame rate of not less than 120fps, a resolution of not less than 400×400, a gaze point positioning error of not more than 1.0°, and a data processing delay of not more than 3ms.

[0154] A red-green-blue (IRB) spatial camera is used to capture environmental images in front of a head-mounted display device. Preferably, the IRB camera has a resolution of at least 13MP, a frame rate of at least 30fps, and supports autofocus and high dynamic range imaging.

[0155] A time-of-flight depth camera is used to collect environmental depth information. Preferably, the time-of-flight depth camera has an accuracy of not less than 1 cm, a frame rate of not less than 30 fps, and a detection distance of not less than 5 m.

[0156] The simultaneous localization and mapping (SLAM) system is used to track the pose information of a head-mounted display device in real time. Preferably, the SLAM system supports real-time six-degree-of-freedom head tracking from the inside out, with a positioning accuracy of no more than 1 cm.

[0157] An inertial measurement unit (IMU) is used to detect the wearing status of the head-mounted display device to detect whether the user intends to unlock the head-mounted display device. Preferably, the IMU is a 9-axis IMU with a sampling rate of not less than 1000 Hz and an error of not more than 0.1°.

[0158] The processor is used to execute the steps of the unlocking method of the present invention. Preferably, the processor has a dedicated neural network processing unit that supports real-time artificial intelligence inference.

[0159] The memory is used to store the template feature set and program instructions. Preferably, the memory includes at least 8GB of RAM and at least 128GB of flash memory, and is equipped with a trusted execution environment for securely storing the template feature set.

[0160] Preferably, the overall power consumption of the head-mounted display device does not exceed 500mW and supports continuous use for no less than 4 hours.

[0161] In a preferred embodiment, the unlocking method also overcomes the impact of environmental changes on unlocking verification in the following way: I. Overcoming problems related to object movement.

[0162] During each verification process, the processor recalculates the 3D spatial position of each object based on the real-time acquired environmental images and environmental depth information, and compares it with the corresponding position information recorded in the template feature set. If the position change of an object is within a preset range (e.g., no more than 50cm), the processor automatically updates the spatial position information of that object in the template feature set.

[0163] Since the semantic features of objects and the eye-tracking dynamics features do not depend on the absolute spatial position of objects, when multiple objects move simultaneously but their relative spatial relationships remain unchanged (for example, when a user moves an entire bookshelf from one place to another), the processor can still correctly identify the sequence of target objects by matching based on the relative spatial relationship features (i.e., the relative distance and relative angle between any two target objects).

[0164] II. Overcoming the problem of changing light.

[0165] The semantic segmentation model uses a training dataset containing various lighting conditions (including strong light, weak light, backlight, and sidelight) during the training phase, which enables it to generalize well to environmental images under different lighting conditions.

[0166] During the unlocking and verification process, the processor detects the light intensity of the ambient image in real time and automatically adjusts the brightness, contrast, and white balance parameters of the ambient image according to the lighting conditions to reduce the impact of lighting changes on the accuracy of object recognition.

[0167] Optionally, in the event of insufficient lighting, the processor acquires object contour information in the environmental image through the infrared acquisition module of the head-mounted display device to assist in object recognition.

[0168] Preferably, when extracting semantic features of an object, the semantic segmentation model prioritizes extracting features that are not sensitive to changes in lighting, such as the object's edges, texture, and shape, thereby reducing its reliance on color features.

[0169] III. Overcoming the problem of object occlusion.

[0170] When a target object is partially occluded, the processor uses the unoccluded image region of the object to extract and match semantic features, thereby reducing the impact of occlusion on recognition accuracy.

[0171] When a target object is completely occluded, the processor skips the completely occluded object and uses the remaining identifiable target objects for matching and verification. In a preferred embodiment, five target objects are preset during registration, and verification is considered successful as long as three or more target objects match successfully.

[0172] IV. Overcoming the interference problem caused by similar objects.

[0173] Optionally, the semantic segmentation model employs a fine-grained semantic segmentation model, capable of identifying differences between individuals within the same category (e.g., distinguishing between two different cups or two different books).

[0174] In a preferred embodiment, the processor also extracts local features of the target object (such as the object's logo, pattern, scratches, etc.) for auxiliary recognition to improve the ability to distinguish between objects with similar appearances.

[0175] At the same time, the processor combines the characteristics of relative spatial relationships (i.e., the relative distance and relative angle between any two target objects) for verification. Even if two objects look similar, the system can correctly distinguish them as long as their spatial positions are different.

[0176] In a preferred embodiment, the unlocking method further enhances overall security through the following security mechanisms: I. Anti-peeping mechanism This invention uses the user's unique eye movement dynamics as one of the verification dimensions. Even if others learn the sequence of target objects that the user is looking at by observation, they cannot replicate the user's unique eye movement dynamics (such as fixation time characteristics, saccade speed characteristics, microsaccade characteristics, and pupil change characteristics), and thus cannot pass the unlock verification.

[0177] Optionally, the processor can randomly adjust the verification order of each target object during each verification to prevent others from learning the user's gaze path pattern through multiple observations.

[0178] Optionally, the processor also supports a "fake" function, which allows users to first look at several non-password objects and then look at the correct sequence of target objects to complete the unlocking process, further enhancing the concealment of the verification process.

[0179] II. Data Security Mechanism The template feature set is encrypted and stored in the trusted execution environment of the head-mounted display device to ensure the security of the template feature set.

[0180] The user's eye-tracking data, eye-movement behavior data, environmental images, and environmental depth information are processed only on the local processor and are not uploaded to the cloud server.

[0181] Optionally, the head-mounted display device allows users to periodically update or delete stored template feature sets.

[0182] Example 8 Figure 8 This is a schematic diagram of the structure of an unlocking device for a head-mounted display device provided in Embodiment 8 of the present invention. It can be applied to scenarios where users need to unlock the head-mounted display device for normal use, such as... Figure 8 As shown, the device includes: The target object determination module 81 is used to determine at least two target objects that the user is sequentially looking at while wearing the head-mounted display device when the system detects that the user intends to unlock the head-mounted display device; wherein the target objects are objects in the real physical environment in which the head-mounted display device is located. The first feature determination module 82 is used to determine the semantic features of each of the target objects and the relative spatial relationship features between the target objects. The second feature determination module 83 determines the eye movement dynamics features corresponding to the user's sequential gaze at each of the target objects; wherein, the eye movement dynamics features include at least one of gaze duration features, saccade speed features, microsaccade features, and pupillary change features; The matching module 84 is used to generate a real-time feature set based on the semantic features of the object, the relative spatial relationship features, and the eye-tracking dynamics features, and to match the real-time feature set with a pre-stored template feature set; The unlocking module 85 is used to unlock the head-mounted display device if the matching is successful.

[0183] Optionally, the target object determination module 81 is specifically used for: The system acquires an environmental image and environmental depth information in front of the head-mounted display device, and constructs a three-dimensional spatial map based on the environmental image and environmental depth information; wherein the three-dimensional spatial map contains three-dimensional bounding boxes corresponding to each object; A gaze ray is generated based on the eye-tracking data of the head-mounted display device, and it is determined whether there are any intersections between the gaze ray and each of the three-dimensional bounding boxes; If the intersection point exists, the object with the intersection point is identified as the target object being viewed by the user.

[0184] Optionally, the first feature determination module 82 is specifically used for: Acquire an image of the environment in front of the head-mounted display device; Semantic segmentation is performed on the image region where the target object is located in the environmental image to obtain the category label and semantic feature vector of the target object; The category label and the semantic feature vector are used as the semantic features of the object.

[0185] Optionally, the first feature determination module 82 is further configured to: The object depth information of each target object and the pose information of the head-mounted display device are obtained, and the position information of each target object in three-dimensional space is determined based on the object depth information and the pose information. Calculate the relative distance between any two target objects based on the position information of each target object; Based on the position information of each target object and the current viewing direction of the head-mounted display device, calculate the relative angle between any two target objects; The relative distance and the relative angle are used as the features of the relative spatial relationship.

[0186] Optionally, the gaze duration feature is determined in the following manner: Acquire data on the user's gaze duration when gazing at the target object; The gaze duration data is fitted using a gamma distribution, and the shape and scale parameters of the gamma distribution are extracted as the gaze duration features.

[0187] Optionally, the saccade speed characteristic is determined in the following manner: Obtain the time series of the user's gaze velocity when looking at the target object; Statistical features are extracted from the saccade velocity time series, and the saccade velocity features are determined based on the statistical features; wherein, the statistical features include at least one of average velocity, velocity standard deviation, peak velocity, average acceleration, and dominant frequency component.

[0188] Optionally, the microsaccade features include at least one of microsaccade frequency, average microsaccade amplitude, and microsaccade directional distribution entropy.

[0189] Optionally, the pupil change characteristics are determined in the following manner: Acquire time-series data on pupil diameter changes when a user gazes at the target object; The time-series data of pupil diameter change is subjected to Fourier transform to obtain frequency domain data, and the amplitude of at least one frequency component is extracted from the frequency domain data as the pupil change feature.

[0190] Optionally, the matching module 84 is specifically used for: Determine the semantic similarity of objects between the real-time feature set and the template feature set; Determine the relative spatial relationship feature similarity between the real-time feature set and the template feature set; Determine the eye-tracking dynamics feature similarity between the real-time feature set and the template feature set; The total similarity is determined based on the semantic feature similarity of the objects, the relative spatial relationship feature similarity, and the eye-tracking dynamics feature similarity. If the total similarity reaches a preset threshold, the match is considered successful.

[0191] Optionally, the device further includes a template feature set update module, specifically used for: If the match is successful, obtain the template feature set before the update and the real-time feature set; The first weighted feature set is obtained by weighting each feature value in the template feature set with a first weight, and the second weighted feature set is obtained by weighting each corresponding feature value in the real-time feature set with a second weight. The first weighted feature set and the second weighted feature set are added together to obtain the updated template feature set. Wherein, the first weight is greater than the second weight, and the sum of the first weight and the second weight is 1.

[0192] Optionally, the device further includes an anti-stress detection module, specifically used for: During the process of the user sequentially looking at each of the target objects, the user's eye movement data is collected in real time; Based on the eye movement data, determine whether a preset coercive eye movement pattern exists; If the coercive eye-tracking pattern is present, the system enters the deception unlocking mode and sends a distress signal in the deception unlocking mode.

[0193] The unlocking device for a head-mounted display device provided in this embodiment of the invention can execute the unlocking method for a head-mounted display device provided in this embodiment of the invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0194] According to embodiments of this disclosure, embodiments of the present invention also provide an electronic device, a readable storage medium, and a computer program product.

[0195] Figure 9A schematic diagram of an electronic device 90 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0196] like Figure 9 As shown, the electronic device 90 includes at least one processor 91 and a memory, such as a read-only memory (ROM) 92 or a random access memory (RAM) 93, communicatively connected to the at least one processor 91. The memory stores computer programs executable by the at least one processor. The processor 91 can perform various appropriate actions and processes based on the computer program stored in the ROM 92 or loaded into the RAM 93 from storage unit 98. The RAM 93 can also store various programs and data required for the operation of the electronic device 90. The processor 91, ROM 92, and RAM 93 are interconnected via a bus 94. An input / output (I / O) interface 95 is also connected to the bus 94.

[0197] Multiple components in electronic device 90 are connected to I / O interface 95, including: input unit 96, such as keyboard, mouse, etc.; output unit 97, such as various types of displays, speakers, etc.; storage unit 98, such as disk, optical disk, etc.; and communication unit 99, such as network card, modem, wireless transceiver, etc. Communication unit 99 allows electronic device 90 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0198] Processor 91 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 91 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 91 performs the various methods and processes described above, such as the unlocking method for a head-mounted display device.

[0199] In some embodiments, the unlocking method for the head-mounted display device may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 98. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 90 via ROM 92 and / or communication unit 99. When the computer program is loaded into RAM 93 and executed by processor 91, one or more steps of the unlocking method for the head-mounted display device described above may be performed. Alternatively, in other embodiments, processor 91 may be configured to perform the unlocking method for the head-mounted display device by any other suitable means (e.g., by means of firmware).

[0200] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0201] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0202] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0203] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0204] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0205] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product within the cloud computing service system to address the shortcomings of traditional physical hosts and virtual private servers, such as high management difficulty and weak business scalability.

[0206] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0207] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for unlocking a head-mounted display device, characterized in that, include: If the system detects that the user intends to unlock the head-mounted display device, it determines at least two target objects that the user is sequentially looking at while wearing the head-mounted display device; wherein, the target objects are objects in the real physical environment in which the head-mounted display device is located. Determine the semantic features of each target object, and determine the relative spatial relationship features between each target object; Determine the eye movement dynamics characteristics corresponding to the user's sequential gaze at each of the target objects; wherein, the eye movement dynamics characteristics include at least one of gaze duration characteristics, saccade speed characteristics, microsaccade characteristics, and pupillary change characteristics; A real-time feature set is generated based on the object semantic features, the relative spatial relationship features, and the eye-tracking dynamics features, and the real-time feature set is matched with a pre-stored template feature set; If the pairing is successful, the head-mounted display device is unlocked.

2. The method according to claim 1, characterized in that, The determination of at least two target objects that the user sequentially gazes at while wearing the head-mounted display device includes: The system acquires an environmental image and environmental depth information in front of the head-mounted display device, and constructs a three-dimensional spatial map based on the environmental image and environmental depth information; wherein the three-dimensional spatial map contains three-dimensional bounding boxes corresponding to each object; A gaze ray is generated based on the eye-tracking data of the head-mounted display device, and it is determined whether there are any intersections between the gaze ray and each of the three-dimensional bounding boxes; If the intersection point exists, the object with the intersection point is identified as the target object being viewed by the user.

3. The method according to claim 1, characterized in that, Determining the semantic features of each of the target objects includes: Acquire an image of the environment in front of the head-mounted display device; Semantic segmentation is performed on the image region where the target object is located in the environmental image to obtain the category label and semantic feature vector of the target object; The category label and the semantic feature vector are used as the semantic features of the object.

4. The method according to claim 1, characterized in that, Determining the relative spatial relationship features between the target objects includes: The object depth information of each target object and the pose information of the head-mounted display device are obtained, and the position information of each target object in three-dimensional space is determined based on the object depth information and the pose information. Calculate the relative distance between any two target objects based on the position information of each target object; Based on the position information of each target object and the current viewing direction of the head-mounted display device, calculate the relative angle between any two target objects; The relative distance and the relative angle are used as the features of the relative spatial relationship.

5. The method according to claim 1, characterized in that, The gaze duration characteristics are determined in the following manner: Acquire data on the user's gaze duration when gazing at the target object; The gaze duration data is fitted using a gamma distribution, and the shape and scale parameters of the gamma distribution are extracted as the gaze duration features.

6. The method according to claim 1, characterized in that, The saccade speed characteristic is determined in the following way: Obtain the time series of the user's gaze velocity when looking at the target object; Statistical features are extracted from the saccade velocity time series, and the saccade velocity features are determined based on the statistical features; wherein, the statistical features include at least one of average velocity, velocity standard deviation, peak velocity, average acceleration, and dominant frequency component.

7. The method according to claim 1, characterized in that, The microsaccade features include at least one of microsaccade frequency, average microsaccade amplitude, and microsaccade directional distribution entropy.

8. The method according to claim 1, characterized in that, The pupillary change characteristics are determined in the following way: Acquire time-series data on pupil diameter changes when a user gazes at the target object; The time-series data of pupil diameter change is subjected to Fourier transform to obtain frequency domain data, and the amplitude of at least one frequency component is extracted from the frequency domain data as the pupil change feature.

9. The method according to claim 1, characterized in that, The step of matching the real-time feature set with the pre-stored template feature set includes: Determine the semantic similarity of objects between the real-time feature set and the template feature set; Determine the relative spatial relationship feature similarity between the real-time feature set and the template feature set; Determine the eye-tracking dynamics feature similarity between the real-time feature set and the template feature set; The total similarity is determined based on the semantic feature similarity of the objects, the relative spatial relationship feature similarity, and the eye-tracking dynamics feature similarity. If the total similarity reaches a preset threshold, the match is considered successful.

10. The method according to claim 1, characterized in that, After matching the real-time feature set with the pre-stored template feature set, the method further includes: If the match is successful, obtain the template feature set before the update and the real-time feature set; The first weighted feature set is obtained by weighting each feature value in the template feature set with a first weight, and the second weighted feature set is obtained by weighting each corresponding feature value in the real-time feature set with a second weight. The first weighted feature set and the second weighted feature set are added together to obtain the updated template feature set. Wherein, the first weight is greater than the second weight, and the sum of the first weight and the second weight is 1.

11. The method according to claim 1, characterized in that, Also includes: During the process of the user sequentially looking at each of the target objects, the user's eye movement data is collected in real time; Based on the eye movement data, determine whether a preset coercive eye movement pattern exists; If the coercive eye-tracking pattern is present, the system enters the deception unlocking mode and sends a distress signal in the deception unlocking mode.

12. An unlocking device for a head-mounted display device, characterized in that, include: The target object determination module is used to determine at least two target objects that the user is sequentially looking at while wearing the head-mounted display device when the system detects that the user intends to unlock the head-mounted display device; wherein the target objects are objects in the real physical environment in which the head-mounted display device is located. The first feature determination module is used to determine the object semantic features of each of the target objects, and to determine the relative spatial relationship features between the target objects. The second feature determination module determines the eye movement dynamics features corresponding to the user's sequential gaze at each of the target objects; wherein, the eye movement dynamics features include at least one of gaze duration features, saccade speed features, microsaccade features, and pupillary change features; The matching module is used to generate a real-time feature set based on the semantic features of the object, the relative spatial relationship features, and the eye-tracking dynamics features, and to match the real-time feature set with a pre-stored template feature set; An unlocking module is used to unlock the head-mounted display device if a match is successful.

13. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a processor to perform the method of any one of claims 1-11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images