Safe unlocking method and system based on AR glasses, equipment and storage medium

Abstract

The invention provides a security unlocking method and system based on AR glasses, a device and a storage medium, and belongs to the technical field of security unlocking, the method is used for unlocking a target device, the target device and the AR glasses are pre-paired, the target device and the AR glasses store a shared encryption key after pairing, the security unlocking method is executed by the AR glasses, and the security unlocking method is executed by the AR glasses. The method comprises the following steps: in response to successful connection with target equipment and receiving an unlocking request, collecting a first eye image of a wearer; determining identity information of the wearer based on the first eye image, and performing first verification on the identity information of the wearer; and if the first verification is passed, encrypting the first eye image based on the shared encryption key to obtain an encrypted eye image, and sending the encrypted eye image and an unlocking instruction to the target device, so that the target device executes an unlocking operation after receiving the encrypted eye image. According to the method and the device, the unlocking continuity can be improved on the premise of ensuring the safety.

Description

Technical Field

[0001] This application belongs to the field of secure unlocking technology, and more specifically, relates to a secure unlocking method, system, device, and storage medium based on AR glasses. Background Technology

[0002] With the rapid popularization of wearable smart devices such as Augmented Reality (AR) glasses, their efficient collaboration with target devices such as mobile terminals, smart home devices, and industrial terminals has become an important scenario for human-computer interaction. Unlocking, as the primary entry point for device interaction, needs to balance security with the consistency and convenience of the user experience.

[0003] However, due to the wide variety of AR glasses currently available, some AR glasses have a large proportion of opaque or low-transparency areas, which can obscure most of the wearer's eye area when worn. This can cause other devices using traditional facial recognition algorithms to fail to recognize the wearer's face when unlocking other devices, easily leading to verification failure. In such cases, the wearer needs to remove the AR glasses to unlock the device.

[0004] Therefore, there is an urgent need for an unlocking method that improves the consistency of unlocking while ensuring security. Summary of the Invention

[0005] The purpose of this application is to provide a secure unlocking method, system, device, and storage medium based on AR glasses, which improves the consistency of unlocking while ensuring security.

[0006] A first aspect of this application provides a secure unlocking method based on AR glasses for unlocking a target device. The target device and AR glasses are pre-paired, and after pairing, the target device and AR glasses store a shared encryption key. The secure unlocking method is executed by the AR glasses and includes: Upon successful connection to the target device and receipt of an unlock request, the system captures the wearer's first eye image. The wearer's identity information is determined based on the first eye image, and the wearer's identity information is verified for the first time. If the first verification passes, the first eye image is encrypted based on the shared encryption key to obtain an encrypted eye image. The encrypted eye image and unlock command are then sent to the target device so that the target device can perform the unlock operation after receiving the encrypted eye image. The unlocking operation includes: decrypting the encrypted eye image based on the shared encryption key to obtain the first eye image; and acquiring the wearer's facial image and determining the occlusion rate of the AR glasses on the second eye image in the facial image. The adjustment method for the second eye image is determined based on the occlusion rate; The second eye image is adjusted based on the adjustment method and the first eye image to obtain the adjusted facial image; the target occlusion rate of the eye image in the adjusted facial image is less than the occlusion rate. The wearer's identity is verified a second time based on the adjusted facial image; if the second verification is successful, the unlocking command is executed.

[0007] A second aspect of this application provides a secure unlocking system based on AR glasses, disposed in AR glasses for unlocking a target device. The target device and AR glasses are pre-paired, and after pairing, the target device and AR glasses store a shared encryption key. The secure unlocking system includes: The image acquisition module is used to acquire the first eye image of the wearer in response to a successful connection with the target device and the receipt of an unlocking request; The identity verification module is used to determine the wearer's identity information based on the first eye image and to perform the first verification of the wearer's identity information; The security unlocking module is used to encrypt the first eye image based on the shared encryption key after the first verification is successful, to obtain the encrypted eye image, and send the encrypted eye image and unlocking command to the target device so that the target device can perform the unlocking operation after receiving the encrypted eye image. The unlocking operation includes: decrypting the encrypted eye image based on the shared encryption key to obtain the first eye image; and acquiring the wearer's facial image and determining the occlusion rate of the AR glasses on the second eye image in the facial image. The adjustment method for the second eye image is determined based on the occlusion rate; The second eye image is adjusted based on the adjustment method and the first eye image to obtain the adjusted facial image; the target occlusion rate of the eye image in the adjusted facial image is less than the occlusion rate. The wearer's identity is verified a second time based on the adjusted facial image; if the second verification is successful, the unlocking command is executed.

[0008] A third aspect of this application provides an AR glasses device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of the above-described AR glasses-based security unlocking method.

[0009] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described secure unlocking method based on AR glasses.

[0010] The beneficial effects of the secure unlocking method, system, device, and storage medium based on AR glasses provided in this application are as follows: In this embodiment, the target device and AR glasses are pre-paired and share an encryption key to ensure data transmission security and prevent feature information from being stolen or tampered with. The AR glasses capture an unobstructed first eye image and complete the first verification, initially filtering out unauthorized users. The first eye image is encrypted and transmitted to the target device, and the second eye image is adjusted based on the occlusion rate to solve the feature loss problem caused by eye occlusion. A second verification is then performed. This dual verification mechanism further ensures the accuracy of identity verification and allows the wearer to unlock the device without removing the glasses, improving the continuity of unlocking. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating a secure unlocking method based on AR glasses provided in an embodiment of this application; Figure 2 This is a structural block diagram of a security unlocking system based on AR glasses provided in an embodiment of this application; Figure 3 This is a schematic block diagram of AR glasses provided in an embodiment of this application. Detailed Implementation

[0013] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0015] In one embodiment of this application, the secure unlocking method based on AR glasses is executed by the AR glasses to unlock a target device, which can be an electronic device such as a computer or mobile phone. Before unlocking, the target device and the AR glasses are pre-paired, for example, through Bluetooth, Near Field Communication (NFC), or a dedicated wireless protocol. After pairing, the target device and the AR glasses store a shared encryption key. This shared encryption key is a unique encryption credential for both the AR glasses and the target device, used to prevent images from being stolen or tampered with during transmission, and to ensure that only devices holding the same shared key can complete decryption, thus ensuring end-to-end security of data transmission.

[0016] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a secure unlocking method based on AR glasses according to an embodiment of this application. The method may include: S101: In response to a successful connection with the target device and receipt of an unlock request, capture the wearer's first eye image.

[0017] In this embodiment, a successful connection between the AR glasses and the target device means that a secure and stable communication connection has been established between the AR glasses and the target device. For example, if the channel quality of the communication link between the AR glasses and the target device is stronger than a preset channel quality threshold, the connection between the AR glasses and the target device is considered successful. The unlock request can be an unlock command initiated by the wearer through voice commands, gesture operations, or button input via the AR glasses, indicating that the wearer currently needs to unlock the target device.

[0018] In this embodiment, the AR glasses are equipped with a camera that can capture images of the wearer's eye area to obtain a first eye image.

[0019] S102: Determine the wearer's identity information based on the first eye image, and perform the first verification of the wearer's identity information.

[0020] In this embodiment, iris information can be extracted from the first eye image to determine the wearer's identity information; that is, the AR glasses pre-store the iris information of individuals authorized to use the security unlocking function. In this embodiment, the iris recognition method will not be described in detail.

[0021] S103: If the first verification is successful, the first eye image is encrypted based on the shared encryption key to obtain an encrypted eye image. The encrypted eye image and unlock command are sent to the target device so that the target device can perform the unlock operation after receiving the encrypted eye image.

[0022] In this embodiment, if the first verification passes, it means that the AR glasses terminal considers the current wearer's identity verification successful, and the target device can perform identity verification again. However, since the current wearer is wearing AR glasses, and the existing AR glasses are larger than regular glasses, they partially obstruct the eye area, which may cause the target device to fail to recognize the wearer. Therefore, in this embodiment, the target device can send an encrypted first eye image (encrypted eye image) and an unlock command, so that the target device can perform the following operations after receiving the encrypted eye image, thereby completing the second verification of identity information and the unlocking operation after successful verification: S1031: Decrypt the encrypted eye image based on the shared encryption key to obtain the first eye image; and collect the face image of the wearer and determine the occlusion rate of the second eye image in the face image by the AR glasses.

[0023] In this embodiment, after receiving the encrypted eye image sent by the AR glasses, the target device can decrypt it into a first eye image, and simultaneously capture the wearer's facial image. In this embodiment, the occlusion rate of the second eye image in the captured facial image refers to the proportion of the eye area occluded by the AR glasses to the total area of ​​the second eye image.

[0024] S1032: Determine the adjustment method for the second eye image based on the occlusion rate.

[0025] In one embodiment of this application, determining the adjustment method for the second eye image based on the occlusion rate includes: In response to the occlusion rate being lower than a preset occlusion rate threshold, the adjustment method of the second eye image is determined as the first adjustment method; In response to the occlusion rate not being lower than the occlusion rate threshold, the adjustment method of the second eye image is determined as the second adjustment method; The first adjustment method is to adjust the image of the occluded area in the second eye image, and the second adjustment method is to adjust the image of the second eye image as a whole.

[0026] In this embodiment, the adjustment method for the second eye image is essentially determined by comparing the occlusion rate with a preset occlusion rate threshold. If the occlusion rate is lower than the threshold, it is considered that the AR glasses obstruct a small area of ​​the eyes (e.g., only obscuring a small portion of the corner of the eye), and the second eye image still retains most of its effective features. In this case, local adjustments can be made, repairing only the obscured area while preserving the original, unobscured features of the second eye image. This not only fills in missing information but also maximizes the naturalness of the adjusted image, avoiding inconsistencies between the eye features and other facial areas caused by overall replacement.

[0027] If the occlusion rate is not lower than the occlusion rate threshold, it is considered that the AR glasses occlude a large area of ​​the eye (e.g., occluding half or the entire eye), and the effective features of the second eye image are insufficient to support verification. In this case, local adjustments cannot supplement enough effective features, and may even cause image distortion due to excessively large repair areas. Therefore, a complete replacement method is adopted, directly replacing the second eye image with the complete first eye image captured by the AR glasses. This ensures that the adjusted eye has complete features and meets the requirements of secondary verification.

[0028] S1033: Adjust the second eye image based on the adjustment method and the first eye image to obtain the adjusted facial image; the target occlusion rate of the eye image in the adjusted facial image is less than the occlusion rate.

[0029] In this embodiment, the facial image obtained after adjusting the second eye image based on the determined adjustment method, regardless of whether the adjustment method is the first adjustment method or the second adjustment method, should theoretically have a target occlusion rate of 0 for the eye image in the adjusted facial image, that is, the target occlusion rate is less than the occlusion rate of the second eye image in the facial image.

[0030] S1034: Perform a second verification of the wearer's identity information based on the adjusted facial image; if the second verification is successful, execute the unlock command.

[0031] In this embodiment, the target device can perform a second verification on the facial image after partial or complete replacement and adjustment. The difference between the second and first verification lies in the executing subject and the verification content. The executing subject for the first verification is the AR glasses, while the executing subject for the second verification is the target device. The verification content for the first verification is the iris information of the wearer, while the verification content for the second verification is the facial information of the wearer. If the second verification passes, an unlocking command can be executed to unlock itself (the target device). In this embodiment, the target device can perform the second verification based on common facial recognition methods in the art.

[0032] As can be seen from the above, in this embodiment, the target device and AR glasses are pre-paired and share an encryption key to ensure data transmission security and prevent feature information from being stolen or tampered with; the AR glasses collect an unobstructed first eye image and complete the first verification, initially filtering out unauthorized users; the first eye image is encrypted and transmitted to the target device, and the second eye image is adjusted based on the occlusion rate to solve the feature loss problem caused by eye occlusion; and a second verification is performed. The dual verification mechanism further ensures the accuracy of identity verification, while also enabling the wearer to unlock without removing the glasses, improving the continuity of unlocking.

[0033] In one embodiment of this application, if the adjustment method is a first adjustment method, then the second eye image is adjusted based on the adjustment method and the first eye image to obtain an adjusted facial image, including: The first eye image is transformed by a perspective transformation to obtain a third eye image, so that the perspective of the third eye image is consistent with that of the second eye image. Identify the extent of the occluded area in the second eye image; The target occlusion image is obtained by determining the position of the third eye image corresponding to the range of the occluded area; The adjusted facial image is obtained by replacing the occluded area in the second eye image with the target occluded image.

[0034] In this embodiment, since the first eye image is captured at close range by the AR glasses, with a frontal viewing angle, while the second eye image is captured by the target device and has an uncertain viewing angle, the viewing angles of the first and second eye images may be inconsistent. If the area of ​​the first eye image is directly used to replace the second eye image, it will cause a distortion in the viewing angle of the adjusted facial image, leading to failure of the secondary verification. Therefore, geometric transformations, such as rotation, scaling, and translation, can be performed on the first eye image to generate a third eye image.

[0035] Secondly, the area obscured by AR glasses in the second eye image can be identified by image semantic segmentation algorithms or edge detection algorithms. The specific range of the obscured area can be represented by position coordinates or range mask. In this embodiment, the coordinate system of the position coordinates can be a coordinate system established with the lower left corner of the second eye image as the origin, the lower edge as the X-axis, the rightward direction as the positive direction, the left edge as the Y-axis, and the upward direction as the positive direction.

[0036] In this embodiment, the image at the corresponding position in the third eye image is determined based on the range of the occluded area in the second eye image; this image is the target occluded image. Since the viewpoints of the second and third eye images correspond, the image within the range with the same position coordinates can be cropped as the target occluded image. By overlaying the occluded area in the second eye image with the target occluded image, and retaining the unoccluded area in the second eye image, the adjusted facial image can be obtained.

[0037] As can be seen from the above, the embodiments of this application perform geometric transformation on the first eye image to generate a third eye image with a matching perspective, eliminating the difference between the AR glasses' frontal view acquisition and the target device's variable perspective acquisition, avoiding image perspective distortion caused by direct replacement, improving the naturalness of the adjusted facial image, and reducing the probability of secondary verification failure; semantic segmentation or edge detection algorithms are used to accurately identify the occlusion area range of the second eye image, and the replacement boundary is clearly defined with coordinates or masks, ensuring that only the occluded area is repaired without interfering with the original facial features of the non-occluded area, further ensuring the authenticity of image features; while eliminating the influence of occlusion, the embodiments of this application maximize the preservation of the original features of the second eye image, improving the accuracy and stability of secondary verification. It not only solves the problem of unlock verification failure caused by AR glasses occlusion and perspective differences, but also allows verification to be completed without removing the glasses, improving unlocking consistency; and strengthens the security of identity verification by preserving original features and ensuring image authenticity, achieving a dual optimization of security and consistency.

[0038] In one embodiment of this application, a third eye image is obtained by performing a perspective transformation on a first eye image, including: Identify feature point information in the first and second eye images; Feature point alignment is performed based on feature point information to obtain the viewpoint deviation parameters of the first eye image and the second eye image; The third eye image is obtained by performing a geometric transformation on the first eye image based on the viewing angle deviation parameter.

[0039] In this embodiment, the feature point information in the first and second eye images may include: the center point of the pupil, the inner corner of the eye, or the outer corner of the eye, etc. Since the second eye image is captured by the target device while the wearer is wearing AR glasses, it may obscure some feature point information. The first eye image, on the other hand, is captured by the image acquisition device built into the AR glasses. By adjusting the installation position of the image acquisition device, all feature point information can be captured in the first eye image. Therefore, when only a portion of the feature points can be identified in the second eye image, feature point alignment can be performed based solely on those feature points to obtain the viewing angle deviation coefficient between the first and second eye images.

[0040] In one embodiment of this application, feature point alignment is performed based on feature point information to obtain the viewing angle deviation parameters of the first eye image and the second eye image, including: The first set of feature points corresponding to the first eye image and the second set of feature points corresponding to the second eye image are determined from the feature point information; each feature point type in the first set of feature points corresponds one-to-one with each feature point type in the second set of feature points. Select any feature point from the first feature point set as the first reference point to establish a first coordinate system; and select a feature point from the second feature points that corresponds to the type of the first reference point as the second reference point to establish a second coordinate system. Calculate the first coordinate vector of each feature point in the first feature point set relative to the first reference point in the first coordinate system, and calculate the second coordinate vector of each feature point in the second feature point set relative to the second reference point in the second coordinate system; Feature point alignment is performed based on the first and second coordinate vectors to obtain the viewing angle deviation parameters of the first and second eye images.

[0041] In this embodiment, feature points that completely correspond to the identifiable feature point types in the second eye image can be selected from the complete feature point set of the first eye image. These feature points are then used to form a first feature point set (from the first image) and a second feature point set (from the second image), with a one-to-one correspondence between the feature point types in the first and second feature point sets. The feature point types can be, for example, the pupil center point, the inner corner of the eye vertex, or the outer corner of the eye vertex.

[0042] In this embodiment, the purpose of establishing the first coordinate system and the second coordinate system is to eliminate the interference of absolute coordinates. Specifically, one feature point can be randomly selected from the first feature point set as the first reference point (e.g., the pupil center point A'). A first coordinate system is established with this point as the origin (e.g., the x-axis is horizontal to the right and the y-axis is vertical upward). From the second feature point set, a feature point of the same type as the first reference point is selected as the second reference point (e.g., the pupil center point A). A second coordinate system is established with this point as the origin, and the coordinate direction of the second coordinate system is consistent with that of the first coordinate system. For each feature point in the first feature point set, its coordinate vector relative to the first reference point in the first coordinate system (i.e., the difference between the coordinates of this point and the reference point, such as vector (Δx1, Δy1)) is calculated to obtain the first coordinate vector set. Similarly, the coordinate vector of each feature point in the second feature point set relative to the second reference point in the second coordinate system is calculated to obtain the second coordinate vector set. The coordinate vector contains both direction and magnitude information; the direction corresponds to the angle of the feature point relative to the reference point, and the magnitude corresponds to the distance between the feature point and the reference point. Finally, the viewpoint deviation parameter can be obtained by comparing the differences between the two sets of vectors.

[0043] In one embodiment of this application, feature point alignment is performed based on a first coordinate vector and a second coordinate vector to obtain a viewing angle deviation parameter between a first eye image and a second eye image, including: For each feature point in the first feature point set, calculate the difference between the first coordinate vector corresponding to the feature point and the second feature vector corresponding to the target feature point to obtain the target vector deviation value corresponding to the feature point; the target feature point is the feature point in the second coordinate vector that corresponds to the feature point. The viewing angle deviation parameters of the first eye image and the second eye image are determined based on the target vector deviation values ​​corresponding to each feature point in the first feature point set.

[0044] In this embodiment, the viewpoint deviation parameter may include rotation angle deviation, scaling ratio deviation, and perspective distortion deviation. Specifically, the viewpoint deviation coefficient can be determined based on the following method: Based on the deviation values ​​of each target vector, the relative pose of the first eye image and the second eye image is fitted using the least squares method to obtain the rotation angle deviation value; the rotation angle deviation value includes the horizontal rotation angle difference and the vertical pitch angle difference; Calculate the vector magnitude ratio of corresponding feature points in the first coordinate vector and the second coordinate vector, and determine the mean of the magnitude ratio as the scaling deviation value; Calculate the distribution gradient of the deviation values ​​of each target vector, and fit the perspective distortion deviation values ​​based on the distribution gradient.

[0045] In this embodiment, it is assumed that the feature points of the first eye image can be aligned with those of the second eye image after rotation. The rotation operation can be represented by a rotation matrix, which contains the difference in horizontal rotation angle and the difference in vertical pitch angle. In this embodiment, the difference in horizontal rotation angle and the difference in vertical pitch angle can be obtained by fitting based on the least squares method.

[0046] In this embodiment, the scaling deviation refers to the difference in eye size between two images, which can be quantified by the ratio of vector magnitudes. Vector magnitude is the length of a vector, representing the relative distance between a feature point and a reference point. The scaling deviation value can be obtained by calculating the ratio of the magnitudes of each pair of corresponding feature points and taking the average.

[0047] In this embodiment, the gradient represents the rate of change of the vector deviation value. The calculation logic is as follows: First, each feature point is sorted according to its distance from the reference point. The difference in vector deviation between adjacent feature points is calculated, and the rate of change of this difference is the distributed gradient. The larger the gradient, the more severe the perspective distortion. In this embodiment, the reference point is preset. Perspective distortion deviation is image distortion caused by differences in three-dimensional viewpoints. Perspective distortion can be described by a perspective transformation matrix, which contains four distortion parameters. Substituting the above-mentioned distributed gradient into the perspective transformation model, the parameter that minimizes the error between the predicted gradient and the actual gradient is obtained through fitting. This parameter is the perspective distortion deviation value.

[0048] As can be seen from the above, the embodiments of this application eliminate the interference of absolute coordinates by establishing a relative coordinate system, calculate the coordinate vector based on the corresponding feature points, and transform the perspective difference into a quantifiable vector parameter. This avoids the translational deviation interference caused by different acquisition positions and improves the accuracy of deviation calculation. Based on the vector deviation, the least squares method is used to fit the rotation angle, the average value of the modulus ratio is obtained to obtain the scaling ratio, and the gradient is used to fit the perspective distortion. This achieves accurate solution of multi-dimensional deviation parameters, fully covers the types of perspective differences, ensures that the perspective of the third eye image is completely matched with that of the second eye image, avoids image distortion after replacement, improves the naturalness and feature authenticity of the adjusted facial image, and improves the continuity of unlocking.

[0049] Considering that feature points in the second eye image may be occluded, and that the calculation of four distortion parameters is required in the aforementioned process of calculating perspective distortion deviation, it may be impossible to extract sufficient feature points from the second eye image and calculate the perspective distortion deviation. Therefore, in one embodiment of this application, the first eye image and the second eye image can be identified to obtain atypical feature points. Specifically, local feature extraction can be performed on the first eye image and the second eye image to obtain multiple local features, and target local features with response values ​​greater than a preset response value threshold are retained. Based on the K-nearest neighbor matching algorithm, feature points that match each target local feature point in the second eye image are determined from each target local feature point in the first eye image to obtain atypical feature points.

[0050] In this embodiment, local feature extraction can be achieved through a scale-invariant feature transformation algorithm or a feature algorithm focused on speed. It should be noted that in this embodiment, after determining the target local features, some feature points should be removed. These feature points refer to the feature points that have already been extracted.

[0051] As can be seen from the above, this embodiment completes the calculation of perspective distortion deviation value based on supplemented atypical feature points, realizes the complete correction of the difference in three-dimensional viewpoint, further improves the viewpoint matching degree between the third eye image and the second eye image, completely avoids the sense of incongruity in image adjustment caused by perspective distortion, ensures the feature authenticity of the adjusted facial image, and solves the problem that the typical feature points of the second eye image are occluded and insufficient in number, resulting in the inability to calculate the perspective distortion deviation value.

[0052] In one embodiment of this application, if the adjustment method is the second adjustment method, then the second eye image is adjusted based on the adjustment method and the first eye image to obtain an adjusted facial image, including: The first eye image is transformed by a perspective transformation to obtain a fourth eye image, so that the perspective of the fourth eye image is consistent with that of the second eye image. The adjusted facial image is obtained by replacing the second eye image with the fourth eye image.

[0053] In this embodiment, the perspective transformation process is the same as the first adjustment method described above, and will not be repeated here. Since the occlusion rate is greater than the preset occlusion rate threshold, the second adjustment method is complete replacement to prevent interference from unclear images.

[0054] Corresponding to the AR glasses-based security unlocking method in the above embodiments, Figure 2 This is a structural block diagram of a security unlocking system based on AR glasses, provided as an embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. References Figure 2 The AR glasses-based security unlocking system 20 includes: an image acquisition module 21, an authentication module 22, and a security unlocking module 23.

[0055] The image acquisition module 21 is used to acquire the first eye image of the wearer in response to a successful connection with the target device and the receipt of an unlocking request. The identity verification module 22 is used to determine the wearer's identity information based on the first eye image and to perform the first verification of the wearer's identity information; The security unlocking module 23 is used to encrypt the first eye image based on the shared encryption key when the first verification is successful, to obtain an encrypted eye image, and send the encrypted eye image and unlocking command to the target device so that the target device can perform the unlocking operation after receiving the encrypted eye image. The unlocking operation includes: decrypting the encrypted eye image based on the shared encryption key to obtain the first eye image; and acquiring the wearer's facial image and determining the occlusion rate of the AR glasses on the second eye image in the facial image. The adjustment method for the second eye image is determined based on the occlusion rate; The second eye image is adjusted based on the adjustment method and the first eye image to obtain the adjusted facial image; the target occlusion rate of the eye image in the adjusted facial image is less than the occlusion rate. The wearer's identity is verified a second time based on the adjusted facial image; if the second verification is successful, the unlocking command is executed.

[0056] In one embodiment of this application, the security unlocking module 23 is used to instruct the target device to determine the adjustment method of the second eye image as the first adjustment method in response to the occlusion rate being lower than a preset occlusion rate threshold. In response to the occlusion rate not being lower than the occlusion rate threshold, the adjustment method of the second eye image is determined as the second adjustment method; The first adjustment method is to adjust the image of the occluded area in the second eye image, and the second adjustment method is to adjust the image of the second eye image as a whole.

[0057] In one embodiment of this application, the security unlocking module 23 is used to instruct the target device to perform a perspective transformation on the first eye image to obtain a third eye image, so that the perspective of the third eye image is consistent with the perspective of the second eye image. Identify the extent of the occluded area in the second eye image; The target occlusion image is obtained by determining the position of the third eye image corresponding to the range of the occluded area; The adjusted facial image is obtained by replacing the occluded area in the second eye image with the target occluded image.

[0058] In one embodiment of this application, the security unlocking module 23 is used to instruct the target device to identify feature point information in the first eye image and the second eye image; Feature point alignment is performed based on feature point information to obtain the viewpoint deviation parameters of the first eye image and the second eye image; The third eye image is obtained by performing a geometric transformation on the first eye image based on the viewing angle deviation parameter.

[0059] In one embodiment of this application, the security unlocking module 23 is used to instruct the target device to determine from the feature point information a first set of feature points corresponding to the first eye image and a second set of feature points corresponding to the second eye image; each feature point type in the first set of feature points corresponds one-to-one with each feature point type in the second set of feature points; Select any feature point from the first feature point set as the first reference point to establish a first coordinate system; and select a feature point from the second feature points that corresponds to the type of the first reference point as the second reference point to establish a second coordinate system. Calculate the first coordinate vector of each feature point in the first feature point set relative to the first reference point in the first coordinate system, and calculate the second coordinate vector of each feature point in the second feature point set relative to the second reference point in the second coordinate system; Feature point alignment is performed based on the first and second coordinate vectors to obtain the viewing angle deviation parameters of the first and second eye images.

[0060] In one embodiment of this application, the security unlocking module 23 is used to instruct the target device to calculate the difference between the first coordinate vector corresponding to each feature point in the first feature point set and the second feature vector corresponding to the target feature point, so as to obtain the target vector deviation value corresponding to the feature point; the target feature point is the feature point in the second coordinate vector that corresponds to the feature point. The viewing angle deviation parameters of the first eye image and the second eye image are determined based on the target vector deviation values ​​corresponding to each feature point in the first feature point set.

[0061] In one embodiment of this application, the security unlocking module 23 is used to instruct the target device to perform a perspective transformation on the first eye image to obtain a fourth eye image, so that the perspective of the fourth eye image is consistent with the perspective of the second eye image. The adjusted facial image is obtained by replacing the second eye image with the fourth eye image.

[0062] See Figure 3 , Figure 3 This is a schematic block diagram of an AR glasses device provided in one embodiment of this application. Figure 3 The AR glasses device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of each module / unit in the above system embodiments, for example... Figure 2 The functions of the image acquisition module 21, the authentication module 22, and the security unlocking module 23 are shown.

[0063] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0064] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0065] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0066] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation method described in the AR glasses security unlocking method provided in the embodiments of this application, or they can execute the AR glasses implementation method described in the embodiments of this application, which will not be repeated here.

[0067] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0068] The computer-readable storage medium can be an internal storage unit of the AR glasses in any of the foregoing embodiments, such as the hard drive or memory of the AR glasses. The computer-readable storage medium can also be an external storage device of the AR glasses, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the AR glasses. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the AR glasses. The computer-readable storage medium is used to store computer programs and other programs and data required by the AR glasses. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0069] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0070] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the AR glasses and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0071] In the several embodiments provided in this application, it should be understood that the disclosed AR glasses and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0072] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0073] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0074] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A secure unlocking method based on AR glasses, characterized in that, For unlocking a target device, the target device and the AR glasses are pre-paired. After pairing, the target device and the AR glasses store a shared encryption key. The secure unlocking method is executed by the AR glasses and includes: Upon successful connection with the target device and receipt of an unlock request, the system acquires a first eye image of the wearer. The identity information of the wearer is determined based on the first eye image, and the identity information of the wearer is verified for the first time. If the first verification passes, the first eye image is encrypted based on the shared encryption key to obtain an encrypted eye image. The encrypted eye image and unlock command are then sent to the target device so that the target device can perform an unlock operation after receiving the encrypted eye image. The unlocking operation includes: decrypting the encrypted eye image based on the shared encryption key to obtain the first eye image; and acquiring the face image of the wearer and determining the occlusion rate of the AR glasses on the second eye image in the face image. The adjustment method for the second eye image is determined based on the occlusion rate; The second eye image is adjusted based on the adjustment method and the first eye image to obtain an adjusted facial image; the target occlusion rate of the eye image in the adjusted facial image is less than the occlusion rate. The identity information of the wearer is verified a second time based on the adjusted facial image; if the second verification is successful, the unlocking command is executed.

2. The secure unlocking method based on AR glasses as described in claim 1, characterized in that, The method for determining the adjustment of the second eye image based on the occlusion rate includes: In response to the occlusion rate being lower than a preset occlusion rate threshold, the adjustment method of the second eye image is determined as the first adjustment method; In response to the occlusion rate being not lower than the occlusion rate threshold, the adjustment method of the second eye image is determined as the second adjustment method; The first adjustment method is to adjust the image of the occluded area in the second eye image, and the second adjustment method is to adjust the image of the second eye image as a whole.

3. The secure unlocking method based on AR glasses as described in claim 2, characterized in that, If the adjustment method is the first adjustment method, then adjusting the second eye image based on the adjustment method and the first eye image to obtain the adjusted facial image includes: The first eye image is transformed by a perspective transformation to obtain a third eye image, so that the perspective of the third eye image is consistent with the perspective of the second eye image; Identify the extent of the occluded region in the second eye image; The target occlusion image is obtained by determining the image at the position corresponding to the range of the occluded area in the third eye image; The adjusted facial image is obtained by replacing the occluded area in the second eye image with the target occluded image.

4. The secure unlocking method based on AR glasses as described in claim 3, characterized in that, The step of performing a perspective transformation on the first eye image to obtain a third eye image includes: Identify feature point information in the first eye image and the second eye image; Based on the feature point information, feature point alignment is performed to obtain the viewing angle deviation parameters of the first eye image and the second eye image; Based on the aforementioned viewpoint deviation parameter, a geometric transformation is performed on the first eye image to obtain a third eye image.

5. The secure unlocking method based on AR glasses as described in claim 4, characterized in that, The step of aligning feature points based on the feature point information to obtain the viewing angle deviation parameters between the first eye image and the second eye image includes: From the feature point information, determine the first feature point set corresponding to the first eye image and the second feature point set corresponding to the second eye image; each feature point type in the first feature point set corresponds one-to-one with each feature point type in the second feature point set; Select any feature point from the first feature point set as a first reference point to establish a first coordinate system; and select a feature point from the second feature points that corresponds to the type of the first reference point as a second reference point to establish a second coordinate system. Calculate the first coordinate vector of each feature point in the first feature point set relative to the first reference point in the first coordinate system, and calculate the second coordinate vector of each feature point in the second feature point set relative to the second reference point in the second coordinate system; Feature point alignment is performed based on the first coordinate vector and the second coordinate vector to obtain the viewing angle deviation parameters of the first eye image and the second eye image.

6. The secure unlocking method based on AR glasses as described in claim 5, characterized in that, The step of aligning feature points based on the first coordinate vector and the second coordinate vector to obtain the viewing angle deviation parameters between the first eye image and the second eye image includes: For each feature point in the first feature point set, the difference between the first coordinate vector corresponding to the feature point and the second feature vector corresponding to the target feature point is calculated to obtain the target vector deviation value corresponding to the feature point; the target feature point is the feature point in the second coordinate vector that corresponds to the feature point. The viewing angle deviation parameters of the first eye image and the second eye image are determined based on the target vector deviation values ​​corresponding to each feature point in the first feature point set.

7. The secure unlocking method based on AR glasses as described in claim 2, characterized in that, If the adjustment method is the second adjustment method, then adjusting the second eye image based on the adjustment method and the first eye image to obtain the adjusted facial image includes: The first eye image is transformed by a perspective transformation to obtain a fourth eye image, so that the perspective of the fourth eye image is consistent with the perspective of the second eye image. The adjusted facial image is obtained by replacing the second eye image with the fourth eye image.

8. A secure unlocking system based on AR glasses, characterized in that, The system is installed in AR glasses and used to unlock a target device. The target device and the AR glasses are pre-paired. After pairing, the target device and the AR glasses store a shared encryption key. The secure unlocking system includes: The image acquisition module is used to acquire the first eye image of the wearer in response to a successful connection with the target device and the receipt of an unlocking request; An identity verification module is used to determine the identity information of the wearer based on the first eye image and to perform the first verification of the wearer's identity information; The security unlocking module is used to encrypt the first eye image based on the shared encryption key when the first verification is successful, to obtain an encrypted eye image, and send the encrypted eye image and unlocking command to the target device, so that the target device performs an unlocking operation after receiving the encrypted eye image; The unlocking operation includes: decrypting the encrypted eye image based on the shared encryption key to obtain the first eye image; and acquiring the face image of the wearer and determining the occlusion rate of the AR glasses on the second eye image in the face image. The adjustment method for the second eye image is determined based on the occlusion rate; The second eye image is adjusted based on the adjustment method and the first eye image to obtain an adjusted facial image; the target occlusion rate of the eye image in the adjusted facial image is less than the occlusion rate. The identity information of the wearer is verified a second time based on the adjusted facial image; if the second verification is successful, the unlocking command is executed.

9. An AR glasses device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images