Image processing method and apparatus for verification, and device and storage medium

Abstract

An image processing method and apparatus for verification, and a device and a storage medium, which relate to the technical field of image processing. The method comprises: displaying instruction information by means of a head-mounted display device (210), wherein the instruction information is used for instructing a user to observe a specified position in a display space of the head-mounted display device; acquiring an eye image of the user (220); acquiring from the eye image the positions of at least two key points of the eyes of the user (230); and when a position observed by the user and indicated by the relationship between the positions of the at least two key points of the eyes of the user matches the specified position, determining that the user passes verification (240). The solution can improve verification efficiency while ensuring verification accuracy.

Description

Image processing method, device, equipment and storage medium for verification

[0001] This application claims priority to the Chinese patent application filed on July 28, 2023, with application number 202310946844.4 and invention name “Image processing method, device, equipment and storage medium for verification”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of image processing technology, and in particular to an image processing method, apparatus, device, and storage medium for verification. Background Art

[0003] With the continuous development of network technology, network / data security has received more and more attention. When users use electronic products or network applications, the scenarios for user verification are becoming more and more diverse.

[0004] In related technologies, user verification is usually carried out using methods such as facial video verification and password verification. For example, taking the liveness verification scenario in facial verification as an example, first, the user needs to adjust the shooting angle and distance of the front camera of the mobile phone so that the front camera can clearly capture the face. Then, follow the prompts to blink, shake the head, etc. The verification program detects whether the current user has passed the liveness verification through the picture captured by the front camera of the mobile phone.

[0005] However, the above verification methods are usually complicated to operate, and the verification process for users usually takes a certain amount of time, resulting in low verification efficiency.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide an image processing method, apparatus, device, and storage medium for verification, which can improve verification efficiency. The technical solution is as follows:

[0008] In one aspect, an image processing method for verification is provided, the method being executed by a computer device, the method comprising:

[0009] Displaying instruction information through a head-mounted display device, wherein the instruction information is used to instruct a user to observe a specified position in a display space of the head-mounted display device;

[0010] Acquiring an eye image of the user;

[0011] Acquire positions of at least two key points of the user's eyes from the eye image;

[0012] When the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matches the designated position, it is determined that the user has passed the verification.

[0013] In another aspect, a head-mounted display device is provided, comprising a display screen and an image acquisition component; the image acquisition component is directed to an eyeball position when the head-mounted display device is worn;

[0014] The display screen is used to display instruction information, where the instruction information is used to instruct the user to observe a specified position in the display space of the head-mounted display device;

[0015] The image acquisition component is used to acquire the eye image of the user;

[0016] The eye image is used to obtain the positions of at least two key points of the user's eyes, so as to determine that the user has passed the verification when the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matches the specified position.

[0017] In another aspect, an image processing apparatus for verification is provided, the apparatus comprising:

[0018] A display module, configured to display instruction information through a head-mounted display device, wherein the instruction information is used to instruct a user to observe a specified position in a display space of the head-mounted display device;

[0019] An image acquisition module, configured to acquire an eye image of the user;

[0020] A key point acquisition module, configured to acquire positions of at least two key points of the user's eyes from the eye image;

[0021] The verification module is used to determine that the user has passed the verification when the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matches the specified position.

[0022] In some embodiments, the head-mounted display device is a virtual reality (VR) display device, or the head-mounted display device is an augmented display (AR) display device.

[0023] In some embodiments, the display module is used to display the indication information at the designated location through the VR display device or the AR display device.

[0024] In some embodiments, the image acquisition module is used to acquire the eye image captured by an image acquisition component provided in the head-mounted display device; the image acquisition component points to the position of the eyeball when the head-mounted display device is worn.

[0025] In some embodiments, the at least two key points of the user's eyes include at least one of the following:

[0026] at least two key points corresponding to the user's left eye;

[0027] And, at least two key points corresponding to the user's right eye.

[0028] In some embodiments, the at least two key points corresponding to the user's left eye include a left corner key point of the user's left eye, a right corner key point of the user's left eye, and a pupil center point of the user's left eye;

[0029] At least two key points corresponding to the user's right eye, including a left corner key point of the user's right eye, a right corner key point of the user's right eye, and a pupil center point of the user's right eye;

[0030] The verification module is used to:

[0031] Obtain a first distance, a second distance, a third distance, and a fourth distance; wherein the first distance is the distance between the left corner key point of the left eye and the center point of the pupil of the left eye; the second distance is the distance between the right corner key point of the left eye and the center point of the pupil of the left eye; the third distance is the distance between the left corner key point of the right eye and the center point of the pupil of the right eye; and the fourth distance is the distance between the right corner key point of the right eye and the center point of the pupil of the right eye;

[0032] If the position observed by the user indicated by the relationship among the first distance, the second distance, the third distance, and the fourth distance matches the designated position, it is determined that the user has passed the authentication.

[0033] In some embodiments, the verification module is used to:

[0034] determining a viewing direction of the user based on a magnitude relationship among the first distance, the second distance, the third distance, and the fourth distance;

[0035] When the viewing direction of the user matches the designated position, it is determined that the user has passed the verification.

[0036] In some embodiments, the verification module is used to:

[0037] When the second distance is greater than the first distance and the fourth distance is greater than the third distance, determining that the viewing direction of the user is leftward;

[0038] When the first distance is greater than the second distance, and the third distance is greater than the fourth distance, it is determined that the viewing direction of the user is rightward.

[0039] In some embodiments, the designated location moves over time; the verification module is configured to:

[0040] Obtaining a change trend of a first ratio between the first distance and the second distance;

[0041] obtaining a change trend of a second ratio between the third distance and the fourth distance;

[0042] In a case where the changing trend of the first ratio and the changing trend of the second ratio match the moving direction of the designated position, it is determined that the user has passed the authentication.

[0043] In some embodiments, the key point acquisition module is configured to input the eye image into a key point detection model to obtain positions of at least two key points of the user's eyes output by the key point detection model;

[0044] The key point detection model is a machine learning model that is pre-trained using eye image samples and the positions of at least two key points marked in the eye image samples.

[0045] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the above-mentioned image processing method for verification.

[0046] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the above-mentioned image processing method for verification.

[0047] In another aspect, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the image processing method for verification provided in the various optional implementations described above.

[0048] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0049] The information displayed by the head-mounted display device instructs the user to observe a specified position in the display space of the head-mounted display device, and at the same time obtains the user's eye image, obtains the relationship between the positions of at least two key points of the user's eyes based on the user image, and determines that the position observed by the user matches the specified position, then determines that the user has passed the verification. Since the head-mounted display device will follow the rotation of the user's head, the relationship between the positions of the key points in the user's eye image can accurately represent the user's current observation position, and then determine whether the user is observing the specified position, so as to achieve accurate verification of the user. At the same time, in the above process, the user only needs to rotate his eyeballs and observe the specified position, and the user does not need to perform other operations, which can greatly simplify the user operations in the verification process, thereby improving verification efficiency while ensuring verification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] FIG1 is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0051] FIG2 is a flowchart of an image processing method for verification provided by one embodiment of the present application;

[0052] FIG3 is a flowchart of an image processing method for verification provided by one embodiment of the present application;

[0053] FIG4 is a flowchart of an image processing method for verification provided by one embodiment of the present application;

[0054] FIG5 is a flowchart of an image processing method for verification provided by one embodiment of the present application;

[0055] FIG6 is a flowchart of an image processing method for verification provided by one embodiment of the present application;

[0056] FIG7 is a schematic diagram of a scene interface involved in this application;

[0057] FIG8 is a schematic diagram of a DeepPose algorithm involved in this application;

[0058] FIG9 is a schematic diagram of another DeepPose algorithm involved in this application;

[0059] FIG10 is a schematic diagram of predicting eye key points using the DeepPose algorithm involved in this application;

[0060] FIG11 is a diagram showing images of both eyes when gazing leftward and rightward according to the present application;

[0061] FIG12 is a diagram showing the effect of key point detection involved in this application;

[0062] FIG13 is a schematic diagram of the connection line between the eye corner and the pupil involved in this application;

[0063] FIG14 is a schematic diagram of the verification process involved in this application;

[0064] FIG15 is a block diagram of an image processing apparatus for verification provided by one embodiment of the present application;

[0065] FIG16 is a structural block diagram of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0067] 1) Infrared image: An infrared image captured by an infrared sensor using infrared light imaging.

[0068] Since the iris of the eye has a relatively special absorption characteristic for infrared, and at the same time avoids the interference of VR color imaging in the virtual reality (VR) scenario, using infrared images to collect eye information can have better anti-interference and recognition effects.

[0069] 2) Liveness detection: A method for determining a subject's true physiological characteristics in identity verification scenarios. For example, in facial recognition applications, liveness detection can verify that the user is truly the user by using key point location and detection techniques based on combinations of actions such as blinking, opening the mouth, shaking the head, and nodding. This method effectively protects against common attacks such as photos, videos, face swaps, masks, occlusions, 3D animations, and screen captures, helping users identify illegal activity and protecting their interests.

[0070] 3) Identity verification: Also known as "verification" or "authorization," this refers to the technology used to confirm a user's identity through certain means. There are many methods for identity verification, including shared key authentication, public key encryption, and reserved information authentication.

[0071] FIG1 is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application. The implementation environment may include: a head-mounted display device 110; optionally, the implementation environment may also include a first terminal 120; optionally, the implementation environment may also include a server 130.

[0072] Among them, the servers involved above can be independent physical servers, or they can be server clusters or distributed systems composed of multiple physical servers, or they can be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to these. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0073] In some embodiments, the head-mounted display device 110 may display images on its own display screen; alternatively, the head-mounted display device 110 may exchange data with the server 130 to display images on the display screen of the head-mounted display device 110. For example, an application may be installed in the head-mounted display device 110, and the application may be a VR application (such as a VR game) or an AR application (such as an AR game).

[0074] FIG1 shows only one head-mounted display device, but in different embodiments, there may be multiple other head-mounted display devices accessing the server 130 or the first terminal 120 .

[0075] Specifically, the first terminal 120 may install and run an application 121 and a streaming application 122. The application 121 may independently display images on the display screen of the head-mounted display device 110, or may interact with the server 130 to display images on the display screen of the head-mounted display device 110. For example, the application 121 may be a VR application (such as a VR game) or an AR application (such as an AR game).

[0076] FIG1 shows only one terminal, but in different embodiments, multiple other terminals may be connected to the head-mounted display device 110 or the server 130. Optionally, one or more terminals may be developer terminals, on which a development platform for developing the aforementioned application 121 or streaming application 122 is installed.

[0077] The head-mounted display device 110 and other head-mounted display devices can be connected to the first terminal 120 or the server 130 via a wireless network or a wired network.

[0078] The head-mounted display device 110 may include a display and an input component, wherein the display may be used to display an encoded video stream of a virtual scene / augmented display scene, and the input component is used to receive user input data and return the input data to the first terminal 120 or the server 130; wherein the above-mentioned input component may include at least one of a sensor (such as a motion sensor, a touch sensor, a light sensor, etc.) and a physical button, and the above-mentioned input data may include sensor data and physical button pressing data, etc.

[0079] Server 130 includes at least one of a single server, a server cluster consisting of multiple servers, a cloud computing platform, and a virtualization center. Server 130 can provide background support for applications 121 running in the terminal or for applications in the head-mounted display device 110. Optionally, server 130 performs primary computing tasks, while the terminal or head-mounted display device 110 performs secondary computing tasks; alternatively, server 130 performs secondary computing tasks, while the terminal or head-mounted display device 110 performs primary computing tasks; alternatively, server 130 and the terminal or head-mounted display device 110 utilize a distributed computing architecture for collaborative computing.

[0080] In an illustrative example, the server 130 includes a memory 131, a processor 132, a user account database 133, an application service module 134, and a user-facing input / output interface (I / O interface) 135. The processor 132 is used to load instructions stored in the server 130 and process data in the user account database 133 and the application service module 134. The user account database 133 is used to store user account data used by the first terminal 110 and other terminals, such as user account avatars, user account names, and user account levels. The application service module 134 is used to provide background support services for applications, etc. The user-facing I / O interface 135 is used to establish communication and exchange data with the head-mounted display device 110 or the first terminal 120 via a wireless network or a wired network.

[0081] Please refer to Figure 2, which shows a flowchart of an image processing method for verification provided by an embodiment of the present application. For the sake of convenience, only the computer device that performs each step is introduced and described. The above-mentioned computer device can be a head-mounted display device, a terminal or a server; the above-mentioned head-mounted display device can be the head-mounted display device 110 shown in Figure 1 above, the above-mentioned terminal can be the terminal 120 shown in Figure 1 above, and the above-mentioned server can be the server 130 shown in Figure 1 above; the method can include the following steps.

[0082] Step 210: Display instruction information through the head-mounted display device, where the instruction information is used to instruct the user to observe a specified position in the display space of the head-mounted display device.

[0083] The display space of the head-mounted display device refers to the space in which the information displayed by the head-mounted display device can exist.

[0084] Optionally, the display space of the head-mounted display device may be a three-dimensional space, and the head-mounted display device may display the designated information element at any position in the three-dimensional space.

[0085] Alternatively, the display space of the head-mounted display device may also be a two-dimensional plane space, such as a display screen, and the head-mounted display device may display the specified information element at any position in the two-dimensional plane space.

[0086] Among them, the head-mounted display device may include a display screen. Optionally, the head-mounted display device may include a single display screen or two display screens. When it includes two display screens, the two display screens correspond to the user's two eyes respectively. The two display screens can respectively display the images when the two eyes observe the image elements / objects in the virtual space, thereby creating a three-dimensional display effect for the user.

[0087] For example, the head-mounted display device is a virtual reality (VR) display device, or the head-mounted display device is an augmented reality (AR) display device.

[0088] Virtual reality, as the name suggests, is the fusion of virtual and real life. Theoretically, virtual reality (VR) technology is a computer simulation system that allows users to create and experience a virtual world. It uses computers to generate a simulated environment, immersing the user in it. VR technology utilizes real-life data, electronic signals generated by computer technology, and combines them with various output devices to transform them into perceptible phenomena. These phenomena can be real objects or invisible materials, represented through three-dimensional models. Because these phenomena are not directly visible to us, but rather simulated through computer technology, they are called virtual reality.

[0089] In the embodiments of the present application, the VR display device can display a three-dimensional virtual scene to the user. In other words, the three-dimensional scene image that the user sees through the VR display device is a simulated world image simulated by a computer, such as a VR game image, etc. Of course, the VR display device can also be used to play three-dimensional videos (3D movies).

[0090] Augmented Reality (AR) technology is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, real-time detection and registration, intelligent interaction, and sensing. It simulates computer-generated virtual information such as text, images, three-dimensional models, music, and videos, and applies them to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0091] Augmented reality, also known as augmented reality, is a relatively new technology that integrates real-world and virtual world information. It uses computer technology to simulate physical information, which is difficult to experience in the real world, and effectively applies virtual information to the real world. This process can be perceived by human senses, creating a sensory experience beyond reality. The real environment and virtual objects overlap, and can exist simultaneously in the same screen and space.

[0092] Augmented reality technology not only effectively displays real-world content but also enables the display of virtual information, with these detailed contents complementing and overlapping each other. In visual augmented reality, users need to use head-mounted display devices to overlap the real world with computer graphics, allowing them to fully see the real world surrounding them. Augmented reality technology primarily includes new technologies and methods such as multimedia, 3D modeling, and scene fusion. The information provided by augmented reality is significantly different from the information that humans can perceive.

[0093] AR display devices can be implemented as single-lens or double-lens AR glasses. In addition to allowing the user to see the real world behind the lenses through light, the lenses of the AR glasses can also reflect virtual images to the user's eye position, allowing the user to simultaneously see virtual image elements / objects superimposed on the real world.

[0094] Whether it is an AR display device or a VR display device, the specified information can be displayed at a specified location.

[0095] In some embodiments, the computer device may display indication information at a specified location through a VR display device or an AR display device.

[0096] In an embodiment of the present application, in order to better prompt the user to observe the specified location, the computer device can directly display the instruction information at the specified location through a VR display device or an AR display device.

[0097] Step 220: Acquire an eye image of the user.

[0098] The above-mentioned eye image of the user may be an eye image of the user when the user wears the above-mentioned head-mounted display device.

[0099] Specifically, when a user wears the above-mentioned head-mounted display device, the head-mounted display device can display specified information so that the user's eyes can focus on a specified position.

[0100] In some embodiments, a computer device may obtain an eye image captured by an image capture component provided in a head-mounted display device; the image capture component is directed to the eye position when the head-mounted display device is worn. For example, the computer device may obtain an eye image captured by an image capture component provided in an AR display device or a VR display device. For example, when the AR display device or the VR display device displays indication information at a specified location, the computer device may obtain an eye image captured by the image capture component provided in the AR display device or the VR display device.

[0101] Because the user's eyes are usually blocked by the head-mounted display device when the user wears it, if the user's eye image is captured from outside the head-mounted display device, it may not be possible to correctly capture a clear image of the user's eyes. To this end, in an embodiment of the present application, the head-mounted display device may be provided with an image capture component, and the image capture component may be pointed at the position of the user's eyeballs when the head-mounted display device is worn. In this way, the image capture component can accurately capture a clear image of the user's eyes, ensuring the accuracy of subsequent verification.

[0102] Optionally, the above-mentioned eye image may be an infrared image.

[0103] Step 230: Obtain positions of at least two key points of the user's eyes from the eye image.

[0104] The human eye area may contain multiple types of key points, and each type of key point may contain multiple key points.

[0105] For example, the eye part of a human body may include one or more key points of an eyelid, one or more key points of an iris, one or more key points of a pupil, and so on.

[0106] Step 240: If the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matches the designated position, it is determined that the user has passed the verification.

[0107] When the user's eyes focus on spatial positions in different directions and depths, the positions of the key points of the human eye will also be different. Conversely, the positions of the key points of the human eye can be used to infer the approximate position (including direction, distance, etc.) where the user's eyes are currently focused / observed.

[0108] Based on the above principles, in an embodiment of the present application, a specified information is displayed by a head-mounted display device to instruct the user to observe a specified position in the display space of the head-mounted display device. Since the head-mounted display device will rotate with the user's head when it is worn on the user's head, that is, no matter how the user turns his head, the position of the information displayed by the head-mounted display device is fixed relative to the user's eyes. For different positions in the display space of the head-mounted display device, the user can only observe by changing the direction and focal length of the eyeballs. In this way, the interference caused by the rotation of the head when the user observes the specified position in the display space of the head-mounted display device can be eliminated, and the position observed by the user can be accurately identified through the relationship between the positions of the key points of the eyes. Then, whether the user has passed the verification is determined by whether the position observed by the user matches the specified position.

[0109] To sum up, the scheme shown in the embodiment of the present application instructs the user to observe the specified position in the display space of the head-mounted display device through the information displayed by the head-mounted display device, and at the same time obtains the user's eye image, obtains the relationship between the positions of at least two key points of the user's eyes based on the user image, and determines that the position domain observed by the user matches the specified position, and determines that the user has passed the verification. Since the head-mounted display device will follow the rotation of the user's head, the relationship between the positions of the key points in the user's eye image can accurately represent the user's current observation position, and then determine whether the user is observing the specified position, so as to achieve accurate verification of the user. At the same time, in the above process, the user only needs to rotate his eyeballs and observe the specified position, and the user does not need to perform other operations, which can greatly simplify the user operations in the verification process, thereby improving verification efficiency while ensuring verification accuracy.

[0110] Among them, the schemes shown in the various embodiments involved in Figure 2 above of the present application can be applied to the user's liveness detection scenario. For example, through the scheme shown in Figure 2 above, when the position of the user's observation indicated by the relationship between the positions of at least two key points of the user's eyes is detected and matches the specified position, it can be considered that the user is currently performing real-life operations.

[0111] Alternatively, the schemes shown in the various embodiments involved in Figure 2 above can also be applied to user identity authentication scenarios. For example, through the scheme shown in Figure 2 above, different verification information can be displayed at different positions in the display space of the head-mounted display device through the head-mounted display device, where the authentication information at the specified position is the correct authentication information. When the position of the user observed indicated by the relationship between the positions of at least two key points of the user's eyes is detected and matches the specified position, it can be considered that the authentication information currently observed by the user is correct, and the user can be confirmed to have passed the identity authentication.

[0112] Based on the solutions shown in the various embodiments involved in Figure 2 above, in some embodiments, the at least two key points of the user's eyes include at least one of the following: at least two key points corresponding to the user's left eye; and at least two key points corresponding to the user's right eye.

[0113] In a possible implementation of an embodiment of the present application, a computer device can roughly determine the direction and / or depth of the user's current observation through at least two key points of a single eye, either the left eye or the right eye, and then match the user's observation position indicated by the relationship between the positions of at least two key points of the user's eyes with the specified position to determine that the user has passed the verification; this solution only needs to obtain at least two key points of a single eye, which can simplify the workload of obtaining and analyzing the key point positions of the eye image, improve verification efficiency, and save hardware resources; in addition, when eye images of both eyes are collected separately and the quality of one of the eye images is poor, verification can also be performed through the eye image of the other eye, thereby ensuring the stability of the verification process.

[0114] In another possible implementation of the embodiment of the present application, the computer device determines the direction and / or depth of the user's current observation by combining at least two key points of the left eye and the right eye, thereby ensuring the accuracy of the user's observation position and thus ensuring the accuracy of the verification.

[0115] Based on the scheme involved in the embodiment shown in Figure 2 above, the process of the computer device performing verification by combining at least two key points of each of the left eye and the right eye can refer to Figure 3. Among them, Figure 3 shows a flowchart of the image processing method for verification provided by an embodiment of the present application. In the case where the at least two key points of the user's eyes include at least two key points corresponding to the user's left eye, and at least one of the at least two key points corresponding to the user's right eye, and the at least two key points corresponding to the user's left eye include the left corner key point of the user's left eye, the right corner key point of the user's left eye, and the center point of the pupil of the user's left eye, and the at least two key points corresponding to the user's right eye include the left corner key point of the user's right eye, the right corner key point of the user's right eye, and the center point of the pupil of the user's right eye, on the basis of the embodiment shown in Figure 2, in the embodiment shown in Figure 3, the above-mentioned step 240 may include steps 240a and 240b:

[0116] Step 240a: Obtain the first distance, the second distance, the third distance, and the fourth distance.

[0117] Among them, the first distance is the distance between the left corner key point of the left eye and the center point of the pupil of the left eye; the second distance is the distance between the right corner key point of the left eye and the center point of the pupil of the left eye; the third distance is the distance between the left corner key point of the right eye and the center point of the pupil of the right eye; the fourth distance is the distance between the right corner key point of the right eye and the center point of the pupil of the right eye.

[0118] In an embodiment of the present application, the at least two key points corresponding to the user's left eye include at least the left corner key point of the user's left eye, the right corner key point of the user's left eye, and the center point of the user's left eye pupil; the at least two key points corresponding to the user's right eye include at least the left corner key point of the user's right eye, the right corner key point of the user's right eye, and the center point of the user's right eye pupil.

[0119] Step 240b: When the position observed by the user indicated by the relationship among the first distance, the second distance, the third distance, and the fourth distance matches the designated position, it is determined that the user has passed the verification.

[0120] In some embodiments, when the position observed by the user indicated by the relationship between the first distance, the second distance, the third distance, and the fourth distance matches the specified position, the step of determining that the user has passed the verification may include:

[0121] Determining a viewing direction of the user based on a magnitude relationship among the first distance, the second distance, the third distance, and the fourth distance;

[0122] When the viewing direction of the user matches the specified position, it is determined that the user has passed the authentication.

[0123] Among them, the above-mentioned determination of the user's viewing direction based on the size relationship between the first distance, the second distance, the third distance and the fourth distance may refer to determining the user's viewing direction through the size relationship between the first distance, the second distance, the third distance and the fourth distance, and the correspondence between the pre-set relationship types of the above-mentioned size relationships (such as greater than, less than, equal to, etc.) and the viewing direction.

[0124] In some embodiments, determining the viewing direction of the user based on the magnitude relationship among the first distance, the second distance, the third distance, and the fourth distance includes:

[0125] When the second distance is greater than the first distance and the fourth distance is greater than the third distance, determining that the viewing direction of the user is leftward;

[0126] When the first distance is greater than the second distance, and the third distance is greater than the fourth distance, it is determined that the viewing direction of the user is rightward.

[0127] It should be noted that the above-mentioned viewing direction refers to the viewing direction of the user; that is, the above-mentioned viewing direction is right, which means that the user's viewing direction points to the right side of the user, and correspondingly, the above-mentioned viewing direction is left, which means that the user's viewing direction points to the left side of the user.

[0128] In an embodiment of the present application, for one image each for the left and right eyes, the computer device can respectively determine the distance between the two corners of the left and right eyes and the pupils. By using the size relationship between the two corners of the left and right eyes and the pupils, it can be determined whether the user's eyes are looking left or right. Combined with the position of the designated position relative to the user's eyes, it can be determined whether the user is observing the designated position, and then determine whether the user has passed the verification. This process only needs to determine whether the user's eyes are looking left or right. The judgment accuracy is high, and the judgment logic is simple, which can ensure the accuracy and efficiency of the verification.

[0129] Among them, the above embodiment is explained by combining at least two key points of the left eye and the right eye for verification as an example. Optionally, verification can also be performed through at least two key points of the left eye or the right eye. For example, still taking the above-mentioned first distance, second distance, third distance and fourth distance as an example, if the second distance is greater than the first distance, it can be determined that the user's viewing direction is to the left. Conversely, if the first distance is greater than the second distance, the user's viewing direction is determined to be to the right; for another example, if the fourth distance is greater than the third distance, the user's viewing direction is determined to be to the left. If the third distance is greater than the fourth distance, the user's viewing direction is determined to be to the right.

[0130] In the above embodiment, the computer device can flexibly choose to perform verification through at least two key points of each of the two eyes, or through at least two key points of one of the eyes. For example, after the computer device captures the eye images of the two eyes, it can first detect the image quality of the eye images of the two eyes, and extract key points of the eye images of the two eyes that meet the quality requirements to obtain at least two key points of the corresponding eyes. Then, based on the at least two extracted key points, it is determined whether the user's viewing direction is left or right, and then it is determined whether the user has passed the verification.

[0131] The above-mentioned quality requirements may include at least one of the following:

[0132] The clarity of the eye image is above the clarity threshold;

[0133] The size of the eyeball region visible in the eye image is greater than the size threshold.

[0134] That is to say, in an embodiment of the present application, the computer device can select an eye image with sufficiently high clarity and a sufficiently large eyeball area from the eye images of the left and right eyes, and determine whether the user's viewing direction is to the left or right based on the key points of the selected eye image, and then determine whether the user has passed the verification, thereby improving the accuracy of the verification.

[0135] In some embodiments, the designated location moves over time; and when a location observed by the user indicated by a relationship among the first distance, the second distance, the third distance, and the fourth distance matches the designated location, determining that the user has passed the verification includes:

[0136] Obtaining a change trend of a first ratio between the first distance and the second distance;

[0137] obtaining a change trend of a second ratio between the third distance and the fourth distance;

[0138] In a case where the changing trend of the first ratio and the changing trend of the second ratio match the moving direction of the designated position, it is determined that the user has passed the authentication.

[0139] In the above embodiment, the computer device can control the specified position to move in the specified direction within a specified time period. At the same time, within the specified time period, multiple eye images of the left eye and the right eye are collected. For the multiple eye images of the left eye, a first ratio of each of the multiple eye images is determined through at least two key points of each of the multiple eye images, and then a change trend of the first ratio is determined according to the order of the collection time points of the multiple eye images from front to back, that is, whether the first ratio gradually increases or gradually decreases; correspondingly, for the multiple eye images of the right eye, a second ratio of each of the multiple eye images is determined through at least two key points of each of the multiple eye images, and then a change trend of the second ratio is determined according to the order of the collection time points of the multiple eye images from front to back, that is, whether the second ratio gradually increases or gradually decreases.

[0140] For example, if the first ratio between the first distance and the second distance gradually decreases, and the second ratio between the third distance and the fourth distance also gradually decreases, it is determined that the moving direction of the specified position is moving to the left; for another example, if the first ratio between the first distance and the second distance gradually increases, and the second ratio between the third distance and the fourth distance also gradually increases, it is determined that the moving direction of the specified position is moving to the right.

[0141] Among them, the above embodiment is explained by combining at least two key points of the left eye and the right eye for verification as an example. Optionally, verification can also be performed through at least two key points of the left eye or the right eye. For example, still taking the above-mentioned first distance, second distance, third distance and fourth distance as an example, if the first ratio between the first distance and the second distance gradually decreases, it can be determined that the moving direction of the specified position is moving to the left. Conversely, if the first ratio gradually increases, it can be determined that the moving direction of the specified position is moving to the right; for another example, if the second ratio between the third distance and the fourth distance gradually decreases, it can be determined that the moving direction of the specified position is moving to the left. Conversely, if the second ratio gradually increases, it can be determined that the moving direction of the specified position is moving to the right.

[0142] In an embodiment of the present application, the computer device can also continuously obtain images of the left and right eyes of the user, and determine the distance between the two corners of the left and right eyes and the pupils respectively, calculate the changing trend of the ratio of the distance between the two corners of the left and right eyes and the pupils in this process, so as to determine the moving direction of the position observed by the user, and combine the moving direction of the user's observation position with the moving direction of the designated position indicated by the designated information to determine whether the user has passed the verification.

[0143] Among them, the scheme in which the above-mentioned computer device determines that the user has passed the verification when the position observed by the user indicated by the relationship between the first distance, the second distance, the third distance and the fourth distance matches the specified position, and the scheme in which the above-mentioned computer device determines that the user has passed the verification when the changing trend of the first ratio and the changing trend of the second ratio match the moving direction of the specified position can be used separately.

[0144] Alternatively, the scheme in which the computer device determines that the user has passed the verification when the position observed by the user indicated by the relationship between the first distance, the second distance, the third distance and the fourth distance matches the specified position, and the scheme in which the computer device determines that the user has passed the verification when the changing trend of the first ratio and the changing trend of the second ratio match the moving direction of the specified position may also be used in combination. For example, the computer device determines that the user has passed the verification when the position observed by the user indicated by the relationship between the first distance, the second distance, the third distance and the fourth distance matches the specified position, and when the changing trend of the first ratio and the changing trend of the second ratio match the moving direction of the specified position.

[0145] Based on the solution involved in the embodiment shown in FIG2 above, the process of the computer device performing verification by combining at least two key points of each of the left eye and the right eye can be referred to FIG4. Among them, FIG4 shows a flowchart of the image processing method for verification provided by an embodiment of the present application. In the case where the at least two key points corresponding to the user's left eye include multiple key points at the edge of the pupil of the user's left eye, and the at least two key points corresponding to the user's right eye include multiple key points at the edge of the pupil of the user's right eye, based on the embodiment shown in FIG2, in the embodiment shown in FIG4, the above-mentioned step 240 may include steps 240c and 204d:

[0146] Step 240c: Obtain the changing trend of the first area and the changing trend of the second area; the first area is the area of ​​the area surrounded by multiple key points on the edge of the pupil of the left eye, and the second area is the area of ​​the area surrounded by multiple key points on the edge of the pupil of the left eye.

[0147] In the above embodiment, the computer device can control the depth of the specified position to gradually increase or decrease within a specified time period, wherein the depth of the above-mentioned specified position may refer to the distance of the specified position relative to the human eye. The closer the distance is, the smaller the depth is, and conversely, the farther the distance is, the greater the depth is.

[0148] At the same time, within the specified time period, multiple eye images of the left eye and the right eye are collected. For the multiple eye images of the left eye, the first area of ​​each of the multiple eye images is determined by multiple key points of the pupil edge of each of the multiple eye images, and then the change trend of the first area is determined according to the order of the collection time points of the multiple eye images from front to back, that is, whether the first area gradually increases or decreases; correspondingly, for the multiple eye images of the right eye, the second area of ​​each of the multiple eye images is determined by multiple key points of the pupil edge of the multiple eye images, and then the change trend of the second area is determined according to the order of the collection time points of the multiple eye images from front to back, that is, whether the second area gradually increases or decreases.

[0149] Step 240d: If the changing trend of the first area and the changing trend of the second area match the changing direction of the depth at the designated location, determine that the user has passed the verification.

[0150] Among them, when the human eye observes objects at different depths (or called far and near), the pupil area of ​​the human eye will change. Specifically, the pupil will shrink when the human eye observes nearby objects, and will enlarge when observing distant objects. Based on the above principle, in an embodiment of the present application, the computer device can control the depth of the specified position to change. At the same time, the pupil area in the collected user's eye image is used to judge the changing trend of the depth of the user's visual focus and whether it matches the changing direction of the depth of the specified position. If so, it is determined that the user has passed the verification.

[0151] Specifically, if the first area and the second area gradually shrink, it is determined that the depth of the user's visual focus is shrinking. At this time, if the depth of the designated position gradually decreases, the changing trend of the depth of the user's visual focus is determined to match the changing direction of the depth of the designated position, and it is determined that the user has passed the verification; correspondingly, if the first area and the second area gradually increase, it is determined that the depth of the user's visual focus is increasing. At this time, if the depth of the designated position gradually deepens, the changing trend of the depth of the user's visual focus is determined to match the changing direction of the depth of the designated position, and it is determined that the user has passed the verification.

[0152] In the solution shown in the embodiment of the present application, user verification can be achieved by changing the depth of the specified position in combination with the change of the user's pupil area, which expands the implementation method of user verification and improves the security of verification.

[0153] Among them, the above embodiment is illustrated by combining multiple key points of the pupil edge of the left eye and the right eye for verification as an example. Optionally, verification can also be performed through multiple key points of the pupil edge of the left eye or the right eye. For example, still taking the above-mentioned first area and second area as an example, if the first area / second area gradually decreases, it can be determined that the depth of the user's visual focus is shrinking. Conversely, if the first area / second area gradually increases, it can be determined that the depth of the user's visual focus is increasing.

[0154] Optionally, since the area of ​​the human pupil is related not only to the distance at which the user observes the object, but also to the intensity of light, in an embodiment of the present application, while controlling the depth change of a specified position, the computer device can also control the display brightness of the head-mounted display device to remain unchanged, thereby avoiding interference with the area detection of the user's pupil and improving the accuracy of verification.

[0155] Based on the solution involved in the embodiment shown in FIG2 above, the process of the computer device performing verification by combining at least two key points of the left eye and the right eye can refer to FIG5. Among them, FIG5 shows a flowchart of the image processing method for verification provided by an embodiment of the present application. In the case where the at least two key points corresponding to the user's left eye include the center point of the pupil of the user's left eye and the center point of the user's left eye, and the at least two key points corresponding to the user's right eye include the center point of the pupil of the user's right eye and the center point of the user's right eye, based on the embodiment shown in FIG2, in the embodiment shown in FIG5, the above-mentioned step 240 may include steps 240e and 204f:

[0156] Step 240e: Obtain a first direction and a second direction; the first direction is the direction of the pupil center of the left eye relative to the center of the left eye, and the second direction is the direction of the pupil center of the right eye relative to the center of the right eye.

[0157] The left eye center point may be the center of the area surrounded by the eyelids in the eye image of the left eye; correspondingly, the right eye center point may be the center of the area surrounded by the eyelids in the eye image of the right eye.

[0158] Step 240f: When the first direction and the second direction match the direction of the designated position relative to the center point of the display area of ​​the head mounted display device, determine that the user has passed the verification.

[0159] In this embodiment of the present application, for each image, the computer device can determine the orientation of the left and right pupils relative to the center of the eye, thereby determining the direction the user is looking. Combined with the orientation of the designated location relative to the center of the screen, this can determine whether the user is viewing the designated location and, therefore, whether the user has passed verification. This solution can combine the user's left-right and up-down viewing patterns to more accurately detect the user's current viewing direction and determine whether the user is looking at the designated location, thereby improving the accuracy and security of verification.

[0160] Among them, the above embodiment is explained by combining the direction of the pupil center of the left eye and the right eye relative to the center of the eye as an example. Optionally, verification can also be performed by the direction of the pupil center of the left eye or the right eye relative to the center of the eye. For example, still taking the above-mentioned first direction and second direction as an example, if the first direction / second direction matches the direction of the specified position relative to the center of the display area of ​​the head-mounted display device, it can be determined that the user has passed the verification.

[0161] Based on the solutions shown in one or more of the embodiments described in FIG. 2 and FIG. 3 , FIG. 6 shows a flow chart of an image processing method for verification provided by one embodiment of the present application. Based on the embodiment shown in FIG. 2 , in the embodiment shown in FIG. 6 , step 230 may include step 230a:

[0162] Step 230a: Input the eye image into the key point detection model to obtain the positions of at least two key points of the user's eyes output by the key point detection model.

[0163] Among them, the key point detection model is a machine learning model that is pre-trained through eye image samples and the positions of at least two key points marked by the eye image samples.

[0164] In an embodiment of the present application, a process of identifying at least two key points from an eye image can be implemented with the help of a machine learning model.

[0165] For example, the key point detection model may be a deep learning model that implements classification or regression, such as an image classification or regression model such as a DeepPose model.

[0166] For example, developers collect several eye image samples in advance, mark the positions of at least two key points in each eye image sample, then input the eye image samples into the DeepPose model to obtain the predicted positions of at least two key points in the eye image sample output by the DeepPose model, then calculate the loss using a pre-set loss function based on the positions of at least two key points marked in the eye image sample and the predicted positions of at least two key points in the eye image sample, and update the model parameters of the DeepPose model using the calculated loss, for example, using a gradient descent algorithm to update the model parameters of the DeepPose model using the calculated loss, repeating the above process until the DeepPose model is trained to convergence, at which point the DeepPose model can be used as the above-mentioned key point detection model. Later in the application process, after the computer device inputs the eye image into the key point detection model, the key point detection model can output the positions of at least two key points in the eye image.

[0167] In one embodiment, the solution shown in the above embodiment of the present application can be applied to realize liveness judgment in the recognition scenario. As the core link, it can effectively ensure the security of recognition and prevent it from being attacked by paper fragments or fake eyes, resulting in recognition attacks.

[0168] For example, this solution proposes a liveness determination algorithm based on attention judgment to identify behavioral intentions. By constructing different object scenes in the VR scene, guiding the eyes to look in different directions, and locating the position and shape of the eyes and irises, it is estimated whether the user's attention is in line with the object guidance direction, thereby determining whether the user is operating in person.

[0169] From the product side, after the user wears the VR device, the recognition camera will continuously collect photos of the user's eyes. By constructing different object scenes in the VR scene, it will guide the eyes to look in different directions. By locating the position of key points in the eyes and the shape or size of the area surrounded by the key points, it will estimate whether the user's attention is in line with the object guidance, thereby determining whether the user is operating in person.

[0170] From a technical perspective, the solution consists of the following three parts:

[0171] 1) VR scene construction: including the left and right positions of moving objects, etc.

[0172] In this solution, Unity can be used to build a VR 3D scene. For example, Figure 7 shows a schematic diagram of a scene interface involved in this application. As shown in Figure 7, in this scene, the panel is always directly in front of the user's eyes, and the position of the eye icon 71 changes. By changing the position of the eye icon 71, the user's gaze is guided, thereby guiding the user's eyes to rotate left and right.

[0173] 2) Locating the position and shape of the eyes and irises: Calculating key points of the pupil, iris, and eyelid regions based on DeepPose

[0174] Eyeball attention point estimation relies on the position of the eyes, pupils, and the relative position of pupils and eyes. This is to correctly estimate the eye's attention point scheme, assess whether the user's attention is consistent with the interactive guidance, and thus determine whether the subject is alive. Regarding the size and position of the iris, key points can be estimated through DeepPose. Based on the calculated key point positions, the precise size and position of the iris can be calculated. The following first explains the principle of the DeepPose algorithm:

[0175] The core idea of ​​the DeepPose algorithm is described as follows:

[0176] In the traditional field of posture recognition, the knowledge system of interactions between limbs is usually used to design more complex models such as image representation information (Feature Representations) or joint topology (Topology between joints).

[0177] The DeepPose algorithm, however, eliminates the need for human anatomy in complex poses and instead transforms keypoint detection into a purely mathematical prediction problem. By manually labeling a large amount of human keypoint data in various poses and using a Deep Neural Network (DNN) convolutional neural network to learn from this sample data, a more general end-to-end keypoint detection algorithm is achieved.

[0178] Please refer to Figure 8, which shows a schematic diagram of the DeepPose algorithm. As shown in Figure 8, after a series of convolutions are performed on the 220*220 image input, the final target position (xi, yi) is obtained through two fully connected layers. It should be noted that this algorithm is not limited to human body use, but is also applicable to the iris area. It is itself a general key point prediction algorithm.

[0179] The DeepPose algorithm itself can be further optimized. For example, please refer to Figure 9, which shows another DeepPose algorithm schematic. As shown in Figure 9, due to the uncertain input target size and the network itself only receiving 220*220 input, this will lead to errors in the final target position prediction. Based on this, in the second stage of the above algorithm, the ideas of the first stage are reused, but the area around the estimated point position is cropped and enlarged to continue the prediction for a more accurate result, thereby improving the final prediction accuracy.

[0180] The core idea of ​​the DeepPose algorithm is to fit the DNN network to the nonlinear regression problem, achieving end-to-end return network prediction results. For example, please refer to Figure 10, which shows a schematic diagram of the eye key points predicted by the DeepPose algorithm involved in this application. As shown in Figure 10, it shows the 19 key points of the eye area returned after DeepPose prediction.

[0181] Among them, the hollow keypoints with dots are estimates of the eyelid keypoints, the solid keypoints are estimates of the points around the iris, and the hollow keypoints are estimates of the points around the pupil. Since the pupil itself is a circular area, and in the iris recognition scenario, the pupil will be guided to become smaller through environmental adjustments, and the imaging distortion is not large at small angles, so 4 keypoints are used for the pupil keypoints. The iris area is larger than the pupil, and the upper right part is more easily blocked by the eyelid, so 8 keypoints are used for the iris area for registration. Similarly, since the upper part of the eyelid area has a larger curve than the lower part, the upper part has more keypoints and 6 keypoints are used for registration.

[0182] 3) Calculate the pupil and iris area size based on pupil, iris, and eyelid key points to determine the user's attention point and thus determine whether the person is alive

[0183] As can be seen from FIG10 above, the key point positions of the human eye can be obtained through the DeepPose algorithm. Please refer to FIG11 , which shows the image representation of both eyes when looking left and right involved in this application.

[0184] Through key point detection, please refer to Figure 12, which shows the effect diagram of the key point detection involved in this application. As shown in Figure 12, the display of the key points of no concern is eliminated, and the key points of the two corner points of the eyelids and the key points of the pupil boundary remain. Based on the iris key points shown in Figure 10 above, the pupil area and the iris area itself are circular areas. Due to distortion such as angles, they will become elliptical areas. However, considering that the pupil area is small, the distortion effect is not significant. At the same time, too many registration points will lead to higher errors. Therefore, the embodiment of this application uses 4 key points of the pupil boundary to determine a circularly marked pupil area, that is, the above-mentioned pupil center point.

[0185] The point of the pupil center area can be roughly calculated through the four points in the pupil area. The calculation formula is: x'=(min(x)+max(x)) / 2 y'=(min(y)+max(y)) / 2

[0186] Where min(x), min(y), max(x), and max(y) represent the minimum and maximum x and y coordinates of all four points.

[0187] Please refer to Figure 13, which shows a schematic diagram of the line connecting the eye corners and pupils involved in this application. As shown in Figure 13, it can be seen that when looking left and right, the ratio of the length of the line connecting the two eye corners and pupils on the left and right eyes is clearly different. Specifically, please refer to Figure 14, which shows a schematic diagram of the verification process involved in this application. As shown in Figure 14, the verification logic can be as follows:

[0188] if(LR1>LR2&&LL2>LL1):left view;

[0189] if(LR1>LR2&&LL1>LL2): look right.

[0190] This is because when the human eye is guided to look left or right in a VR scene, the design of the guiding animation prevents the user from turning their head to see the object and can only see the object by rotating their eyes. During the left and right eye rotation process, the position of the pupil relative to the eyelid corner changes. Based on this change, eye movement can be judged to determine whether the user is following the direction specified in the VR animation for liveness verification.

[0191] The above position determination uses a simple size judgment, but it can also introduce multiple judgment factors such as ratio judgment. At the same time, the above solution only describes the judgment of left and right directions. Similarly, it can also achieve estimation judgment of various angle positions (such as up and down) by using different distance judgment conditions for different scenarios.

[0192] This solution primarily relies on the DeepPose algorithm to estimate iris key points. Combined with VR animation guidance, it determines the positional relationship between the pupil and the corner of the eye, thereby determining whether the gaze is directed left or right, and ultimately determining whether the subject is alive. This solution ensures liveness determination in a natural user experience within VR scenarios.

[0193] In some embodiments, a head-mounted display device is further provided, comprising a display screen and an image capture component. The image capture component is directed to an eye position when the head-mounted display device is worn. The display screen is configured to display instruction information, the instruction information being configured to instruct a user to observe a specified position in a display space of the head-mounted display device. The image capture component is configured to capture an image of the user's eye. The eye image is configured to obtain the positions of at least two key points of the user's eye, so that if the position observed by the user, as indicated by the relationship between the positions of the at least two key points of the user's eye, matches the specified position, the user is determined to have passed verification. The step of determining that the user has passed verification if the position observed by the user, as indicated by the relationship between the positions of the at least two key points of the user's eye, matches the specified position can be performed by the head-mounted display device, for example, by a controller / processor built into the head-mounted display device. Alternatively, the step of determining that the user has passed verification if the position observed by the user, as indicated by the relationship between the positions of the at least two key points of the user's eye, matches the specified position, can also be performed by a terminal connected to the head-mounted display device. The scheme for user verification by the head-mounted display device or terminal can refer to the various embodiments described above.

[0194] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0195] Please refer to Figure 15, which shows a block diagram of an image processing device for verification provided by an embodiment of the present application. The device may include the following modules.

[0196] A display module 1501 is configured to display instruction information through a head-mounted display device, wherein the instruction information is configured to instruct a user to observe a designated position in a display space of the head-mounted display device;

[0197] An image acquisition module 1502 is configured to acquire an eye image of the user;

[0198] A key point acquisition module 1503 is configured to acquire positions of at least two key points of the user's eyes from the eye image;

[0199] The verification module 1504 is configured to determine that the user has passed the verification if the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matches the designated position.

[0200] In some embodiments, the head-mounted display device is a virtual reality (VR) display device, or the head-mounted display device is an augmented display (AR) display device.

[0201] In some embodiments, the display module 1501 is used to display the indication information at the designated location through the VR display device or the AR display device.

[0202] In some embodiments, the image acquisition module 1502 is used to acquire the eye image captured by an image acquisition component provided in the head-mounted display device; the image acquisition component points to the position of the eyeball when the head-mounted display device is worn.

[0203] In some embodiments, the at least two key points of the user's eyes include at least one of the following:

[0204] at least two key points corresponding to the user's left eye;

[0205] And, at least two key points corresponding to the user's right eye.

[0206] In some embodiments, the at least two key points corresponding to the user's left eye include a left corner key point of the user's left eye, a right corner key point of the user's left eye, and a pupil center point of the user's left eye;

[0207] At least two key points corresponding to the user's right eye, including a left corner key point of the user's right eye, a right corner key point of the user's right eye, and a pupil center point of the user's right eye;

[0208] The verification module 1504 is used to:

[0209] Obtain a first distance, a second distance, a third distance, and a fourth distance; wherein the first distance is the distance between the left corner key point of the left eye and the center point of the pupil of the left eye; the second distance is the distance between the right corner key point of the left eye and the center point of the pupil of the left eye; the third distance is the distance between the left corner key point of the right eye and the center point of the pupil of the right eye; and the fourth distance is the distance between the right corner key point of the right eye and the center point of the pupil of the right eye;

[0210] If the position observed by the user indicated by the relationship among the first distance, the second distance, the third distance, and the fourth distance matches the designated position, it is determined that the user has passed the authentication.

[0211] In some embodiments, the verification module 1504 is used to:

[0212] determining a viewing direction of the user based on a magnitude relationship among the first distance, the second distance, the third distance, and the fourth distance;

[0213] When the viewing direction of the user matches the designated position, it is determined that the user has passed the verification.

[0214] In some embodiments, the verification module 1504 is used to:

[0215] When the second distance is greater than the first distance and the fourth distance is greater than the third distance, determining that the viewing direction of the user is leftward;

[0216] When the first distance is greater than the second distance, and the third distance is greater than the fourth distance, it is determined that the viewing direction of the user is rightward.

[0217] In some embodiments, the designated location moves over time; the verification module 1504 is used to,

[0218] Obtaining a change trend of a first ratio between the first distance and the second distance;

[0219] obtaining a change trend of a second ratio between the third distance and the fourth distance;

[0220] In a case where the changing trend of the first ratio and the changing trend of the second ratio match the moving direction of the designated position, it is determined that the user has passed the authentication.

[0221] In some embodiments, the key point acquisition module 1503 is configured to input the eye image into a key point detection model to obtain positions of at least two key points of the user's eyes output by the key point detection model;

[0222] The key point detection model is a machine learning model that is pre-trained using eye image samples and the positions of at least two key points marked in the eye image samples.

[0223] FIG16 illustrates a block diagram of a computer device 1600 according to an exemplary embodiment of the present application. Computer device 1600 may be a terminal device, such as a personal computer, a VR device, an AR device, a smartphone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), or an MP4 player (Moving Picture Experts Group Audio Layer IV). Computer device 1600 may also be referred to as a user device, a portable terminal, or other similar terminology.

[0224] Typically, the computer device 1600 includes a processor 1601 and a memory 1602 .

[0225] Memory 1602 may include one or more computer-readable storage media, which may be tangible and non-transitory. Memory 1602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in memory 1602 is used to store at least one instruction, which is used to be executed by processor 1601 to implement all or part of the steps performed by the terminal in the method provided in the embodiment of the present application.

[0226] In some embodiments, the computer device 1600 may further optionally include a peripheral device interface 1603 and at least one peripheral device. Specifically, the peripheral device includes at least one of a radio frequency circuit 1604 , a touch screen display 1605 , a camera 1606 , an audio circuit 1607 , and a power supply 1608 .

[0227] In some embodiments, the computer device 1600 further includes one or more sensors 1609 , including but not limited to an acceleration sensor 1610 , a gyroscope sensor 1611 , a pressure sensor 1612 , an optical sensor 1613 , and a proximity sensor 1614 .

[0228] Those skilled in the art will appreciate that the structure shown above does not limit the computer device 1600 , and the computer device 1600 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0229] In an exemplary embodiment, a computer-readable storage medium is further provided for storing at least one computer program, which is loaded and executed by a processor to implement all or part of the steps of the methods described in the above embodiments. For example, the computer-readable storage medium may be a read-only memory, a random access memory, a read-only optical disc, a magnetic tape, a floppy disk, or an optical data storage device.

[0230] In an exemplary embodiment, a computer program product is also provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform all or part of the steps of the methods described in the various embodiments above.

[0231] Before collecting user-related data and during the process of collecting user-related data, this application can display a prompt interface, pop-up window or output voice prompt information. The prompt interface, pop-up window or voice prompt information is used to remind the user that their relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining user-related data after obtaining the user's confirmation operation on the prompt interface or pop-up window. Otherwise (that is, when the user's confirmation operation on the prompt interface or pop-up window is not obtained), the relevant steps of obtaining user-related data are terminated, that is, the user's relevant data is not obtained.

[0232] In other words, all user data collected by this application is collected with the user's consent and authorization, and the collection, use and processing of relevant user data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0233] The above-mentioned user-related data includes information (including but not limited to user accounts), data (including but not limited to text data entered by users, stored text / image data, displayed text / image data, etc.), and signal data. For example, the user data involved in this application is obtained with full authorization.

Claims

1. An image processing method for verification, characterized in that: The method is performed by a computer device, and the method comprises: Displaying instruction information through a head mounted display device, wherein the instruction information is used to instruct a user to observe a specified position in a display space of the head mounted display device; Acquiring an eye image of the user; Acquire positions of at least two key points of the user's eyes from the eye image; When the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matches the designated position, it is determined that the user has passed the authentication.

2. The method according to claim 1, characterized in that The head-mounted display device is a virtual reality (VR) display device, or the head-mounted display device is an augmented reality (AR) display device.

3. The method according to claim 2, characterized in that The displaying of the indication information by the head mounted display device includes: The indication information is displayed at the designated location through the VR display device or the AR display device.

4. The method according to any one of claims 1 to 3, characterized in that: The acquiring the eye image of the user comprises: The eye image captured by an image capture component provided in the head-mounted display device is acquired; the image capture component points to the position of the eyeball when the head-mounted display device is worn.

5. The method according to any one of claims 1 to 4, characterized in that: The at least two key points of the user's eyes include at least one of the following: at least two key points corresponding to the left eye of the user; And, at least two key points corresponding to the right eye of the user.

6. The method according to claim 5, characterized in that At least two key points corresponding to the left eye of the user include a left corner key point of the left eye of the user, a right corner key point of the left eye of the user, and a pupil center point of the left eye of the user; At least two key points corresponding to the right eye of the user include a left corner key point of the right eye of the user, a right corner key point of the right eye of the user, and a pupil center point of the right eye of the user; When the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matches the designated position, determining that the user has passed the verification includes: Obtain a first distance, a second distance, a third distance, and a fourth distance; wherein the first distance is the distance between the left eye corner key point of the left eye and the center point of the pupil of the left eye; the second distance is the distance between the right eye corner key point of the left eye and the center point of the pupil of the left eye; the third distance is the distance between the left eye corner key point of the right eye and the center point of the pupil of the right eye; the fourth distance is the distance between the right eye corner key point of the right eye and the center point of the pupil of the right eye; In a case where the position observed by the user indicated by the relationship among the first distance, the second distance, the third distance, and the fourth distance matches the designated position, it is determined that the user has passed the authentication.

7. The method according to claim 6, characterized in that When the position observed by the user indicated by the relationship among the first distance, the second distance, the third distance, and the fourth distance matches the designated position, determining that the user has passed the verification includes: Determining a viewing direction of the user based on a magnitude relationship among the first distance, the second distance, the third distance, and the fourth distance; When the viewing direction of the user matches the designated position, it is determined that the user has passed the verification.

8. The method according to claim 7, characterized in that The determining the viewing direction of the user based on the magnitude relationship among the first distance, the second distance, the third distance, and the fourth distance includes: When the second distance is greater than the first distance, and the fourth distance is greater than the third distance, determining that the viewing direction of the user is leftward; When the first distance is greater than the second distance, and the third distance is greater than the fourth distance, it is determined that the viewing direction of the user is rightward.

9. The method according to any one of claims 6 to 8, characterized in that: The designated position moves over time; When the position observed by the user indicated by the relationship among the first distance, the second distance, the third distance, and the fourth distance matches the designated position, determining that the user has passed the verification includes: Obtaining a change trend of a first ratio between the first distance and the second distance; Acquire a change trend of a second ratio between the third distance and the fourth distance; In a case where the changing trend of the first ratio and the changing trend of the second ratio match the moving direction of the designated position, it is determined that the user has passed the authentication.

10. The method according to any one of claims 1 to 9, characterized in that: The acquiring positions of at least two key points of the user's eyes from the eye image includes: Inputting the eye image into a key point detection model to obtain positions of at least two key points of the user's eyes output by the key point detection model; Among them, the key point detection model is a machine learning model that is pre-trained through eye image samples and the positions of at least two key points marked by the eye image samples.

11. A head mounted display device, characterized in that: The head mounted display device comprises a display screen and an image acquisition component; the image acquisition component points to the eyeball position when the head mounted display device is worn; The display screen is used to display instruction information, where the instruction information is used to instruct the user to observe a specified position in the display space of the head mounted display device; The image acquisition component is used to acquire the eye image of the user; The eye image is used to obtain the positions of at least two key points of the user's eyes, so as to determine that the user has passed the verification when the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matches the specified position.

12. An image processing device for verification, characterized in that: The device comprises: A display module, used for displaying instruction information through a head mounted display device, wherein the instruction information is used for instructing a user to observe a designated position in a display space of the head mounted display device; An image acquisition module, used to acquire an eye image of the user; A key point acquisition module, used to acquire positions of at least two key points of the user's eyes from the eye image; The verification module is used to determine that the user has passed the verification when the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matches the specified position.

13. The device according to claim 12, characterized in that The head-mounted display device is a virtual reality (VR) display device, or the head-mounted display device is an augmented reality (AR) display device.

14. The device according to claim 13, characterized in that The display module is used to display the indication information at the designated location through the VR display device or the AR display device.

15. The device according to any one of claims 12 to 14, characterized in that: The image acquisition module is used to acquire the eye image acquired by the image acquisition component provided in the head-mounted display device; the image acquisition component points to the position of the eyeball when the head-mounted display device is worn.

16. The device according to any one of claims 12 to 15, characterized in that The at least two key points of the user's eyes include at least one of the following: at least two key points corresponding to the left eye of the user; And, at least two key points corresponding to the right eye of the user.

17. The device according to claim 16, characterized in that At least two key points corresponding to the left eye of the user include a left corner key point of the left eye of the user, a right corner key point of the left eye of the user, and a pupil center point of the left eye of the user; At least two key points corresponding to the right eye of the user include a left corner key point of the right eye of the user, a right corner key point of the right eye of the user, and a pupil center point of the right eye of the user; The verification module is used to: Obtain a first distance, a second distance, a third distance, and a fourth distance; wherein the first distance is the distance between the left eye corner key point of the left eye and the center point of the pupil of the left eye; the second distance is the distance between the right eye corner key point of the left eye and the center point of the pupil of the left eye; the third distance is the distance between the left eye corner key point of the right eye and the center point of the pupil of the right eye; the fourth distance is the distance between the right eye corner key point of the right eye and the center point of the pupil of the right eye; In a case where the position observed by the user indicated by the relationship among the first distance, the second distance, the third distance, and the fourth distance matches the designated position, it is determined that the user has passed the authentication.

18. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the image processing method for verification as claimed in any one of claims 1 to 10.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the computer program is loaded and executed by a processor to implement the image processing method for verification as claimed in any one of claims 1 to 10.

20. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; the computer instructions are read and executed by a processor of a computer device to implement the image processing method for verification as described in any one of claims 1 to 10.

Images