Image processing method and device for verification, equipment and storage medium

Abstract

The invention provides an image processing method and device for verification, equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: displaying indication information through a head-mounted display device, wherein the indication information is used for indicating a user to observe a specified position in a display space of the head-mounted display device; acquiring an eye image of the user; acquiring the positions of at least two key points of the eyes of the user from the eye image; and responding to the position observed by the user indicated by the relationship between the positions of the at least two key points of the eyes of the user, matching the position with the specified position, and determining that the user passes the verification. According to the scheme, the verification efficiency can be improved under the condition of ensuring the verification accuracy.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image processing method, apparatus, device and storage medium for verification. Background Art

[0002] With the continuous development of network technology, network / data security has received more and more attention. When users use electronic products or network applications, the scenarios for user verification are becoming more and more diverse.

[0003] In related technologies, user verification is usually performed using methods such as face video verification and password verification. For example, taking the liveness verification scenario in face verification as an example, first, the user needs to adjust the shooting angle and distance of the front camera of the mobile phone so that the front camera can clearly capture the face. Then, follow the prompts to blink, shake the head, etc. The verification program detects whether the current user has passed the liveness verification through the picture captured by the front camera of the mobile phone.

[0004] However, the above verification methods are usually complicated to operate, and the verification process for users usually takes a certain amount of time, resulting in low verification efficiency. Summary of the invention

[0005] The embodiment of the present application provides an image processing method, device, equipment and storage medium for verification, which can improve the verification efficiency. The technical solution is as follows:

[0006] In one aspect, there is provided an image processing method for verification, the method comprising:

[0007] Displaying instruction information through a head mounted display device, wherein the instruction information is used to instruct a user to observe a specified position in a display space of the head mounted display device;

[0008] Acquiring an eye image of the user;

[0009] Acquire positions of at least two key points of the user's eyes from the eye image;

[0010] In response to the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matching the designated position, it is determined that the user has passed the authentication.

[0011] On the other hand, a head-mounted display device is provided, the head-mounted display device comprising a display screen and an image acquisition component; the image acquisition component points to the eyeball position when the head-mounted display device is worn;

[0012] The display screen is used to display instruction information, where the instruction information is used to instruct the user to observe a specified position in the display space of the head mounted display device;

[0013] The image acquisition component is used to acquire the eye image of the user;

[0014] The eye image is used to obtain the positions of at least two key points of the user's eyes, so as to determine that the user has passed the verification when the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matches the specified position.

[0015] In another aspect, there is provided an image processing apparatus for verification, the apparatus comprising:

[0016] A display module, used for displaying instruction information through a head mounted display device, wherein the instruction information is used for instructing a user to observe a designated position in a display space of the head mounted display device;

[0017] An image acquisition module, used to acquire an eye image of the user;

[0018] A key point acquisition module, used to acquire positions of at least two key points of the user's eyes from the eye image;

[0019] The verification module is used to determine that the user has passed the verification in response to the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matching with the designated position.

[0020] In some embodiments, the head-mounted display device is a virtual reality (VR) display device, or the head-mounted display device is an augmented display (AR) display device.

[0021] In some embodiments, displaying the indication information through a head mounted display device includes:

[0022] The indication information is displayed at the designated location through the VR display device or the AR display device.

[0023] In some embodiments, the image acquisition module is used to acquire the eye image acquired by an image acquisition component provided in the head-mounted display device; the image acquisition component points to the position of the eyeball when the head-mounted display device is worn.

[0024] In some embodiments, the at least two key points of the user's eyes include at least one of the following:

[0025] at least two key points corresponding to the left eye of the user;

[0026] And, at least two key points corresponding to the right eye of the user.

[0027] In some embodiments, the at least two key points corresponding to the left eye of the user include a left corner key point of the left eye of the user, a right corner key point of the left eye of the user, and a pupil center point of the left eye of the user;

[0028] At least two key points corresponding to the right eye of the user include a left corner key point of the right eye of the user, a right corner key point of the right eye of the user, and a pupil center point of the right eye of the user;

[0029] The verification module is used to:

[0030] Obtain a first distance, a second distance, a third distance, and a fourth distance; wherein the first distance is the distance between the left eye corner key point of the left eye and the center point of the pupil of the left eye; the second distance is the distance between the right eye corner key point of the left eye and the center point of the pupil of the left eye; the third distance is the distance between the left eye corner key point of the right eye and the center point of the pupil of the right eye; the fourth distance is the distance between the right eye corner key point of the right eye and the center point of the pupil of the right eye;

[0031] In response to the position observed by the user indicated by the relationship among the first distance, the second distance, the third distance, and the fourth distance matching the designated position, it is determined that the user has been authenticated.

[0032] In some embodiments, the verification module is used to:

[0033] Determining a viewing direction of the user based on a magnitude relationship among the first distance, the second distance, the third distance, and the fourth distance;

[0034] In response to the viewing direction of the user matching the designated position, it is determined that the user has passed the authentication.

[0035] In some embodiments, the verification module is used to:

[0036] In response to the second distance being greater than the first distance, and the fourth distance being greater than the third distance, determining that the viewing direction of the user is leftward;

[0037] In response to the first distance being greater than the second distance, and the third distance being greater than the fourth distance, it is determined that the viewing direction of the user is rightward.

[0038] In some embodiments, the designated location moves over time; the verification module is used to,

[0039] The step of determining that the user has passed the verification in response to the position observed by the user indicated by the relationship among the first distance, the second distance, the third distance, and the fourth distance matching the designated position comprises:

[0040] Obtaining a change trend of a first ratio between the first distance and the second distance;

[0041] Acquire a change trend of a second ratio between the third distance and the fourth distance;

[0042] In response to the changing trend of the first ratio and the changing trend of the second ratio matching the moving direction of the designated position, it is determined that the user has passed the authentication.

[0043] In some embodiments, the key point acquisition module is used to input the eye image into a key point detection model to obtain the positions of at least two key points of the user's eyes output by the key point detection model;

[0044] Among them, the key point detection model is a machine learning model that is pre-trained through eye image samples and the positions of at least two key points marked by the eye image samples.

[0045] On the other hand, a computer device is provided, comprising a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the above-mentioned image processing method for verification.

[0046] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the above-mentioned image processing method for verification.

[0047] In another aspect, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the image processing method for verification provided in the above-mentioned various optional implementations.

[0048] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0049] The information displayed by the head-mounted display device instructs the user to observe a designated position in the display space of the head-mounted display device, and at the same time obtains an eye image of the user, obtains the relationship between the positions of at least two key points of the user's eyes based on the user image, and when it is determined that the position observed by the user matches the designated position, it is determined that the user has passed the verification. Since the head-mounted display device will rotate with the user's head, the relationship between the positions of the key points in the user's eye image can accurately represent the user's current observation position, and then determine whether the user is observing the designated position, so as to achieve accurate verification of the user. At the same time, in the above process, the user only needs to rotate his eyeballs and observe the designated position, and the user does not need to perform other operations, which can greatly simplify the user operations in the verification process, thereby improving the verification efficiency while ensuring the accuracy of verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0051] Figure 2 is a flowchart of an image processing method for verification provided by an embodiment of the present application;

[0052] Figure 3 is a flowchart of an image processing method for verification provided by an embodiment of the present application;

[0053] Figure 4 is a flowchart of an image processing method for verification provided by an embodiment of the present application;

[0054] Figure 5 is a flowchart of an image processing method for verification provided by an embodiment of the present application;

[0055] Figure 6 is a flowchart of an image processing method for verification provided by an embodiment of the present application;

[0056] Figure 7 This is a schematic diagram of a scene interface involved in this application;

[0057] Figure 8 This is a schematic diagram of a DeepPose algorithm involved in this application;

[0058] Fig. 9 This is another DeepPose algorithm schematic diagram involved in this application;

[0059] Fig.10 is a schematic diagram of predicting eye key points by DeepPose algorithm involved in this application;

[0060] Fig.11It is the image representation diagram of both eyes when gazing to the left and to the right involved in this application;

[0061] Fig.12 This is a rendering of the key point detection involved in this application;

[0062] Fig.13 It is a schematic diagram of the connection between the eye corner and the pupil involved in this application;

[0063] Fig.14 It is a schematic diagram of the verification process involved in this application;

[0064] Fig.15 is a block diagram of an image processing device for verification provided by an embodiment of the present application;

[0065] Fig.16 It is a structural block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0067] 1) Infrared image: An infrared image captured by an infrared sensor using infrared light imaging.

[0068] Since the iris of the eye has a special absorption characteristic for infrared, and at the same time avoids the interference of VR color imaging in the virtual reality (VR) scenario, using infrared images to collect eye information can have better anti-interference and recognition effects.

[0069] 2) Liveness detection: A method for determining the true physiological characteristics of an object in some identity authentication scenarios. For example, in face recognition applications, liveness detection can verify whether the user is the real person through combined actions such as blinking, opening the mouth, shaking the head, and nodding, using key point positioning and detection technologies. It can effectively resist common attack methods such as photos, videos, face swapping, masks, occlusion, 3D animation, and screen reshoots, thereby helping users identify illegal behavior and protect their interests.

[0070] 3) Identity verification: Also known as "verification" or "authorization", it refers to the technology of confirming the identity of a user through certain means. There are many methods of identity verification, such as identity verification based on shared keys, identity verification based on public key encryption algorithms, identity verification based on reserved information, etc.

[0071] Figure 1A schematic diagram of an implementation environment provided by an exemplary embodiment of the present application is shown. The implementation environment may include: a head mounted display device 110; optionally, the implementation environment may also include a first terminal 120; optionally, the implementation environment may also include a server 130.

[0072] Among them, the server involved above can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0073] In some embodiments, the head mounted display device 110 may display images on the display screen by itself; or, the head mounted display device 110 may exchange data with the server 130 to display images on the display screen of the head mounted display device 110. For example, an application may be installed in the head mounted display device 110, and the application may be a VR application (such as a VR game) or an AR application (such as an AR game), and so on.

[0074] Figure 1 Only one head mounted display device is shown in the figure, but in different embodiments, there may be multiple other head mounted display devices accessing the server 130 or the first terminal 120.

[0075] Specifically, the first terminal 120 may install and run an application 121 and a streaming application 122, and the application 121 may display images on the display screen of the head mounted display device 110 alone, or may interact with the server 130 to display images on the display screen of the head mounted display device 110. For example, the application 121 may be a VR application (such as a VR game) or an AR application (such as an AR game), and so on.

[0076] Figure 1 Only one terminal is shown, but in different embodiments, multiple other terminals can access the head mounted display device 110 or the virtual server 130. Optionally, there are one or more terminals corresponding to the developer, and a development platform for developing the above application 121 or streaming application 122 is installed on the terminal.

[0077] The head mounted display device 110 and other head mounted display devices may be connected to the first terminal 120 or the server 130 via a wireless network or a wired network.

[0078] The head-mounted display device 110 may include a display and an input component, wherein the display may be used to display an encoded video stream of a virtual scene / enhanced display scene, and the input component is used to receive user input data and return the input data to the first terminal 120 or the server 130; wherein the input component may include at least one of a sensor (such as a motion sensor, a touch sensor, a light sensor, etc.) and a physical button, and the input data may include sensor data and physical button pressing data, etc.

[0079] The server 130 includes at least one of a single server, a server cluster consisting of multiple servers, a cloud computing platform, and a virtualization center. The server 130 can provide background support for the application 121 running in the terminal, or for the application in the head-mounted display device 110. Optionally, the server 130 undertakes the main computing work, and the terminal or the head-mounted display device 110 undertakes the secondary computing work; or, the server 130 undertakes the secondary computing work, and the terminal or the head-mounted display device 110 undertakes the main computing work; or, the server 130, and the terminal or the head-mounted display device 110 adopt a distributed computing architecture for collaborative computing.

[0080] In an illustrative example, the server 130 includes a memory 131, a processor 132, a user account database 133, an application service module 134, and a user-oriented input / output interface (I / O interface) 135. The processor 132 is used to load instructions stored in the server 130 and process data in the user account database 133 and the application service module 134; the user account database 133 is used to store data of user accounts used by the first terminal 110 and other terminals, such as the user account avatar, the user account name, and the user account level; the application service module 134 is used to provide background support services for the application program, etc.; the user-oriented I / O interface 135 is used to establish communication and exchange data with the head-mounted display device 110 or the first terminal 120 through a wireless network or a wired network.

[0081] Please refer to Figure 2 , which shows a flowchart of an image processing method for verification provided by an embodiment of the present application. For the sake of convenience, only the computer device that executes each step is introduced and described. The above-mentioned computer device can be a head-mounted display device, a terminal or a server; the above-mentioned head-mounted display device can be the above-mentioned Figure 1The head mounted display device 110 shown in the figure, the terminal may be the Figure 1 The terminal 120 shown in the figure, the server can be the Figure 1 The server 130 is shown; the method may include the following steps.

[0082] Step 210: Display instruction information through the head-mounted display device, where the instruction information is used to instruct the user to observe a specified position in the display space of the head-mounted display device.

[0083] Among them, the display space of the head-mounted display device refers to the space where information displayed by the head-mounted display device can exist.

[0084] Optionally, the display space of the head-mounted display device may be a three-dimensional space, and the head-mounted display device may display the designated information element at any position in the three-dimensional space.

[0085] Alternatively, the display space of the head-mounted display device may also be a two-dimensional plane space, such as a display screen, and the head-mounted display device may display the specified information element at any position in the two-dimensional plane space.

[0086] Among them, the head-mounted display device may include a display screen. Optionally, the head-mounted display device may include a single display screen or two display screens. When it includes two display screens, the two display screens correspond to the two eyes of the user respectively. The two display screens can respectively display the images when the two eyes observe the image elements / objects in the virtual space, thereby creating a three-dimensional display effect for the user.

[0087] For example, the head-mounted display device is a virtual reality VR display device, or the head-mounted display device is an augmented display AR display device.

[0088] Virtual reality, as the name implies, is the combination of virtuality and reality. Theoretically, virtual reality technology (VR) is a computer simulation system that can create and experience a virtual world. It uses computers to generate a simulated environment and immerse users in it. Virtual reality technology uses real-life data, electronic signals generated by computer technology, and combines them with various output devices to transform them into phenomena that people can feel. These phenomena can be real objects in reality or substances that are invisible to our naked eyes, expressed through three-dimensional models. Because these phenomena are not what we can see directly, but are simulated in the real world through computer technology, they are called virtual reality.

[0089] In the embodiment of the present application, the VR display device can display a three-dimensional virtual scene to the user, that is, the three-dimensional scene picture seen by the user through the VR display device is a simulated world picture simulated by a computer, such as a VR game picture, etc. Of course, the VR display device can also be used to play three-dimensional videos (three-dimensional movies).

[0090] Augmented Reality (AR) technology is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, real-time detection and registration, intelligent interaction, and sensing. It simulates computer-generated virtual information such as text, images, three-dimensional models, music, and videos, and applies them to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0091] Augmented reality technology is also known as augmented reality. AR augmented reality technology is a relatively new technology that integrates real-world information and virtual-world information content. It simulates and processes physical information that is difficult to experience in the real world based on computer and other scientific technologies, and effectively applies virtual information content in the real world. In this process, it can be perceived by human senses, thus achieving a sensory experience beyond reality. After the real environment and virtual objects overlap, they can exist simultaneously in the same screen and space.

[0092] Augmented reality technology can not only effectively reflect the content of the real world, but also enable the display of virtual information content. These delicate contents complement and overlap each other. In visual augmented reality, users need to overlap the real world with computer graphics based on head-mounted display devices, and after the overlap, they can fully see the real world around it. Augmented reality technology mainly includes new technologies and means such as multimedia, 3D modeling, and scene fusion. There is a clear difference between the information content provided by augmented reality and the information content that humans can perceive.

[0093] The AR display device can be implemented as a single-lens or dual-lens AR glasses. In addition to transmitting light and allowing the user to see the real world behind the lenses, the lenses of the AR glasses can also reflect virtual images to the user's eye position, allowing the user to simultaneously see virtual image elements / objects superimposed on the real world.

[0094] Whether it is an AR display device or a VR display device, the specified information can be displayed at a specified location.

[0095] In some embodiments, the computer device may display indication information at a specified location through a VR display device or an AR display device.

[0096] In an embodiment of the present application, in order to better prompt the user to observe the specified location, the computer device can directly display the instruction information at the specified location through a VR display device or an AR display device.

[0097] Step 220: Acquire the user's eye image.

[0098] The eye image of the user may be an eye image of the user when the user wears the head-mounted display device.

[0099] Specifically, when a user wears the above-mentioned head-mounted display device, the head-mounted display device can display specified information so that the user's eyes can focus on the specified position.

[0100] In some embodiments, a computer device may obtain an eye image captured by an image acquisition component provided in a head-mounted display device; the image acquisition component points to the position of the eyeball when the head-mounted display device is worn.

[0101] When a user wears a head-mounted display device, the user's eyes are usually blocked by the head-mounted display device. Therefore, if the user's eye image is collected from outside the head-mounted display device, a clear image of the user's eye may not be correctly collected. In this regard, in an embodiment of the present application, an image acquisition component may be provided in the head-mounted display device, and the image acquisition component may point to the position of the user's eyeball when the head-mounted display device is worn, so that the image acquisition component can accurately collect a clear user's eye image to ensure the accuracy of subsequent verification.

[0102] Optionally, the above-mentioned eye image may be an infrared image.

[0103] Step 230: Obtain positions of at least two key points of the user's eyes from the eye image.

[0104] The human eye area may contain multiple types of key points, and each type of key point may contain multiple key points.

[0105] For example, the eye part of a human body may include one or more key points of an eyelid, one or more key points of an iris, one or more key points of a pupil, and so on.

[0106] Step 240: In response to the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matching the designated position, it is determined that the user has passed the verification.

[0107] When the user's eyes focus on spatial positions in different directions and depths, the positions of the key points of the human eye part will also be different. Conversely, through the positions of the key points of the human eye part, the approximate position (including direction, distance, etc.) of the user's eyes currently focusing / observing can be inferred.

[0108] Based on the above principle, in an embodiment of the present application, a specified information is displayed by a head-mounted display device to instruct the user to observe a specified position in the display space of the head-mounted display device. When the head-mounted display device is worn on the user's head, it will rotate with the user's head. That is to say, no matter how the user turns his head, the position of the information displayed by the head-mounted display device is fixed relative to the user's eyes. For different positions in the display space of the head-mounted display device, the user can only observe by changing the direction and focal length of the eyeballs. This can eliminate the interference caused by the rotation of the head when the user observes the specified position in the display space of the head-mounted display device, and accurately identify the position observed by the user through the relationship between the positions of the key points of the eyes, and then determine whether the user has passed the verification through whether the position observed by the user matches the specified position.

[0109] To summarize, the scheme shown in the embodiment of the present application instructs the user to observe a specified position in the display space of the head-mounted display device through the information displayed by the head-mounted display device, and at the same time obtains the user's eye image, obtains the relationship between the positions of at least two key points of the user's eyes based on the user image, and determines that the position domain observed by the user matches the specified position, then determines that the user has passed the verification. Since the head-mounted display device will rotate with the user's head, the relationship between the positions of the key points in the user's eye image can accurately represent the user's current observation position, and then determine whether the user is observing the specified position, so as to achieve accurate verification of the user. At the same time, in the above process, the user only needs to rotate his eyeballs and observe the specified position, and the user does not need to perform other operations, which can greatly simplify the user operations in the verification process, thereby improving the verification efficiency while ensuring the accuracy of verification.

[0110] Among them, the above application Figure 2 The solution involved in the embodiment shown can be applied to the user's living body detection scenario, for example, through the above Figure 2 In the scheme shown, when the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes is detected to match the specified position, it can be considered that the user is currently performing a real-person operation.

[0111] Alternatively, the above Figure 2 The solution involved in the embodiment shown can also be applied to the user identity authentication scenario, for example, through the above Figure 2The scheme shown can display different verification information at different positions in the display space of the head-mounted display device through the head-mounted display device, wherein the identity verification information at the designated position is the correct identity verification information. When the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes is detected and matches the designated position, it can be considered that the identity verification information currently observed by the user is correct, and the user can be confirmed to have passed the identity authentication.

[0112] Based on the above Figure 2 In some embodiments of the scheme involved in the embodiment shown, the at least two key points of the user's eyes include at least one of the following: at least two key points corresponding to the user's left eye; and at least two key points corresponding to the user's right eye.

[0113] In a possible implementation of an embodiment of the present application, a computer device can roughly determine the direction and / or depth of the user's current observation through at least two key points of a single eye, the left eye or the right eye, and then match the user's observation position indicated by the relationship between the positions of at least two key points of the user's eyes with the specified position to determine that the user has passed the verification; this solution only needs to obtain at least two key points of a single eye, which can simplify the workload of obtaining and analyzing the key point positions of the eye image, improve verification efficiency, and save hardware resources; in addition, when the eye images of the two eyes are collected separately and the quality of one of the eye images is poor, verification can also be performed through the eye image of the other eye, thereby ensuring the stability of the verification process.

[0114] In another possible implementation of the embodiment of the present application, the computer device determines the direction and / or depth of the user's current observation by combining at least two key points of the left eye and the right eye, thereby ensuring the accuracy of the user's observation position and further ensuring the accuracy of the verification.

[0115] Based on the above Figure 2 In the embodiment shown, the computer device performs verification by combining at least two key points of the left eye and the right eye, which can be referred to as Figure 3 .in, Figure 3 The flowchart of the image processing method for verification provided by an embodiment of the present application is shown. In the case where at least two key points corresponding to the user's left eye include the left corner key point of the user's left eye, the right corner key point of the user's left eye, and the center point of the pupil of the user's left eye, and at least two key points corresponding to the user's right eye include the left corner key point of the user's right eye, the right corner key point of the user's right eye, and the center point of the pupil of the user's right eye, in Figure 2 Based on the embodiment shown, Figure 3In the illustrated embodiment, the above step 240 may include step 240a and step 240b:

[0116] Step 240a: Obtain the first distance, the second distance, the third distance, and the fourth distance.

[0117] Among them, the first distance is the distance between the left corner key point of the left eye and the center point of the pupil of the left eye; the second distance is the distance between the right corner key point of the left eye and the center point of the pupil of the left eye; the third distance is the distance between the left corner key point of the right eye and the center point of the pupil of the right eye; the fourth distance is the distance between the right corner key point of the right eye and the center point of the pupil of the right eye.

[0118] In an embodiment of the present application, at least two key points corresponding to the user's left eye include at least the left corner key point of the user's left eye, the right corner key point of the user's left eye, and the center point of the pupil of the user's left eye; at least two key points corresponding to the user's right eye include at least the left corner key point of the user's right eye, the right corner key point of the user's right eye, and the center point of the pupil of the user's right eye.

[0119] Step 240b: In response to the position observed by the user indicated by the relationship among the first distance, the second distance, the third distance and the fourth distance matching the designated position, it is determined that the user has passed the verification.

[0120] In some embodiments, in response to a position observed by the user indicated by the relationship among the first distance, the second distance, the third distance, and the fourth distance matching a designated position, determining that the user has passed the verification includes:

[0121] Determining the viewing direction of the user based on the magnitude relationship among the first distance, the second distance, the third distance, and the fourth distance;

[0122] In response to the user's viewing direction matching the designated position, it is determined that the user is authenticated.

[0123] In some embodiments, determining the viewing direction of the user based on the magnitude relationship among the first distance, the second distance, the third distance, and the fourth distance includes:

[0124] In response to the second distance being greater than the first distance and the fourth distance being greater than the third distance, determining that the viewing direction of the user is leftward;

[0125] In response to the first distance being greater than the second distance, and the third distance being greater than the fourth distance, it is determined that the viewing direction of the user is rightward.

[0126] In an embodiment of the present application, for one image for each left eye, the computer device can respectively determine the distance between the two corners of the left and right eyes and the pupils. Through the size relationship between the two corners of the left and right eyes and the pupils, it can be determined whether the user's eyes are looking left or right. Combined with the position of the specified position relative to the user's eyes, it can be determined whether the user is observing the specified position, and then determine whether the user has passed the verification. This process only needs to determine whether the user's eyes are looking left or right. The judgment accuracy is high, and the judgment logic is simple, which can ensure the accuracy and efficiency of the verification.

[0127] Among them, the above embodiment is explained by combining at least two key points of the left eye and the right eye for verification as an example. Optionally, verification can also be performed through at least two key points of the left eye or the right eye. For example, still taking the above-mentioned first distance, second distance, third distance and fourth distance as an example, if the second distance is greater than the first distance, it can be determined that the user's viewing direction is to the left. Conversely, if the first distance is greater than the second distance, it is determined that the user's viewing direction is to the right; for another example, if the fourth distance is greater than the third distance, it is determined that the user's viewing direction is to the left. If the third distance is greater than the fourth distance, it is determined that the user's viewing direction is to the right.

[0128] In the above embodiment, the computer device can flexibly choose to verify through at least two key points of each of the two eyes, or through at least two key points of one of the eyes. For example, after the computer device captures the eye images of the two eyes, it can first detect the image quality of the eye images of the two eyes, and extract key points of the eye images of the two eyes that meet the quality requirements to obtain at least two key points of the corresponding eyes, and then determine whether the user's viewing direction is left or right based on the at least two extracted key points, and then determine whether the user has passed the verification.

[0129] The above-mentioned quality requirements may include at least one of the following:

[0130] The clarity of the eye image is above the clarity threshold;

[0131] The size of the eyeball region visible in the eye image is greater than the size threshold.

[0132] That is to say, in an embodiment of the present application, the computer device can select an eye image with sufficiently high clarity and a sufficiently large eyeball area from the eye images of the left and right eyes, and determine whether the user's viewing direction is to the left or right based on the key points of the selected eye image, and then determine whether the user has passed the verification, thereby improving the accuracy of the verification.

[0133] In some embodiments, the designated location moves over time; in response to the location observed by the user indicated by the relationship among the first distance, the second distance, the third distance, and the fourth distance, matching the designated location, determining that the user has passed the verification includes:

[0134] Obtaining a change trend of a first ratio between the first distance and the second distance;

[0135] obtaining a change trend of a second ratio between the third distance and the fourth distance;

[0136] In response to the changing trend of the first ratio and the changing trend of the second ratio matching the moving direction of the designated position, it is determined that the user has passed the authentication.

[0137] In the above embodiment, the computer device can control the specified position to move in the specified direction within a specified time period. At the same time, within the specified time period, multiple eye images of the left eye and the right eye are collected. For the multiple eye images of the left eye, first ratios of the multiple eye images are determined through at least two key points of the multiple eye images, and then the change trend of the first ratio is determined according to the order of the collection time points of the multiple eye images from front to back, that is, whether the first ratio gradually increases or decreases; correspondingly, for the multiple eye images of the right eye, second ratios of the multiple eye images are determined through at least two key points of the multiple eye images, and then the change trend of the second ratio is determined according to the order of the collection time points of the multiple eye images from front to back, that is, whether the second ratio gradually increases or decreases.

[0138] For example, if the first ratio between the first distance and the second distance gradually decreases, and the second ratio between the third distance and the fourth distance also gradually decreases, it is determined that the moving direction of the specified position is moving to the left; for another example, if the first ratio between the first distance and the second distance gradually increases, and the second ratio between the third distance and the fourth distance also gradually increases, it is determined that the moving direction of the specified position is moving to the right.

[0139] Among them, the above embodiment is explained by combining at least two key points of the left eye and the right eye for verification as an example. Optionally, verification can also be performed through at least two key points of the left eye or the right eye. For example, still taking the above-mentioned first distance, second distance, third distance and fourth distance as an example, if the first ratio between the first distance and the second distance gradually decreases, it can be determined that the moving direction of the specified position is moving to the left. Conversely, if the first ratio gradually increases, it can be determined that the moving direction of the specified position is moving to the right. For another example, if the second ratio between the third distance and the fourth distance gradually decreases, it can be determined that the moving direction of the specified position is moving to the left. Conversely, if the second ratio gradually increases, it can be determined that the moving direction of the specified position is moving to the right.

[0140] In an embodiment of the present application, the computer device may also continuously acquire images of the left and right eyes of the user, and respectively determine the distances between the two corners of the left and right eyes and the pupils, calculate the changing trend of the ratio of the distances between the two corners of the left and right eyes and the pupils during this process, thereby determining the moving direction of the position observed by the user, and combining the moving direction of the user's observation position with the moving direction of the designated position indicated by the designated information to determine whether the user has passed the verification.

[0141] Based on the above Figure 2 In the embodiment shown, the computer device performs verification by combining at least two key points of the left eye and the right eye, which can be referred to as Figure 4 .in, Figure 4 A flowchart of an image processing method for verification provided by an embodiment of the present application is shown. In the case where at least two key points corresponding to the user's left eye include multiple key points at the edge of the pupil of the user's left eye, and at least two key points corresponding to the user's right eye include multiple key points at the edge of the pupil of the user's right eye, Figure 2 Based on the embodiment shown, Figure 4 In the illustrated embodiment, the above step 240 may include step 240c and step 240d:

[0142] Step 240c: Obtain the changing trend of the first area and the changing trend of the second area; the first area is the area of ​​the area surrounded by multiple key points on the edge of the pupil of the left eye, and the second area is the area of ​​the area surrounded by multiple key points on the edge of the pupil of the left eye.

[0143] In the above embodiment, the computer device can control the depth of the specified position to gradually increase or decrease within a specified time period, wherein the depth of the above specified position may refer to the distance of the specified position relative to the human eye, the closer the distance is, the smaller the depth is, and conversely, the farther the distance is, the greater the depth is.

[0144] At the same time, within the specified time period, multiple eye images of the left eye and the right eye are collected. For the multiple eye images of the left eye, the first areas of the multiple eye images are determined through the multiple key points of the pupil edges in the multiple eye images, and then the change trend of the first area is determined according to the order of the collection time points of the multiple eye images from front to back, that is, whether the first area gradually increases or decreases; correspondingly, for the multiple eye images of the right eye, the second areas of the multiple eye images are determined through the multiple key points of the pupil edges in the multiple eye images, and then the change trend of the second area is determined according to the order of the collection time points of the multiple eye images from front to back, that is, whether the second area gradually increases or decreases.

[0145] Step 240d: In response to the change trend of the first area and the change trend of the second area matching the change direction of the depth at the designated location, it is determined that the user has passed the verification.

[0146] Among them, when the human eye observes objects at different depths (or called far and near), the pupil area of ​​the human eye will change. Specifically, the pupil will shrink when the human eye observes nearby objects, and will expand when observing distant objects. Based on the above principle, in an embodiment of the present application, the computer device can control the depth of a specified position to change. At the same time, the pupil area in the collected eye image of the user is used to judge the changing trend of the depth of the user's line of sight focus and whether it matches the changing direction of the depth of the specified position. If so, it is determined that the user has passed the verification.

[0147] Specifically, if the first area and the second area gradually decrease, it is determined that the depth of the user's line of sight is shrinking. At this time, if the depth of the designated position gradually decreases, the changing trend of the depth of the user's line of sight is determined to match the changing direction of the depth of the designated position, and it is determined that the user has passed the verification; correspondingly, if the first area and the second area gradually increase, it is determined that the depth of the user's line of sight is increasing. At this time, if the depth of the designated position gradually becomes deeper, the changing trend of the depth of the user's line of sight is determined to match the changing direction of the depth of the designated position, and it is determined that the user has passed the verification.

[0148] In the solution shown in the embodiment of the present application, user verification can be achieved by changing the depth of the specified position in combination with the change in the user's pupil area, which expands the implementation method of user verification and improves the security of verification.

[0149] Among them, the above embodiment is explained by combining multiple key points of the pupil edge of the left eye and the right eye for verification as an example. Optionally, verification can also be performed through multiple key points of the pupil edge of the left eye or the right eye. For example, still taking the above-mentioned first area and second area as an example, if the first area / second area gradually decreases, it can be determined that the depth of the user's line of sight is shrinking. Conversely, if the first area / second area gradually increases, it can be determined that the depth of the user's line of sight is increasing.

[0150] Optionally, since the area of ​​the human pupil is related not only to the distance at which the user observes the object, but also to the intensity of light, in an embodiment of the present application, while controlling the depth change of a specified position, the computer device can also control the display brightness of the head-mounted display device to remain unchanged, thereby avoiding interference with the area detection of the user's pupil and improving the accuracy of verification.

[0151] Based on the above Figure 2In the embodiment shown, the computer device performs verification by combining at least two key points of the left eye and the right eye, which can be referred to as Figure 5 .in, Figure 5 A flowchart of an image processing method for verification provided by an embodiment of the present application is shown. In the case where at least two key points corresponding to the left eye of the user include the pupil center point of the left eye of the user and the center point of the left eye of the user, and at least two key points corresponding to the right eye of the user include the pupil center point of the right eye of the user and the center point of the right eye of the user, Figure 2 Based on the embodiment shown, Figure 5 In the illustrated embodiment, the above step 240 may include step 240e and step 204f:

[0152] Step 240e: Acquire a first direction and a second direction; the first direction is the direction of the pupil center of the left eye relative to the center of the left eye, and the second direction is the direction of the pupil center of the right eye relative to the center of the right eye.

[0153] The left eye center point may be the center of an area surrounded by eyelids in the eye image of the left eye; correspondingly, the right eye center point may be the center of an area surrounded by eyelids in the eye image of the right eye.

[0154] Step 240f: In response to the first direction and the second direction matching the direction of the designated position relative to the center point of the display area of ​​the head mounted display device, it is determined that the user has passed the verification.

[0155] In the embodiment of the present application, for each image of the left and right eyes, the computer device can determine the direction of the two pupils of the left and right eyes relative to the center of the eyes, and then determine which direction the user is looking at. Combined with the direction of the designated position relative to the center of the screen, it can be determined whether the user is observing the designated position, and then determine whether the user has passed the verification. This solution can combine the user's observation in the left and right and up and down directions to more accurately detect the direction in which the user is currently observing, and then determine whether the user is looking at the designated position, thereby improving the accuracy and security of verification.

[0156] Among them, the above embodiment is explained by taking the direction of the pupil center of the left eye and the right eye relative to the center of the eye as an example for verification. Optionally, verification can also be performed by the direction of the pupil center of the left eye or the right eye relative to the center of the eye. For example, still taking the above-mentioned first direction and second direction as an example, if the first direction / second direction matches the direction of the specified position relative to the center of the display area of ​​the head-mounted display device, it can be determined that the user has passed the verification.

[0157] Based on the above Figure 2 or Figure 3 The embodiments shown involve the following scheme: Figure 6 A flowchart of an image processing method for verification provided by an embodiment of the present application is shown. Figure 2 Based on the embodiment shown, Figure 6 In the illustrated embodiment, the above step 230 may include step 230a:

[0158] Step 230a: Input the eye image into the key point detection model to obtain the positions of at least two key points of the user's eyes output by the key point detection model.

[0159] Among them, the key point detection model is a machine learning model that is pre-trained through eye image samples and the positions of at least two key points marked by the eye image samples.

[0160] In an embodiment of the present application, a process of identifying at least two key points from an eye image can be implemented with the help of a machine learning model.

[0161] For example, the key point detection model may be a deep learning model for implementing classification or regression, such as an image classification or regression model such as a DeepPose model.

[0162] For example, the developer collects several eye image samples in advance, and marks the positions of at least two key points in each eye image sample. Then, the eye image samples are input into the DeepPose model to obtain the predicted positions of at least two key points in the eye image sample output by the DeepPose model. Then, according to the positions of at least two key points marked in the eye image sample and the predicted positions of at least two key points in the eye image sample, the loss is calculated by a pre-set loss function, and the model parameters of the DeepPose model are updated by the calculated loss, for example, by using a gradient descent algorithm, and the model parameters of the DeepPose model are updated by the calculated loss. The above process is repeated until the DeepPose model is trained to convergence, at which point the DeepPose model can be used as the above-mentioned key point detection model. In the subsequent application process, after the computer device inputs the eye image into the key point detection model, the key point detection model can output the positions of at least two key points in the eye image.

[0163] In one embodiment, the solution shown in the above embodiment of the present application can be applied to realize liveness judgment in the recognition scenario. As the core link, it can effectively ensure the security of recognition and prevent it from being attacked by paper pieces or fake eyes, resulting in recognition attacks.

[0164] For example, this solution proposes a liveness judgment algorithm for identifying behavioral intention based on attention judgment. By constructing different object scenes in the VR scene, guiding the eyes to look in different directions, and locating the position and shape of the eyes and irises, it is estimated whether the user's attention is in line with the direction guided by the object, thereby judging whether it is a real user operation.

[0165] From the product side, after the user puts on the VR device, the recognition camera will continuously collect photos of the user's eyes, guide the eyes to look in different directions by constructing different object scenes in the VR scene, and estimate whether the user's attention is in line with the object guidance by locating the position of the key points in the eyes and the shape or size of the area surrounded by the key points, thereby determining whether the user is operating in person.

[0166] From a technical perspective, the solution consists of the following three parts:

[0167] 1) VR scene construction: including the left and right positions of moving objects, etc.

[0168] In this solution, Unity can be used to build VR 3D scenes, for example, Figure 7 A schematic diagram of a scene interface involved in this application is shown. Figure 7 As shown, in this scene, the panel is always in front of the user's eyes, and what changes is the position of the eye icon 71. By changing the position of the eye icon 71, the user is guided to look, thereby guiding the user's eyes to rotate left and right.

[0169] 2) Locate the position and shape of the eyes and irises: Calculate the key points of the pupil, iris, and eyelid areas based on DeepPose

[0170] Eyeball attention point estimation depends on the eye position, pupil position, and the relative position of pupil position and eye, so as to correctly estimate the eye attention point scheme, evaluate whether the user's attention is in line with the interactive guidance, and thus determine whether it is alive. Regarding the size and position information of the iris, key points can be estimated through DeepPose, and the exact size and position of the iris can be calculated based on the calculated key points. The principle of DeepPose algorithm is explained as follows:

[0171] The core idea of ​​the DeepPose algorithm is described as follows:

[0172] In the traditional field of posture recognition, traditional learning methods are still in the style of expert systems, using some knowledge systems of interactions between limbs to design more complex models such as feature representations of images or topology between joints.

[0173] The idea of ​​the DeepPose algorithm is to turn the key point detection algorithm into a purely mathematical prediction problem instead of considering the anthropomorphic problems under complex postures. By manually annotating a large amount of human key point data under various postures and learning the sample data through the DNN convolutional neural network, a more general end-to-end key point detection algorithm can be achieved.

[0174] Please refer to Figure 8 , which shows a schematic diagram of the DeepPose algorithm, such as Figure 8 As shown, after a series of convolutions are performed on the 220*220 image input, the final target position (xi, yi) is obtained through two fully connected layers. It should be noted that this algorithm is not limited to human body, but is also applicable to the iris area. It is a general key point prediction algorithm.

[0175] At the same time, the algorithm itself can be further optimized, for example, please refer to Fig. 9 , which shows another DeepPose algorithm schematic, such as Fig. 9 As shown in the figure, since the input target size is uncertain and the network itself only accepts 220*220 input, the overly large image will cause errors in the final target position prediction due to scaling. Based on the second stage of this algorithm, the idea of ​​the first stage is reused, but the area around the estimated point position is cropped and enlarged to continue further and more accurate prediction, thereby improving the final prediction accuracy.

[0176] The core idea of ​​the DeepPose algorithm is to fit the DNN network to the nonlinear regression problem and achieve end-to-end return network prediction results. For example, please refer to Fig.10 , which shows a schematic diagram of predicting eye key points through the DeepPose algorithm involved in this application, such as Fig.10 As shown, it shows the 19 key points of the eye area returned after prediction by DeepPose.

[0177] Among them, the hollow key points with dots are estimates of the eyelid key points, the solid key points are estimates of the points around the iris, and the hollow key points are estimates of the points around the pupil. Since the pupil itself is a circular area, and in the iris recognition scenario, the pupil will be guided to become smaller through environmental adjustment, and the imaging distortion is not large at small angles, so 4 key points are used for the pupil key points. The iris area is larger than the pupil, and the upper right is more easily blocked by the eyelid, so 8 key points are used for registration in the iris area. Similarly, since the upper part of the eyelid area has a larger curve than the lower part, the upper part has more key points and 6 key points are used for registration.

[0178] 3) Calculate the size of the pupil and iris area based on the pupil, iris, and eyelid key points to determine the user's attention point and thus determine whether the body is alive

[0179] As mentioned above Fig.10 It can be seen that the key point positions of the human eye can be obtained through the DeepPose algorithm. Please refer to Fig.11 , which shows the image presentation diagram of both eyes when gazing leftward and rightward involved in the present application.

[0180] Through key point detection, please refer to Fig.12 , which shows the effect diagram of key point detection involved in this application, such as Fig.12 As shown, the display of irrelevant key points is eliminated, and the key points of the two corner points of the eyelids and the key points of the pupil boundary remain. Fig.10 The displayed iris key points, since the pupil area and the iris area itself are circular areas, will become elliptical areas due to distortions such as angles. However, considering that the pupil area is small, the distortion has little effect, so too many registration points at the same time will lead to higher errors. Therefore, for the pupil area, the embodiment of the present application uses 4 key points of the pupil boundary to determine a circularly marked pupil area, that is, the above-mentioned pupil center point.

[0181] The point in the pupil center area can be roughly calculated through the four points in the pupil area. The calculation formula is:

[0182] x'=(min(x)+max(x)) / 2

[0183] y'=(min(y)+max(y)) / 2

[0184] Among them, min(x), min(y), max(x), and max(y) represent the minimum and maximum values ​​of the x and y coordinate positions of all the points in the four points.

[0185] Please refer to Fig.13 , which shows a schematic diagram of the connection between the eye corner and the pupil involved in this application, such as Fig.13 As shown in the figure, it can be seen that when looking left and right, the length ratio of the line connecting the two corners of the left and right eyes and the pupil is obviously different. For details, please refer to Fig.14 , which shows a schematic diagram of the verification process involved in this application. Fig.14 As shown, the verification logic can be as follows:

[0186] if(LR1>LR2&&LL2>LL1): left view;

[0187] if(LR1>LR2&&LL1>LL2): look right.

[0188] This is because when the human eye is guided to look left or right in a VR scene, due to the design of the guiding animation, the user cannot see the object by turning his head, and can only see the object by rotating the human eye. During the left and right rotation of the human eye, the relative position of the pupil relative to the eyelid corner will change, and the eyeball movement is judged based on this. In this way, it is determined whether the liveness verification is carried out in accordance with the direction requirements in the VR animation.

[0189] Here, the simplest size judgment is used for distance judgment, and multiple judgment factors such as ratio judgment can also be introduced. At the same time, this solution only describes the judgment about left and right. Similarly, it can realize the estimated judgment of various angles and positions. It only needs to use different distance judgment conditions for different scenarios.

[0190] Since this solution mainly relies on the DeepPose algorithm to estimate the key points of the iris, combined with animation guidance under VR, it determines the positional relationship between the pupil and the corner of the eye to determine whether the gaze is to the left or right, and thus determines whether the live body meets the requirements. This solution can achieve live body judgment in a VR scene while ensuring a natural user experience.

[0191] In some embodiments, a head-mounted display device is also provided, which includes a display screen and an image acquisition component; the image acquisition component points to the eyeball position when the head-mounted display device is worn; the display screen is used to display indication information, and the indication information is used to instruct the user to observe a specified position in the display space of the head-mounted display device; the image acquisition component is used to capture the user's eye image; the eye image is used to obtain the position of at least two key points of the user's eyes, so as to determine that the user has passed the verification when the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matches the specified position. Among them, the scheme for verifying the user through the head-mounted display device can refer to the above-mentioned embodiments.

[0192] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0193] Please refer to Fig.15 , which shows a block diagram of an image processing device for verification provided by an embodiment of the present application. The device may include the following modules.

[0194] The display module 1501 is used to display instruction information through a head mounted display device, where the instruction information is used to instruct a user to observe a designated position in a display space of the head mounted display device;

[0195] An image acquisition module 1502 is used to acquire an eye image of the user;

[0196] A key point acquisition module 1503, configured to acquire positions of at least two key points of the user's eyes from the eye image;

[0197] The verification module 1504 is used to determine that the user has passed the verification in response to the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matching the designated position.

[0198] In some embodiments, the head-mounted display device is a virtual reality (VR) display device, or the head-mounted display device is an augmented display (AR) display device.

[0199] In some embodiments, displaying the indication information through a head mounted display device includes:

[0200] The indication information is displayed at the designated location through the VR display device or the AR display device.

[0201] In some embodiments, the image acquisition module 1502 is used to acquire the eye image acquired by an image acquisition component provided in the head-mounted display device; the image acquisition component points to the position of the eyeball when the head-mounted display device is worn.

[0202] In some embodiments, the at least two key points of the user's eyes include at least one of the following:

[0203] at least two key points corresponding to the left eye of the user;

[0204] And, at least two key points corresponding to the right eye of the user.

[0205] In some embodiments, the at least two key points corresponding to the left eye of the user include a left corner key point of the left eye of the user, a right corner key point of the left eye of the user, and a pupil center point of the left eye of the user;

[0206] At least two key points corresponding to the right eye of the user include a left corner key point of the right eye of the user, a right corner key point of the right eye of the user, and a pupil center point of the right eye of the user;

[0207] The verification module 1504 is used to:

[0208] Obtain a first distance, a second distance, a third distance, and a fourth distance; wherein the first distance is the distance between the left eye corner key point of the left eye and the center point of the pupil of the left eye; the second distance is the distance between the right eye corner key point of the left eye and the center point of the pupil of the left eye; the third distance is the distance between the left eye corner key point of the right eye and the center point of the pupil of the right eye; the fourth distance is the distance between the right eye corner key point of the right eye and the center point of the pupil of the right eye;

[0209] In response to the position observed by the user indicated by the relationship among the first distance, the second distance, the third distance, and the fourth distance matching the designated position, it is determined that the user has been authenticated.

[0210] In some embodiments, the verification module 1504 is used to:

[0211] Determining a viewing direction of the user based on a magnitude relationship among the first distance, the second distance, the third distance, and the fourth distance;

[0212] In response to the viewing direction of the user matching the designated position, it is determined that the user has passed the authentication.

[0213] In some embodiments, the verification module 1504 is used to:

[0214] In response to the second distance being greater than the first distance, and the fourth distance being greater than the third distance, determining that the viewing direction of the user is leftward;

[0215] In response to the first distance being greater than the second distance, and the third distance being greater than the fourth distance, it is determined that the viewing direction of the user is rightward.

[0216] In some embodiments, the designated location moves over time; the verification module 1504 is used to,

[0217] The step of determining that the user has passed the verification in response to the position observed by the user indicated by the relationship among the first distance, the second distance, the third distance, and the fourth distance matching the designated position comprises:

[0218] Obtaining a change trend of a first ratio between the first distance and the second distance;

[0219] Acquire a change trend of a second ratio between the third distance and the fourth distance;

[0220] In response to the changing trend of the first ratio and the changing trend of the second ratio matching the moving direction of the designated position, it is determined that the user has passed the authentication.

[0221] In some embodiments, the key point acquisition module 1503 is used to input the eye image into a key point detection model to obtain the positions of at least two key points of the user's eyes output by the key point detection model;

[0222] Among them, the key point detection model is a machine learning model that is pre-trained through eye image samples and the positions of at least two key points marked by the eye image samples.

[0223] Fig.16 The following is a block diagram of a computer device 1600 provided by an exemplary embodiment of the present application. The computer device 1600 may be a terminal device, such as a personal computer, a VR device, an AR device, a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), or an MP4 player (Moving Picture Experts Group Audio Layer IV). The computer device 1600 may also be referred to as a user device, a portable terminal, or other names.

[0224] Typically, the computer device 1600 includes a processor 1601 and a memory 1602 .

[0225] The memory 1602 may include one or more computer-readable storage media, which may be tangible and non-transitory. The memory 1602 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1602 is used to store at least one instruction, which is used to be executed by the processor 1601 to implement all or part of the steps performed by the terminal in the method provided in the embodiment of the present application.

[0226] In some embodiments, the computer device 1600 may further include: a peripheral device interface 1603 and at least one peripheral device. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1604 , a touch display screen 1605 , a camera 1606 , an audio circuit 1607 and a power supply 1608 .

[0227] In some embodiments, the computer device 1600 further includes one or more sensors 1609 , including but not limited to: an acceleration sensor 1610 , a gyroscope sensor 1611 , a pressure sensor 1612 , an optical sensor 1613 , and a proximity sensor 1614 .

[0228] Those skilled in the art will appreciate that the structure shown above does not limit the computer device 1600 , and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0229] In an exemplary embodiment, a computer-readable storage medium is also provided for storing at least one computer program, which is loaded and executed by a processor to implement all or part of the steps in the methods shown in the above embodiments. For example, the computer-readable storage medium can be a read-only memory, a random access memory, a read-only optical disk, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0230] In an exemplary embodiment, a computer program product is also provided, the computer program product includes a computer program, the computer program is stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes all or part of the steps in the methods shown in the above embodiments.

[0231] Before collecting relevant user data and during the process of collecting relevant user data, the present application can display a prompt interface, pop-up window or output voice prompt information. The prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that the present application starts to execute the relevant steps of obtaining relevant user data only after obtaining the user's confirmation operation on the prompt interface or pop-up window. Otherwise (that is, when the user's confirmation operation on the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant user data are terminated, that is, the user's relevant data is not obtained.

[0232] In other words, all user data collected by this application is collected with the user's consent and authorization, and the collection, use and processing of relevant user data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0233] The above user-related data includes information (including but not limited to user accounts), data (including but not limited to text data entered by users, stored text / image data, displayed text / image data, etc.) and signal data. For example, the user data involved in this application is obtained with full authorization.

[0234] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0235] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method for verification, characterized in that: The method comprises: Displaying instruction information through a head mounted display device, wherein the instruction information is used to instruct a user to observe a specified position in a display space of the head mounted display device; Acquiring an eye image of the user; Acquire positions of at least two key points of the user's eyes from the eye image; In response to the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matching the designated position, it is determined that the user has passed the authentication.

2. The method according to claim 1, characterized in that The head-mounted display device is a virtual reality (VR) display device, or the head-mounted display device is an augmented display (AR) display device.

3. The method according to claim 2, characterized in that The displaying of the indication information by the head mounted display device includes: The indication information is displayed at the designated location through the VR display device or the AR display device.

4. The method according to claim 1, characterized in that The acquiring the eye image of the user comprises: The eye image captured by an image capture component provided in the head-mounted display device is acquired; the image capture component points to the position of the eyeball when the head-mounted display device is worn.

5. The method according to claim 1, characterized in that The at least two key points of the user's eyes include at least one of the following: at least two key points corresponding to the left eye of the user; And, at least two key points corresponding to the right eye of the user.

6. The method according to claim 5, characterized in that At least two key points corresponding to the left eye of the user include a left corner key point of the left eye of the user, a right corner key point of the left eye of the user, and a pupil center point of the left eye of the user; At least two key points corresponding to the right eye of the user include a left corner key point of the right eye of the user, a right corner key point of the right eye of the user, and a pupil center point of the right eye of the user; The step of determining that the user has passed the verification, in response to the position indicated by the relationship between the positions of at least two key points of the user's eyes matching the designated position, comprises: Obtain a first distance, a second distance, a third distance, and a fourth distance; wherein the first distance is the distance between the left eye corner key point of the left eye and the center point of the pupil of the left eye; the second distance is the distance between the right eye corner key point of the left eye and the center point of the pupil of the left eye; the third distance is the distance between the left eye corner key point of the right eye and the center point of the pupil of the right eye; the fourth distance is the distance between the right eye corner key point of the right eye and the center point of the pupil of the right eye; In response to the position observed by the user indicated by the relationship among the first distance, the second distance, the third distance, and the fourth distance matching the designated position, it is determined that the user has been authenticated.

7. The method according to claim 6, characterized in that The step of determining that the user has passed the verification in response to the position observed by the user indicated by the relationship among the first distance, the second distance, the third distance, and the fourth distance matching the designated position comprises: Determining a viewing direction of the user based on a magnitude relationship among the first distance, the second distance, the third distance, and the fourth distance; In response to the viewing direction of the user matching the designated position, it is determined that the user has passed the authentication.

8. The method according to claim 7, characterized in that The determining the viewing direction of the user based on the magnitude relationship among the first distance, the second distance, the third distance, and the fourth distance includes: In response to the second distance being greater than the first distance, and the fourth distance being greater than the third distance, determining that the viewing direction of the user is leftward; In response to the first distance being greater than the second distance, and the third distance being greater than the fourth distance, it is determined that the viewing direction of the user is rightward.

9. The method according to claim 6, characterized in that The designated position moves over time; The step of determining that the user has passed the verification in response to the position observed by the user indicated by the relationship among the first distance, the second distance, the third distance, and the fourth distance matching the designated position comprises: Obtaining a change trend of a first ratio between the first distance and the second distance; Acquire a change trend of a second ratio between the third distance and the fourth distance; In response to the changing trend of the first ratio and the changing trend of the second ratio matching the moving direction of the designated position, it is determined that the user has passed the authentication.

10. The method according to claim 1, characterized in that The acquiring positions of at least two key points of the user's eyes from the eye image includes: Inputting the eye image into a key point detection model to obtain positions of at least two key points of the user's eyes output by the key point detection model; Among them, the key point detection model is a machine learning model that is pre-trained through eye image samples and the positions of at least two key points marked by the eye image samples.

11. A head mounted display device, characterized in that: The head mounted display device comprises a display screen and an image acquisition component; the image acquisition component points to the eyeball position when the head mounted display device is worn; The display screen is used to display instruction information, where the instruction information is used to instruct the user to observe a specified position in the display space of the head mounted display device; The image acquisition component is used to acquire the eye image of the user; The eye image is used to obtain the positions of at least two key points of the user's eyes, so as to determine that the user has passed the verification when the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matches the specified position.

12. An image processing device for verification, characterized in that: The device comprises: A display module, used for displaying instruction information through a head mounted display device, wherein the instruction information is used for instructing a user to observe a designated position in a display space of the head mounted display device; An image acquisition module, used to acquire an eye image of the user; A key point acquisition module, used to acquire positions of at least two key points of the user's eyes from the eye image; The verification module is used to determine that the user has passed the verification in response to the position observed by the user indicated by the relationship between the positions of at least two key points of the user's eyes matching the designated position.

13. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the image processing method for verification as claimed in any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the computer program is loaded and executed by a processor to implement the image processing method for verification as claimed in any one of claims 1 to 10.

15. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; the computer instructions are read and executed by a processor of a computer device to implement the image processing method for verification as described in any one of claims 1 to 10.

Images