Pupil detection method and related products

By using the human eye image template library for image repair in pupil detection technology, the problem of low detection efficiency and accuracy caused by occlusion of the pupil is solved, and efficient pupil detection at different angles and situations is achieved.

CN112989878BActive Publication Date: 2024-11-29GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN201911283540.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-12-13
Publication Date
2024-11-29
Estimated Expiration
2039-12-13

AI Technical Summary

Technical Problem

The existing pupil detection technology allows the pupil to be blocked when the user's eyelids are drooping, looking down or wearing glasses, resulting in a decrease in pupil detection efficiency and accuracy, and the location of the pupil center cannot be effectively identified.

Method used

By obtaining the eye image of the obstructed pupil area, determine the target human eye image template in the preset human eye image template library that matches the image, perform image repair, obtain a complete pupil area image, and based on the repaired image Perform pupil detection.

Benefits of technology

The efficiency and accuracy of pupil detection are improved, and the center position of the pupil can be effectively identified when the pupil is blocked, solving the problem of low pupil detection efficiency and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112989878B_ABST
    Figure CN112989878B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses a pupil detection method and related products, the method comprising the following steps: acquiring a first eye image; if the first eye image includes an occluded area, determining a target human eye image template in a preset human eye image template library that successfully matches the first eye image; repairing the first eye image according to the target human eye image template to obtain a second eye image; performing pupil detection based on the second eye image to obtain a pupil center position, so that when the pupil is occluded in the acquired first eye image, the occluded first eye image is repaired using the human eye image template, thereby solving the problem of occluded pupil and improving the efficiency and accuracy of pupil detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of pupil detection technology, and in particular to a pupil detection method and related products. Background Art

[0002] With the widespread popularization of electronic devices (such as mobile phones, tablet computers, etc.), electronic devices can support more and more applications and have more and more powerful functions. Electronic devices are developing in a diversified and personalized direction and have become an indispensable electronic product in users' lives.

[0003] Electronic devices can be used to implement pupil detection technology and apply pupil detection technology to actual interactive scenarios of electronic devices. For example, pupil detection can be performed through electronic devices to implement eye tracking, determine the focus point of the user on the screen of the electronic device, and then perform the next operation based on the focus point of the user on the screen. The current eye tracking system identifies the position of the pupil by capturing a frame of image, but in the actual process of capturing eye images, when the user's eyelids droop, or the user looks down, or when the user wears glasses, the pupil will be blocked in the eye images captured at some angles, resulting in the inability to identify the center position of the pupil, affecting the efficiency and accuracy of the eye tracking system. Summary of the invention

[0004] The embodiments of the present application provide a pupil detection method and related products, which can solve the problem of pupil occlusion and improve the efficiency and accuracy of pupil detection.

[0005] In a first aspect, an embodiment of the present application provides a pupil detection method, the method comprising the following steps:

[0006] acquiring a first ocular image;

[0007] If the first eye image includes an occluded area, determining a target human eye image template that successfully matches the first eye image in a preset human eye image template library, wherein the occluded area at least includes a portion of a pupil area;

[0008] Repairing the first eye image according to the target human eye image template to obtain a second eye image;

[0009] Pupil detection is performed based on the second eye image to obtain the pupil center position.

[0010] In a second aspect, an embodiment of the present application provides a pupil detection device, the device comprising:

[0011] An acquisition unit, configured to acquire a first eye image;

[0012] a determination unit, configured to determine a target human eye image template that successfully matches the first eye image in a preset human eye image template library if the first eye image includes an obscured area, wherein the obscured area at least includes a portion of a pupil area;

[0013] a restoration unit, configured to restore the first eye image according to the target human eye image template to obtain a second eye image;

[0014] A detection unit is used to perform pupil detection based on the second eye image to obtain a pupil center position.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for executing the steps in the first aspect of the embodiment of the present application.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps described in the first aspect of the embodiment of the present application.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0018] The implementation of the embodiments of the present application has the following beneficial effects:

[0019] It can be seen that the pupil detection method and related products provided in the embodiments of the present application obtain a first eye image; if the first eye image includes an occluded area, determine a target human eye image template in a preset human eye image template library that successfully matches the first eye image; repair the first eye image according to the target human eye image template to obtain a second eye image; perform pupil detection based on the second eye image to obtain the center position of the pupil. In this way, when the pupil is occluded in the collected first eye image, the occluded first eye image is repaired by using the human eye image template, thereby solving the problem of occluded pupil and improving the efficiency and accuracy of pupil detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1A It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0022] Figure 1B is a schematic diagram showing a demonstration of running an eye tracking application in an electronic device in an eye tracking scenario provided by an embodiment of the present application;

[0023] Figure 1C It is a flowchart of a pupil detection method provided in an embodiment of the present application;

[0024] Figure 1D It is a schematic diagram of a scene of capturing an image when pupil detection is performed by a camera, provided in an embodiment of the present application;

[0025] Figure 1E is a schematic diagram for demonstrating a first eye image in which a pupil area is completely blocked, provided in an embodiment of the present application;

[0026] Figure 1F is a schematic diagram for demonstrating a first eye image in which a pupil area is partially blocked, provided in an embodiment of the present application;

[0027] Figure 1G It is a schematic diagram of demonstrating that each template pixel point of the template pupil area is used to replace the regional pixel point corresponding to the template pixel point in the occluded area provided by the embodiment of the present application;

[0028] Figure 1H is a demonstration schematic diagram of repairing a plurality of first region pixel points according to a plurality of first template pixel points provided in an embodiment of the present application;

[0029] Fig. 1I is another demonstration schematic diagram of repairing a plurality of first region pixels according to a plurality of first template pixels provided in an embodiment of the present application;

[0030] Figure 2 is a flow chart of another pupil detection method provided in an embodiment of the present application;

[0031] Figure 3A It is a schematic diagram of a process for determining a gaze point in an eye tracking scenario provided by an embodiment of the present application;

[0032] Figure 3BIt is a schematic diagram of a flow chart of pupil detection in an eye tracking scenario provided in an embodiment of the present application;

[0033] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0034] Figure 5 It is a structural schematic diagram of a pupil detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0037] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0038] The electronic devices involved in the embodiments of the present application may include various handheld devices with wireless communication functions, vehicle-mounted devices, wearable devices (smart watches, smart bracelets, wireless headphones, augmented reality / virtual reality devices, smart glasses), computing devices or other processing devices connected to wireless modems, and various forms of user equipment (UE), mobile stations (MS), terminal devices, etc. For the convenience of description, the above-mentioned devices are collectively referred to as electronic devices.

[0039] The following is a detailed description of the embodiments of the present application.

[0040] See also Figure 1A , Figure 1A 1 is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. The electronic device 100 includes a storage and processing circuit 110 and a sensor 170 connected to the storage and processing circuit 110, wherein:

[0041] The electronic device 100 may include a control circuit, which may include a storage and processing circuit 110. The storage and processing circuit 110 may include a memory, such as a hard disk drive memory, a non-volatile memory (such as a flash memory or other electronic programmable read-only memory for forming a solid-state drive, etc.), a volatile memory (such as a static or dynamic random access memory, etc.), etc., which is not limited by the embodiments of the present application. The processing circuit in the storage and processing circuit 110 can be used to control the operation of the electronic device 100. The processing circuit can be implemented based on one or more microprocessors, microcontrollers, digital signal processors, baseband processors, power management units, audio codec chips, application-specific integrated circuits, display driver integrated circuits, etc.

[0042] The storage and processing circuit 110 may be used to run software in the electronic device 100, such as an Internet browsing application, a Voice over Internet Protocol (VOIP) phone call application, an email application, a media playback application, an operating system function, etc. These software may be used to perform some control operations, such as image acquisition based on a camera, ambient light measurement based on an ambient light sensor, proximity sensor measurement based on a proximity sensor, information display functions implemented based on status indicators such as status indicators of light-emitting diodes, touch event detection based on a touch sensor, functions associated with displaying information on multiple (e.g., layered) display screens, operations associated with performing wireless communication functions, operations associated with collecting and generating audio signals, control operations associated with collecting and processing button press event data, and other functions in the electronic device 100, etc., which are not limited by the embodiments of the present application.

[0043] The electronic device 100 may include an input-output circuit 150. The input-output circuit 150 may be used to enable the electronic device 100 to input and output data, that is, to allow the electronic device 100 to receive data from an external device and also allow the electronic device 100 to output data from the electronic device 100 to an external device. The input-output circuit 150 may further include a sensor 170. The sensor 170 may include an ultrasonic fingerprint recognition module, and may also include an ambient light sensor, a proximity sensor based on light and capacitance, a touch sensor (for example, based on a light touch sensor and / or a capacitive touch sensor, wherein the touch sensor may be part of a touch display screen, or may be used independently as a touch sensor structure), an acceleration sensor, and other sensors, etc. The ultrasonic fingerprint recognition module may be integrated under the screen, or the ultrasonic fingerprint recognition module may be disposed on the side or back of the electronic device, which is not limited here, and the ultrasonic fingerprint recognition module may be used to collect fingerprint images.

[0044] Sensor 170 may include an infrared (IR) camera or an RGB camera. When the IR camera is shooting, the pupil reflects infrared light, so the IR camera is more accurate than the RGB camera in capturing pupil images. The RGB camera requires more subsequent pupil detection, and has higher calculation precision and accuracy than the IR camera, and better versatility than the IR camera, but the amount of calculation is large.

[0045] The input-output circuit 150 may also include one or more display screens, such as the display screen 130. The display screen 130 may include a liquid crystal display screen, an organic light-emitting diode display screen, an electronic ink display screen, a plasma display screen, a display screen using other display technologies, or a combination of several of them. The display screen 130 may include a touch sensor array (i.e., the display screen 130 may be a touch-controlled display screen). The touch sensor may be a capacitive touch sensor formed by an array of transparent touch sensor electrodes (e.g., indium tin oxide (ITO) electrodes), or may be a touch sensor formed using other touch technologies, such as sonic touch, pressure-sensitive touch, resistive touch, optical touch, etc., which is not limited in the embodiments of the present application.

[0046] The electronic device 100 may further include an audio component 140. The audio component 140 may be used to provide audio input and output functions for the electronic device 100. The audio component 140 in the electronic device 100 may include a speaker, a microphone, a buzzer, a tone generator, and other components for generating and detecting sound.

[0047] The communication circuit 120 can be used to provide the electronic device 100 with the ability to communicate with external devices. The communication circuit 120 may include analog and digital input-output interface circuits, and wireless communication circuits based on radio frequency signals and / or optical signals. The wireless communication circuit in the communication circuit 120 may include a radio frequency transceiver circuit, a power amplifier circuit, a low noise amplifier, a switch, a filter, and an antenna. For example, the wireless communication circuit in the communication circuit 120 may include a circuit for supporting near field communication (NFC) by transmitting and receiving near field coupled electromagnetic signals. For example, the communication circuit 120 may include a near field communication antenna and a near field communication transceiver. The communication circuit 120 may also include a cellular phone transceiver and antenna, a wireless local area network transceiver circuit and antenna, etc.

[0048] The electronic device 100 may further include a battery, power management circuits and other input-output units 160. The input-output units 160 may include buttons, joysticks, click wheels, scroll wheels, touch pads, keypads, keyboards, cameras, light emitting diodes and other status indicators, etc.

[0049] A user may input commands through input-output circuitry 150 to control the operation of electronic device 100 , and may use output data of input-output circuitry 150 to enable receiving status information and other outputs from electronic device 100 .

[0050] In the eye tracking scenario, the electronic device can run the eye tracking application, see Figure 1B , Figure 1BA schematic diagram of a demonstration of running an eye tracking application in an electronic device in an eye tracking scenario provided by an embodiment of the present application, wherein the memory in the storage and processing circuit 110 can store multiple applications, such as e-books, browsers, payment applications, and system applications, and one or more microprocessors in the storage and processing circuit 110 can also be used to run installed applications to implement various functions, such as an unlocking function, and another example is a function of tracking the user's eye gaze point. To implement the gaze point tracking function, eye tracking is required. The eye gaze point is the gaze point position of the user's eye on the plane where the electronic device is located. The eye tracking software development kit interface is a software development kit (SDK) interface provided by the electronic device for the eye tracking application, which is responsible for providing the eye tracking application with an application programming interface (API) interface for obtaining the gaze point and inputting. The eye tracking service is responsible for managing the gaze point algorithm, determining the gaze point post-processing, input processing, authentication and parameter setting. The eye tracking core algorithm may include a calibration algorithm and an estimated gaze point algorithm. The eye tracking strategy is related to the gaze point algorithm post-processing. It mainly includes applications for filtering gaze point jumps, gaze point conversion, and monitoring gaze point input. Eye tracking authentication is used to call back various modules and is responsible for whether the authentication requester is allowed. The parameter setting module is used to parse the configuration and update the configuration in real time. The pupil repair module is used to create a human eye image template library and repair the first eye image collected. Among them, the eye tracking service can also call the camera application through the camera Native Development Kit (NDK) interface. The camera application will call the camera to collect the first eye image through the camera.

[0051] See also Figure 1C , Figure 1C is a flow chart of a pupil detection method provided in an embodiment of the present application, which is applied to Figure 1A The electronic equipment shown, such as Figure 1C As shown, the pupil detection method provided by the present application includes:

[0052] 101. Obtain a first eye image.

[0053] The first eye image is an image containing the user's eye area.

[0054] In the embodiment of the present application, the first eye image can be obtained by taking the first eye image through a camera provided on the electronic device. The acquired first eye image can be used for eye tracking. In the eye tracking scenario, the electronic device can collect the first eye image through the camera to perform pupil detection during the eye tracking process to detect the pupil center position of the pupil in the first eye image, thereby further determining the gaze point of the user focusing on the electronic device screen based on the pupil center position, and then controlling the operation of the electronic device based on the gaze point. Please refer to Figure 1D , Figure 1D A schematic diagram of a scene in which a first eye image is captured by a camera in an embodiment of the present application. In a specific shooting scene, when the user's eyelids droop, or the user looks downward, or the user wears glasses, the pupil may be blocked in the first eye image captured at some angles. If the pupil area is blocked in the captured first eye image, the center position of the pupil cannot be directly obtained when pupil detection is performed on the first eye image. Therefore, the first eye image can be repaired.

[0055] 102. If the first eye image includes an occluded area, determine a target human eye image template that successfully matches the first eye image in a preset human eye image template library, wherein the occluded area at least includes a portion of a pupil area.

[0056] The blocked area includes at least part of the pupil area, which may specifically include the following situations: the pupil area in the first eye image is completely blocked, but other eye areas are not completely blocked, or part of the pupil area in the first eye image is blocked, but another part of the pupil area is not blocked, and other eye areas are not completely blocked. Figure 1E , Figure 1E For a demonstration diagram of a first eye image in which the pupil area is completely blocked provided in an embodiment of the present application, please refer to Figure 1F , Figure 1F A schematic diagram illustrating a first eye image in which a portion of the pupil area is blocked, provided in an embodiment of the present application.

[0057] In a specific implementation, the electronic device may pre-set a human eye image template library, which may include multiple human eye image templates collected at different angles and distances, and the human eye area of ​​each human eye image template is not blocked. Therefore, if the first eye image is a partially blocked eye image, a target human eye image template that successfully matches the first eye image may be determined in the preset human eye image template library.

[0058] Optionally, in the above step 102, determining a target human eye image template in a preset human eye image template library that successfully matches the first eye image may include the following steps:

[0059] 21. Match the first eye image with a plurality of human eye image templates in the preset human eye image template library to obtain a plurality of matching values;

[0060] 22. If there is a target matching value greater than a preset matching value among the multiple matching values, determine that the human eye image template corresponding to the target matching value successfully matches the first eye image, and determine the human eye image template corresponding to the target matching value as the target human eye image template.

[0061] Among them, the first eye image is matched with multiple human eye image templates in the preset human eye image template library. First, the first shooting angle and the first shooting distance of the camera shooting the first eye image can be obtained, and then the first shooting angle is matched with the shooting angle corresponding to each human eye image template in the multiple human eye image templates to obtain a first matching score, and the first shooting distance is matched with the shooting distance corresponding to each human eye image template in the multiple human eye image templates to obtain a second matching score, and then the sum of the first matching score and the second matching score is determined to obtain a matching value corresponding to the human eye image template, so that multiple matching values ​​of the multiple human eye image templates matching the first eye image can be determined, and each human eye image template corresponds to a matching value. Finally, the target human eye image template corresponding to the target matching value greater than the preset matching value can be determined in the multiple matching values. In this way, the target human eye image template with the shooting angle closest to the first shooting angle and the shooting distance closest to the first shooting distance can be determined.

[0062] Optionally, the first eye image is matched with multiple human eye image templates in the preset human eye image template library, and the first eye image can also be subjected to feature extraction to obtain a first human eye feature set, and then the first feature set is matched with the human eye feature set corresponding to each human eye image template in the multiple human eye image templates to obtain a matching value corresponding to the human eye image template, thereby, multiple matching values ​​for matching the multiple human eye image templates with the first eye image can be determined, and each human eye image template corresponds to a matching value, and finally, a target human eye image template corresponding to a target matching value greater than the preset matching value among the multiple matching values ​​can be determined, in this way, the target human eye image template whose human eye features are closest to the first human eye feature set can be determined.

[0063] Optionally, in the embodiment of the present application, the following steps may also be included:

[0064] 1021. Perform feature extraction on the first eye image to obtain a plurality of eye image features, wherein the plurality of eye image features include a plurality of eye contour features;

[0065] 1022. Predicting a pupil reference area according to the multiple eye contour features;

[0066] 1023. Match some eye image features in the multiple eye image features that are in the pupil reference area with preset pupil image features to obtain a matching result. If the matching result does not meet a preset condition, determine that the first eye image includes an occluded area.

[0067] In the embodiment of the present application, after acquiring the first eye image, feature extraction may be performed on the first eye image to obtain a plurality of eye image features. Since the eye image features presented in different areas of the first eye image are different, the pupil reference area may be predicted based on the plurality of eye contour features. Then, the partial eye image features in the pupil reference area of ​​the plurality of eye image features may be matched with the preset pupil image features. The preset pupil image features may be pre-stored in the electronic device. Thus, the partial eye image features in the predicted pupil reference area may be matched with the preset pupil image features to obtain a matching result. The result is specifically, assuming that the predicted partial eye image features of the pupil reference area include m eye image features, m is an integer greater than 1, each of the m eye image features can be matched with the preset pupil image feature to obtain m feature matching values, and the m feature matching values ​​are used as the matching results, if there are n feature matching values ​​less than or equal to the preset matching value among the m feature matching values, and the ratio of n to m is greater than the preset ratio, it is determined that the first eye image includes an occluded area, wherein n is a positive integer less than m, and further, it can be determined that the first eye image is a partially occluded eye image.

[0068] 103. Repair the first eye image according to the target human eye image template to obtain a second eye image.

[0069] Among them, taking into account the situation that the pupil area is occluded in the first eye image, since the target human eye image template is the human eye image template closest to the first eye image, the first eye image can be repaired according to the target human eye image template, thereby obtaining a second eye image including a complete pupil area.

[0070] Optionally, in the above step 103, repairing the first eye image according to the target human eye image template to obtain the second eye image may include the following steps:

[0071] 31. Determine the blocked area;

[0072] 32. Determine a template pupil area corresponding to the blocked area in the target human eye image template;

[0073] 33. Repair the blocked area according to the template pupil area to obtain the second eye image.

[0074] Among them, image analysis can be performed on the first eye image to determine that the area corresponding to the set of pixel points in the first eye image that is inconsistent with the preset eye pixel points is the occluded area. The preset eye pixel points can be preset pupil area pixel points. Alternatively, it can also be determined that the area corresponding to the set of pixel points in the first eye image that is inconsistent with the pixel point information corresponding to the target human eye image template is the occluded area.

[0075] Among them, to determine the template pupil area in the target human eye image template, pupil features can be extracted from the target human eye image template to obtain pupil feature points, and an area of ​​a preset range size containing the pupil feature points is determined as the template pupil area based on the pupil feature points.

[0076] The blocked area is repaired according to the template pupil area to obtain the second eye image, and the pixel points of the template pupil area can replace the corresponding pixel points in the first eye image to obtain the second eye image.

[0077] Optionally, in the above step 33, repairing the blocked area according to the template pupil area to obtain the second eye image may include the following steps:

[0078] 3301. Use each template pixel point of the template pupil area to replace the regional pixel point corresponding to the template pixel point in the occluded area to obtain the second eye image.

[0079] See also Figure 1G , Figure 1G A demonstration schematic diagram of using each template pixel point of the template pupil area to replace the regional pixel point corresponding to the template pixel point in the occluded area provided in the embodiment of the present application, when the occluded area includes the entire occluded pupil area, or the occluded area includes the partially occluded pupil area, each template pixel point of the template pupil area can be used to replace the regional pixel point corresponding to the template pixel point in the occluded area, thereby directly using the template pupil area in the target human eye image template as the entire repaired pupil area in the first eye image to obtain the second eye image.

[0080] Optionally, if the blocked area includes a partially blocked pupil area, in the above step 33, repairing the blocked area according to the template pupil area to obtain the second eye image may include the following steps:

[0081] 3302. Determine a plurality of first region pixel points in the blocked region whose noise is greater than a preset threshold;

[0082] 3303. Determine a plurality of first template pixel points in the template pupil region that correspond one-to-one to the plurality of first region pixel points;

[0083] 3304. Repair the multiple first area pixel points according to the multiple first template pixel points to obtain the second eye image.

[0084] See also Figure 1H , Figure 1H A demonstration schematic diagram of repairing multiple first-area pixel points based on multiple first template pixel points is provided for an embodiment of the present application, wherein, if the user's pupil area is partially occluded, in order to repair the partially occluded pupil area in the first eye image, multiple first-area pixel points in the occluded area whose noise is greater than a preset threshold can be determined, and the multiple first-area pixel points are used as pixel points to be repaired, and then multiple first template pixel points in the template pupil area that correspond one-to-one to the multiple first-area pixel points are determined, and finally, the multiple first-area pixel points can be repaired according to the multiple first template pixel points to obtain the second eye image.

[0085] In one possible embodiment, the second region pixel point set is repaired according to the second template pixel point set. Specifically, each template pixel point in the second template pixel point set may replace a corresponding pixel point in the second region pixel point set to obtain a second eye image.

[0086] Optionally, in the above step 3304, repairing the plurality of first region pixels according to the plurality of first template pixels to obtain the second eye image may include the following steps:

[0087] 3341. Determine a plurality of pupil area pixels that are not blocked in the first eye image;

[0088] 3342. Determine a weighting coefficient according to the plurality of pupil area pixels;

[0089] 3343. Determine a plurality of target template pixel points according to the weighting coefficient and the plurality of first template pixel points, wherein the plurality of target template pixel points correspond one-to-one to the plurality of first region pixel points;

[0090] 3344. Replace the pixel point of the first area corresponding to the target template pixel point in the occluded area with each target template pixel point among the multiple target template pixel points to obtain the second eye image.

[0091] Among them, considering that when the user's pupil area is partially blocked, there is still a part of the pupil area that is not blocked, therefore, multiple unblocked pupil area pixels in the first eye image can be determined. Fig. 1I , Fig. 1IAnother demonstration schematic diagram of repairing multiple first region pixels according to multiple first template pixels provided in an embodiment of the present application, wherein the grayscale value of each pupil region pixel in the multiple pupil region pixels can be determined to obtain multiple pixel values, and then the average value of the multiple pixel values ​​is determined to obtain the pixel average value, and the electronic device can pre-store the mapping relationship between the average value and the weighting coefficient, so that the weighting coefficient corresponding to the pixel average value can be determined according to the mapping relationship, and then, the weighting coefficient is sequentially weighted with the pixel value of each first template pixel in the multiple first template pixels to obtain multiple target template pixels, and finally each target template pixel in the multiple target template pixels replaces the first region pixel corresponding to the target template pixel in the occluded area to obtain the second eye image, so that the occluded part of the pupil area can be repaired. In this way, the other occluded part of the pupil area can be repaired according to the unoccluded part of the pupil area, and the first eye image can be repaired more accurately to obtain the second eye image.

[0092] 104. Perform pupil detection based on the second eye image to obtain a pupil center position.

[0093] After the first eye image is repaired to obtain the second eye image including pupil information, pupil detection can be performed based on the second eye image to obtain the pupil center position. In a specific embodiment, pupil feature extraction can be performed on the second eye image to obtain the target pupil feature, and then the pupil center position in the second eye image is determined based on the target pupil feature point.

[0094] It can be seen that in the embodiment of the present application, a first eye image is obtained; if the first eye image includes an occluded area, a target human eye image template that successfully matches the first eye image is determined in a preset human eye image template library; the first eye image is repaired according to the target human eye image template to obtain a second eye image; pupil detection is performed based on the second eye image to obtain the center position of the pupil. In this way, when the pupil is occluded in the collected first eye image, the occluded first eye image is repaired by using the human eye image template, thereby solving the problem of occluded pupil and improving the efficiency and accuracy of pupil detection.

[0095] See also Figure 2 , Figure 2 A schematic flow chart of a pupil detection method provided in an embodiment of the present application, the method comprising:

[0096] 201. Obtain a first eye image.

[0097] 202. If the first eye image includes an occluded area, determine a target human eye image template that successfully matches the first eye image in a preset human eye image template library, wherein the occluded area at least includes a portion of a pupil area.

[0098] 203. Determine the blocked area.

[0099] 204. Determine a template pupil area corresponding to the blocked area in the target human eye image template.

[0100] 205. Repair the blocked area according to the template pupil area to obtain the second eye image.

[0101] 206. Perform pupil detection based on the second eye image to obtain a pupil center position.

[0102] The specific implementation process of the above steps 201-206 can refer to the corresponding description in steps 101-104, which will not be repeated here.

[0103] It can be seen that in the embodiment of the present application, a first eye image is acquired; if the first eye image includes an occluded area, a target human eye image template that successfully matches the first eye image is determined in a preset human eye image template library; the occluded area is determined; a template pupil area corresponding to the occluded area in the target human eye image template is determined; the occluded area is repaired according to the template pupil area to obtain a second eye image; pupil detection is performed based on the second eye image to obtain the center position of the pupil. In this way, when the pupil is occluded in the acquired first eye image, the occluded first eye image is repaired by using the human eye image template, thereby solving the problem of pupil occlusion and improving the efficiency and accuracy of pupil detection.

[0104] See also Figure 3A-3B , Figure 3A A schematic diagram of a process for determining a gaze point in an eye tracking scenario provided in an embodiment of the present application, Figure 3B A schematic diagram of a process for pupil detection in an eye tracking scenario provided in an embodiment of the present application, the method comprising:

[0105] Among them, Figure 1B consistent, such as Figure 3AAs shown, when an electronic device runs an eye tracking application, it can request the eye tracking service to obtain the eye tracking gaze point position, and the eye tracking service can call the camera application. Specifically, the eye tracking service can request the camera application to obtain the first eye image, so that the camera application will turn on the camera on the electronic device and collect the first eye image through the camera. The camera will transmit the collected first eye image to the camera application, and then the camera application will transmit the first eye image to the eye tracking service; then, the eye tracking service can call the eye tracking core algorithm stored in the memory, and if the pupil is blocked, the first eye image can be repaired to obtain a second eye image with complete pupil information, and then, eye tracking can be performed on the second eye image to obtain the eye gaze point position, and finally, the eye tracking service can transmit the eye gaze point position to the eye tracking application.

[0106] like Figure 3B As shown, after acquiring a first eye image through a camera, feature extraction can be performed on the first eye image to obtain a plurality of eye image features, wherein the plurality of eye image features include a plurality of eye contour features; a pupil reference area is predicted based on the plurality of eye contour features; part of the eye image features in the plurality of eye image features that are in the pupil reference area are matched with preset pupil image features to obtain a matching result; if the matching result does not meet a preset condition, it is determined that the first eye image includes an occluded area, a target human eye image template that successfully matches the first eye image is determined in a preset human eye image template library, the occluded area in the first eye image is determined, and a template pupil area corresponding to the occluded area in the target human eye image template is determined; if the occluded area Including the entire pupil area, using each template pixel point of the template pupil area to replace the regional pixel point corresponding to the template pixel point in the occluded area to obtain the second eye image, if the occluded area includes the occluded part of the pupil area, determine a plurality of first regional pixel points in the occluded area whose noise is greater than a preset threshold; determine a plurality of first template pixel points in the template pupil area corresponding one-to-one to the plurality of first regional pixel points; repair the plurality of first regional pixel points according to the plurality of first template pixel points to obtain the second eye image, perform pupil detection based on the second eye image to obtain the pupil center position, and finally determine the eye gaze point position according to the pupil center position, thereby realizing the determination of the eye gaze point position in the eye tracking scenario.

[0107] It can be seen that in the embodiment of the present application, a first eye image is obtained; if the first eye image includes an occluded area, a target human eye image template that successfully matches the first eye image is determined in a preset human eye image template library; if the occluded area includes the entire pupil area, each template pixel point of the template pupil area is used to replace the regional pixel point corresponding to the template pixel point in the occluded area to obtain a second eye image, and if the occluded area includes an occluded partial pupil area, a plurality of first regional pixel points whose noise in the occluded area is greater than a preset threshold are determined; a plurality of first template pixel points in the template pupil area corresponding one-to-one to the plurality of first regional pixel points are determined; the plurality of first regional pixel points are repaired according to the plurality of first template pixel points to obtain a second eye image; pupil detection is performed based on the second eye image to obtain the pupil center position, so that when the pupil is occluded in the acquired first eye image, the occluded first eye image is repaired by the human eye image template, thereby solving the problem of pupil occlusion and improving the efficiency and accuracy of pupil detection.

[0108] It can be seen that through the above scheme, the position of the pupil of the human eye can be detected in different situations, thereby improving the efficiency of pupil detection and improving the limitations of pupil detection in the prior art.

[0109] The following is a device for implementing the above pupil detection method, which is as follows:

[0110] Consistent with the above, see Figure 4 , Figure 4 4 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, the electronic device includes: a processor 410, a communication interface 430 and a memory 420; and one or more programs 421, the one or more programs 421 are stored in the memory 420 and are configured to be executed by the processor, and the program 421 includes instructions for executing the following steps:

[0111] acquiring a first ocular image;

[0112] If the first eye image includes an occluded area, determining a target human eye image template that successfully matches the first eye image in a preset human eye image template library, wherein the occluded area at least includes a portion of a pupil area;

[0113] Repairing the first eye image according to the target human eye image template to obtain a second eye image;

[0114] Pupil detection is performed based on the second eye image to obtain the pupil center position.

[0115] In a possible example, in determining the target human eye image template that successfully matches the first eye image in the preset human eye image template library, the program 421 includes instructions for performing the following steps:

[0116] Matching the first eye image with a plurality of human eye image templates in the preset human eye image template library to obtain a plurality of matching values;

[0117] If there is a target matching value greater than the preset matching value among the multiple matching values, it is determined that the human eye image template corresponding to the target matching value successfully matches the first eye image, and the human eye image template corresponding to the target matching value is determined as the target human eye image template.

[0118] In a possible example, in the aspect of repairing the first eye image according to the target human eye image template to obtain the second eye image, the program 421 includes instructions for executing the following steps:

[0119] Determining the blocked area;

[0120] Determine a template pupil area corresponding to the blocked area in the target human eye image template;

[0121] The blocked area is repaired according to the template pupil area to obtain the second eye image.

[0122] In a possible example, in the aspect of repairing the blocked area according to the template pupil area to obtain the second eye image, the program 421 includes instructions for performing the following steps:

[0123] Each template pixel point of the template pupil area is used to replace the regional pixel point in the blocked area corresponding to the template pixel point to obtain the second eye image.

[0124] In a possible example, if the occluded area includes an occluded portion of the pupil area, in terms of repairing the occluded area according to the template pupil area to obtain the second eye image, the program 421 includes instructions for performing the following steps:

[0125] Determine a plurality of first region pixel points in the blocked region whose noise is greater than a preset threshold;

[0126] Determine a plurality of first template pixel points in the template pupil region corresponding one-to-one to the plurality of first region pixel points;

[0127] The plurality of first region pixel points are repaired according to the plurality of first template pixel points to obtain the second eye image.

[0128] In a possible example, in the aspect of repairing the plurality of first region pixels according to the plurality of first template pixels to obtain the second eye image, the program 421 includes instructions for performing the following steps:

[0129] Determine a plurality of pupil area pixels that are not blocked in the first eye image;

[0130] Determining a weighting coefficient according to the plurality of pupil area pixels;

[0131] Determine a plurality of target template pixel points according to the weighting coefficients and the plurality of first template pixel points, wherein the plurality of target template pixel points correspond one-to-one to the plurality of first region pixel points;

[0132] Each target template pixel point among the multiple target template pixel points replaces the first area pixel point in the blocked area corresponding to the target template pixel point to obtain the second eye image.

[0133] In a possible example, the program 421 further includes instructions for executing the following steps:

[0134] Performing feature extraction on the first eye image to obtain a plurality of eye image features, wherein the plurality of eye image features include a plurality of eye contour features;

[0135] predicting a pupil reference area according to the plurality of eye contour features;

[0136] Part of the eye image features in the pupil reference area among the multiple eye image features are matched with preset pupil image features to obtain a matching result. If the matching result does not meet a preset condition, it is determined that the first eye image includes an occluded area.

[0137] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a pupil detection device provided in this embodiment. The pupil detection device 500 is applied to an electronic device, and the electronic device includes an acceleration sensor and a camera. The device 500 includes an acquisition unit 501, a determination unit 502, a repair unit 503 and a detection unit 504, wherein:

[0138] The acquisition unit 501 is used to acquire a first eye image;

[0139] The determining unit 502 is configured to determine a target human eye image template that successfully matches the first eye image in a preset human eye image template library if the first eye image includes an occluded area, wherein the occluded area at least includes a portion of a pupil area;

[0140] The restoration unit 503 is used to restore the first eye image according to the target human eye image template to obtain a second eye image;

[0141] The detection unit 504 is used to perform pupil detection based on the second eye image to obtain the pupil center position.

[0142] Optionally, in determining the target human eye image template that successfully matches the first eye image in the preset human eye image template library, the determining unit 502 is specifically configured to:

[0143] Matching the first eye image with a plurality of human eye image templates in the preset human eye image template library to obtain a plurality of matching values;

[0144] If there is a target matching value greater than the preset matching value among the multiple matching values, it is determined that the human eye image template corresponding to the target matching value successfully matches the first eye image, and the human eye image template corresponding to the target matching value is determined as the target human eye image template.

[0145] Optionally, in the aspect of repairing the first eye image according to the target human eye image template to obtain the second eye image, the repairing unit 503 is specifically used to:

[0146] Determining the blocked area;

[0147] Determine a template pupil area corresponding to the blocked area in the target human eye image template;

[0148] The blocked area is repaired according to the template pupil area to obtain the second eye image.

[0149] Optionally, in the aspect of repairing the blocked area according to the template pupil area to obtain the second eye image, the repairing unit 503 is specifically used for:

[0150] Each template pixel point of the template pupil area is used to replace the regional pixel point in the blocked area corresponding to the template pixel point to obtain the second eye image.

[0151] Optionally, if the blocked area includes a partially blocked pupil area, in the aspect of repairing the blocked area according to the template pupil area to obtain the second eye image, the repairing unit 503 is specifically used to:

[0152] Determine a plurality of first region pixel points in the blocked region whose noise is greater than a preset threshold;

[0153] Determine a plurality of first template pixel points in the template pupil region corresponding one-to-one to the plurality of first region pixel points;

[0154] The plurality of first region pixel points are repaired according to the plurality of first template pixel points to obtain the second eye image.

[0155] Optionally, in the aspect of repairing the plurality of first region pixels according to the plurality of first template pixels to obtain the second eye image, the detecting unit 503 is specifically configured to:

[0156] Determine a plurality of pupil area pixels that are not blocked in the first eye image;

[0157] Determining a weighting coefficient according to the plurality of pupil area pixels;

[0158] Determine a plurality of target template pixel points according to the weighting coefficients and the plurality of first template pixel points, wherein the plurality of target template pixel points correspond one-to-one to the plurality of first region pixel points;

[0159] Each target template pixel point among the multiple target template pixel points replaces the first area pixel point in the blocked area corresponding to the target template pixel point to obtain the second eye image.

[0160] Optionally, the detection unit 504 is further configured to extract features from the first eye image to obtain a plurality of eye image features, wherein the plurality of eye image features include a plurality of eye contour features; predict a pupil reference area according to the plurality of eye contour features; match a portion of the eye image features in the plurality of eye image features that are in the pupil reference area with preset pupil image features to obtain a matching result, and if the matching result does not meet a preset condition, determine that the first eye image includes an occluded area. .

[0161] It can be seen that the pupil detection device described in the embodiment of the present application obtains a first eye image; if the first eye image includes an occluded area, determines a target human eye image template in a preset human eye image template library that successfully matches the first eye image; repairs the first eye image according to the target human eye image template to obtain a second eye image; performs pupil detection based on the second eye image to obtain the center position of the pupil. In this way, when the pupil is occluded in the captured first eye image, the occluded first eye image is repaired by using the human eye image template, thereby solving the problem of occluded pupil and improving the efficiency and accuracy of pupil detection.

[0162] It can be understood that the functions of each program module of the pupil detection device of this embodiment can be specifically implemented according to the method in the above method embodiment, and its specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here.

[0163] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps of any method described in the above method embodiments, and the above computer includes an electronic device.

[0164] The embodiment of the present application also provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, the computer program is operable to cause a computer to execute some or all of the steps of any method described in the method embodiment. The computer program product may be a software installation package, and the computer includes an electronic device.

[0165] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0166] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0167] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0168] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0170] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.

[0171] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.

[0172] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A pupil detection method, characterized in that: The method comprises: When the user's eyelids droop or the user looks downward, acquiring a first eye image; If the first eye image includes an occluded area, determining a target human eye image template that successfully matches the first eye image in a preset human eye image template library, wherein the occluded area at least includes a portion of a pupil area; the human eye image template library includes a plurality of human eye image templates collected at different angles and distances, and the human eye area of ​​each human eye image template is not occluded; Repairing the first eye image according to the target human eye image template to obtain a second eye image; Perform pupil detection based on the second eye image to obtain a pupil center position; The step of repairing the first eye image according to the target human eye image template to obtain a second eye image includes: Determining the blocked area; Determine a template pupil area corresponding to the blocked area in the target human eye image template; Repairing the blocked area according to the template pupil area to obtain the second eye image; If the blocked area includes a partially blocked pupil area, repairing the blocked area according to the template pupil area to obtain the second eye image includes: Determine a plurality of first region pixel points in the blocked region whose noise is greater than a preset threshold; Determine a plurality of first template pixel points in the template pupil region corresponding one-to-one to the plurality of first region pixel points; The plurality of first region pixel points are repaired according to the plurality of first template pixel points to obtain the second eye image.

2. The method according to claim 1, characterized in that The step of determining a target human eye image template in a preset human eye image template library that successfully matches the first eye image comprises: Matching the first eye image with a plurality of human eye image templates in the preset human eye image template library to obtain a plurality of matching values; If there is a target matching value greater than the preset matching value among the multiple matching values, it is determined that the human eye image template corresponding to the target matching value successfully matches the first eye image, and the human eye image template corresponding to the target matching value is determined as the target human eye image template.

3. The method according to claim 1, characterized in that The repairing of the plurality of first region pixels according to the plurality of first template pixels to obtain the second eye image includes: Determine a plurality of pupil area pixels that are not blocked in the first eye image; Determining a weighting coefficient according to the plurality of pupil area pixels; Determine a plurality of target template pixel points according to the weighting coefficients and the plurality of first template pixel points, wherein the plurality of target template pixel points correspond one-to-one to the plurality of first region pixel points; Each target template pixel point among the multiple target template pixel points replaces the first area pixel point in the blocked area corresponding to the target template pixel point to obtain the second eye image.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Performing feature extraction on the first eye image to obtain a plurality of eye image features, wherein the plurality of eye image features include a plurality of eye contour features; predicting a pupil reference area according to the plurality of eye contour features; Part of the eye image features in the pupil reference area among the multiple eye image features are matched with preset pupil image features to obtain a matching result. If the matching result does not meet a preset condition, it is determined that the first eye image includes an occluded area.

5. A pupil detection device, characterized in that: The device comprises: an acquisition unit, configured to acquire a first eye image when the user's eyelids droop or the user looks downward; a determination unit, configured to determine, if the first eye image includes an obstructed area, a target human eye image template that successfully matches the first eye image in a preset human eye image template library, wherein the obstructed area includes at least a portion of a pupil area; the human eye image template library includes a plurality of human eye image templates acquired at different angles and distances, and the human eye area of ​​each human eye image template is not obstructed; a restoration unit, configured to restore the first eye image according to the target human eye image template to obtain a second eye image; a detection unit, configured to perform pupil detection based on the second eye image to obtain a pupil center position; The step of repairing the first eye image according to the target human eye image template to obtain a second eye image includes: Determining the blocked area; Determine a template pupil area corresponding to the blocked area in the target human eye image template; Repairing the blocked area according to the template pupil area to obtain the second eye image; If the blocked area includes a partially blocked pupil area, repairing the blocked area according to the template pupil area to obtain the second eye image includes: Determine a plurality of first region pixel points in the blocked region whose noise is greater than a preset threshold; Determine a plurality of first template pixel points in the template pupil region corresponding one-to-one to the plurality of first region pixel points; The plurality of first region pixel points are repaired according to the plurality of first template pixel points to obtain the second eye image.

6. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store one or more programs and is configured to be executed by the processor, wherein the program comprises instructions for executing the steps in the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: A computer program for electronic data exchange is stored, wherein the computer program enables a computer to execute the method according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • Iris information acquisition method and acquisition system

    CN105335720A

  • Display device

    JP2023054581A

  • Camera device-based occlusion detection and repair device, and occlusion detection and repair method therefor

    WO2019080061A1