Pupil center coordinate acquisition system and acquisition method

Abstract

The embodiment of the present application provides a system and method for acquiring pupil center coordinates, the acquisition system includes a first camera, a second camera, a memory and a processor, the first camera captures a first image containing a face and outputs it to the processor, the second camera captures a second image containing a pupil and outputs it to the processor, the resolution of the first camera is lower than that of the second camera; the memory is used to store processing data, and the processor can operate to: acquire the first image and the second image; extract the first eye region corresponding to the human eye in the first image; convert the first eye region to the second image to obtain the second eye region corresponding to the human eye in the second image; detect the pupil in the second eye region and obtain the pupil center coordinates. Such a design can improve the calculation speed and accuracy of the pupil center coordinates, and can adapt to scenes with high requirements for the calculation accuracy and real-time performance of the pupil center coordinates.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of pupil detection technology, and in particular, to a system and method for acquiring pupil center coordinates. Background Art

[0002] The method for calculating the pupil center coordinates generally includes the following steps: using a camera to capture a face image containing a human face, performing face detection in the face image to obtain an eye region corresponding to a human eye, and performing pupil detection in the eye region to obtain the pupil center coordinates.

[0003] The calculation accuracy and speed of the pupil center coordinates are related to the camera resolution. Face images taken by a camera with a higher resolution will result in a decrease in the calculation speed, while face images taken by a camera with a lower resolution will result in a decrease in the calculation accuracy. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a system and method for obtaining pupil center coordinates, so as to obtain pupil center coordinates quickly and accurately.

[0005] Based on the above purpose, in a first aspect, an embodiment of the present application provides a system for acquiring pupil center coordinates, which includes a first camera, a second camera, a memory, and a processor, wherein the first camera captures a first image containing a face and outputs it to the processor, the second camera captures a second image containing a pupil and outputs it to the processor, and the resolution of the first camera is lower than that of the second camera; the memory is used to store processing data, and the processor can be operated to:

[0006] Acquire the first image and the second image;

[0007] Extracting a first human eye region corresponding to a human eye from the first image;

[0008] Converting the first human eye region to the second image to obtain a second human eye region corresponding to the human eye in the second image;

[0009] In the second human eye region, the pupil is detected and the pupil center coordinates are obtained.

[0010] In the pupil center coordinate acquisition system of the above embodiment, a first camera with a lower resolution is used to capture a first image containing a face, and a second camera with a higher resolution is used to capture a second image containing a pupil; a processor is used to perform face recognition on the first image to extract a first eye region, and then the first eye region is mapped to the second image to obtain a second eye region corresponding to the eye in the second image; pupil detection and pupil center coordinate calculation are then performed in the second eye region.

[0011] Since the resolution of the first image is relatively low, the first eye region can be quickly extracted, and since the resolution of the second image is relatively high, the accuracy of the through-hole center coordinates obtained through the second eye region is relatively high. In other words, the pupil center coordinate acquisition system provided by the embodiment of the present application can improve the calculation speed of the pupil center coordinate while improving the calculation accuracy, and can adapt to scenarios with high requirements for pupil center coordinate calculation accuracy and real-time performance.

[0012] In a possible implementation, detecting the pupil and obtaining the pupil center coordinates includes:

[0013] Converting the second image in the second eye area into a grayscale image;

[0014] Calculate and obtain an image gradient matrix of the grayscale image;

[0015] The pupil center coordinates are calculated according to the image gradient matrix.

[0016] In a possible implementation, calculating pupil center coordinates according to the image gradient matrix includes:

[0017] Calculating an adaptive threshold corresponding to the image gradient matrix;

[0018] Normalizing the gradient values ​​in the image gradient matrix according to the adaptive threshold to obtain an updated image gradient matrix;

[0019] The pupil center coordinates are calculated according to the updated image gradient matrix.

[0020] In a possible implementation, the calculating an adaptive threshold corresponding to the image gradient matrix includes:

[0021] Calculate the mean M of the image gradient matrix arg and standard deviation M dev ;

[0022] Calculate the grayscale average M of the pupil area gth ;

[0023] The adaptive threshold M is calculated by the following formula threshold :

[0024]

[0025] In a possible implementation, calculating pupil center coordinates according to the image gradient matrix includes:

[0026] Select any point in the image gradient matrix as the target point;

[0027] Calculate the vector dot product sum of the target point and the remaining points in the image gradient matrix;

[0028] Select all the target points and repeat the above steps in sequence to obtain a plurality of vector dot product sums;

[0029] All the vector dot product sums are compared, and the coordinates of the target point corresponding to the minimum vector dot product sum are the pupil center coordinates.

[0030] In a possible implementation, calculating pupil center coordinates according to the image gradient matrix includes:

[0031] Create a weight matrix;

[0032] Selecting any point in the image gradient matrix as a target point;

[0033] In combination with the weight vector matrix, calculating the vector dot product sum of the target point and the remaining points in the image gradient matrix;

[0034] Select all the target points and repeat the above steps in sequence to obtain a plurality of vector dot product sums;

[0035] Compare all the vector dot product sums, and the coordinates of the target point corresponding to the maximum vector dot product sum are the pupil center coordinates;

[0036] Wherein, the weight matrix is ​​MAX-P x，y , MAX is the maximum grayscale value corresponding to the color bit depth, P x，y is the pixel value in the grayscale image.

[0037] In a possible implementation, the processor is further configured to perform the following steps:

[0038] Perform camera calibration on the first camera and the second camera to obtain inter-camera coordinate conversion parameters between the first camera and the second camera.

[0039] In a possible implementation, the processor is further configured to perform the following steps:

[0040] Performing face detection on the first image and obtaining LandMark points;

[0041] The LandMark points are compared with a standard face model to obtain space coordinate transformation parameters.

[0042] In a possible implementation manner, after detecting the pupil and obtaining the pupil center coordinates, the method further includes:

[0043] Converting the pupil center coordinates into a second coordinate value with reference to the first picture according to the inter-camera coordinate conversion parameters;

[0044] The second coordinate value is converted into a three-dimensional coordinate value according to the spatial coordinate conversion parameter.

[0045] In a possible implementation, the first camera is a color camera, and the first image is a color image taken by the first camera; the second camera is an infrared camera, and the second image is an infrared image taken by the second camera; and the acquisition system further includes an infrared light source for performing infrared fill light under the control of the processor.

[0046] In a second aspect, an embodiment of the present application provides a method for obtaining pupil center coordinates, comprising:

[0047] Acquire a first image containing a face and a second image containing a pupil;

[0048] Extracting a first human eye region corresponding to a human eye from the first image;

[0049] Converting the first human eye region to the second image to obtain a second human eye region corresponding to the human eye in the second image;

[0050] In the second human eye region, detecting the pupil and obtaining the pupil center coordinates;

[0051] The resolution of the first image is smaller than the resolution of the second image.

[0052] In the method for obtaining the pupil center coordinates of the above embodiment, face recognition using a first image with a lower resolution can extract the first eye area more quickly, and then map the first eye area to the second image through coordinate transformation to obtain the second eye area corresponding to the eye in the second image. Since the resolution of the second image is higher, the accuracy of pupil detection and pupil center coordinate calculation in the second eye area is higher.

[0053] That is to say, the method for obtaining pupil center coordinates provided in the embodiment of the present application can improve calculation speed while improving calculation accuracy, and can adapt to scenarios with high requirements for pupil center coordinate calculation accuracy and real-time performance.

[0054] In a possible implementation, the step of detecting the pupil and obtaining the pupil center coordinates includes:

[0055] Converting the second image in the second eye area into a grayscale image;

[0056] Calculate and obtain an image gradient matrix of the grayscale image;

[0057] Calculating an adaptive threshold corresponding to the image gradient matrix;

[0058] Normalizing the gradient values ​​in the image gradient matrix according to the adaptive threshold to obtain an updated image gradient matrix;

[0059] The pupil center coordinates are calculated according to the updated image gradient matrix.

[0060] In a possible implementation, the step of calculating pupil center coordinates according to the updated image gradient matrix includes:

[0061] Create a weight matrix;

[0062] Select any point in the image gradient matrix as the target point;

[0063] In combination with the weight vector matrix, calculating the vector dot product sum of the target point and the remaining points in the image gradient matrix;

[0064] Select all the target points and repeat the above steps in sequence to obtain a plurality of vector dot product sums;

[0065] Compare all the vector dot product sums, and the coordinates of the target point corresponding to the maximum vector dot product sum are the pupil center coordinates;

[0066] Wherein, the weight matrix is ​​MAX-P x，y , MAX is the maximum grayscale value corresponding to the color bit depth, P x，y is the pixel value in the grayscale image.

[0067] In a possible implementation manner, after the step of detecting the pupil and obtaining the pupil center coordinates, the step further includes:

[0068] Converting the pupil center coordinates into a second coordinate value with reference to the first picture according to the inter-camera coordinate conversion parameter;

[0069] Converting the second coordinate value into a three-dimensional coordinate value according to a spatial coordinate conversion parameter;

[0070] The inter-camera coordinate conversion parameters are obtained by calibrating the first camera and the second camera;

[0071] The spatial coordinate transformation parameters are obtained by the following steps:

[0072] Performing face detection on the first image and obtaining LandMark points;

[0073] The LandMark points are compared with a standard face model to obtain space coordinate transformation parameters.

[0074] In a third aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the processor to perform the steps described in the embodiment of the first aspect. The computer instructions stored in the storage medium of this embodiment are used to cause the processor to calculate the pupil center coordinates, and therefore have the beneficial effect of obtaining the system embodiment, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0076] Figure 1 A schematic diagram of the structure of a system for acquiring the coordinates of the pupil center provided in an embodiment of the present application;

[0077] Figure 2 A method for obtaining pupil center coordinates provided in an embodiment of the present application Figure 1 ;

[0078] Figure 3 A method for obtaining pupil center coordinates provided in an embodiment of the present application Figure 2 ;

[0079] Figure 4 A method for obtaining pupil center coordinates provided in an embodiment of the present application Figure 3 ;

[0080] Figure 5 A method for obtaining pupil center coordinates provided in an embodiment of the present application Figure 4 . DETAILED DESCRIPTION

[0081] The calculation method of pupil center coordinates in the related art generally includes the following steps: using a camera to capture a face image containing a human face, performing face detection in the face image to obtain an eye area corresponding to the human eye, and performing pupil detection in the eye area to obtain pupil center coordinates.

[0082] The calculation accuracy and speed of the pupil center coordinates are related to the camera resolution. Although facial images taken by a higher-resolution camera can obtain more accurate pupil center coordinates, the calculation speed will be reduced. Although facial images taken by a lower-resolution camera can increase the calculation speed, the calculation accuracy will deteriorate.

[0083] In view of this, an embodiment of the present application provides a system for acquiring pupil center coordinates, which includes a first camera with lower resolution, a second camera with higher resolution, and a processor. During the calculation process of the pupil center coordinates, the processor uses the first camera to capture a face image, uses the second camera to capture a pupil image, performs face detection in the low-resolution image obtained by the first camera and obtains a first eye area corresponding to the human eye, then maps the first eye area to the second image to obtain a second eye area corresponding to the human eye in the second image, then uses a pupil detection algorithm to perform pupil detection in the second eye area and obtains the pupil center coordinates.

[0084] With this design, the human eye area can be detected more quickly through a lower resolution image, and then pupil detection can be performed in a higher resolution image to obtain a high-precision calculation result. The entire calculation process has a small amount of calculation, a fast calculation speed, and a relatively accurate calculation result, so that the pupil center coordinates can be obtained quickly and accurately.

[0085] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0086] Figure 1 A schematic diagram of the structure of a system for acquiring pupil center coordinates provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the acquisition system includes a first camera 10, a second camera 30, a memory 40 and a processor 20. The first camera 10 captures a first image containing a face and outputs it to the processor 20. The second camera 30 captures a second image containing a pupil and outputs it to the processor 20. The resolution of the first camera 10 is lower than that of the second camera 30. The memory 40 is used to store processing data, and the processor 20 is used to execute the method for acquiring pupil center coordinates provided in the following embodiments.

[0087] In a possible implementation, the first camera 10 is an RGB camera, i.e., a color camera. The color camera captures a color image containing a human face, and the detection effect of performing face detection in the color first image is better. The second camera 30 is an IR camera, i.e., an infrared camera. The infrared camera captures an infrared picture containing the pupil. The pupil feature is more obvious in the infrared image captured by the infrared camera than in the color picture captured by the color camera, which is conducive to pupil detection; and the infrared camera shooting is not limited by the use environment and can be used in brighter or darker environments.

[0088] In the embodiment where the second camera 30 is an IR camera, an infrared light source for infrared fill light is also provided. Multiple infrared light sources can be provided according to the luminous power and shooting requirements. For example, the infrared light source can be multiple infrared LED lights surrounding the first camera and the second camera.

[0089] In order to simplify the coordinate conversion between cameras, the focal lengths and visual angles of the first camera 10 and the second camera 30 are equal.

[0090] In a possible implementation, the acquisition system is a SmartView or a naked-eye 3D display product.

[0091] The present application embodiment also provides a method for obtaining pupil center coordinates, which can be used in the acquisition system in the above embodiment, such as Figure 2 As shown, the method for obtaining the pupil center coordinates includes:

[0092] Step S10: Acquire a first image including a face and a second image including a pupil;

[0093] The first image is an image containing a face taken by a first camera, and the second image is an image containing a pupil taken by a second camera. The resolution of the first camera is lower than that of the second camera, that is, the resolution of the first image is lower than that of the second image.

[0094] In a possible implementation, the first camera is an RGB camera, i.e., a color camera. The color camera captures a color image containing a human face, and the detection effect of performing face detection in the color first image is better. The second camera is an IR camera, i.e., an infrared camera. The infrared camera captures an infrared picture containing the pupil. The pupil feature is more obvious in the infrared image captured by the infrared camera than in the color picture captured by the color camera, which is beneficial for pupil detection; and the infrared camera shooting is not limited by the use environment and can be used in brighter or darker environments. In the implementation of the second camera being an IR camera, an infrared light source for infrared fill light is also provided.

[0095] The embodiment of the present application is described by taking the first camera as an RGB camera and the second camera as an IR camera as an example. However, the embodiment of the present application is not limited thereto. For example, in another possible implementation, the first camera and the second camera may both be RGB cameras, and the resolution of the first camera is lower than that of the second camera. The method for obtaining the pupil center coordinates may refer to the description herein.

[0096] Step S20: extracting a first human eye region corresponding to a human eye from the first image;

[0097] The face detection algorithm is used to search for faces globally within the image range of the first image. After obtaining the face frame, LandMark point detection is performed to obtain LandMark points. LandMark points are a number of marker points drawn on the face. Marker points are generally drawn at key positions such as edges, corners, contours, intersections, and equal divisions. With their help, the morphology of the face can be described.

[0098] For example, in a face recognition model, the LandMark points contain 68 points, and the correspondence between the 68 points and different positions of the face is as follows: 0-16 is the jaw line, 17-21 is the right eyebrow, 22-26 is the left eyebrow, 27-35 is the nose, 36-41 is the right eye, 42-47 is the left eye, 48-60 is the outer contour of the mouth, and 61-67 is the inner contour of the mouth.

[0099] From the above description, it can be seen that the LandMark points include feature points corresponding to the boundary of the human eye, and the area enclosed by the feature points corresponding to the boundary of the human eye is the human eye area. The human eye area can be extracted based on the feature points corresponding to the boundary of the human eye in the LandMark points. The human eye area usually includes the coordinates of the upper left vertex of the area and the size of the area. The human eye area in the first image is defined as the first human eye area.

[0100] Optionally, after obtaining 68 LandMark points of the face, you can choose to perform Kalman filtering on each LandMark point according to the actual situation to reduce the jump of the LandMark points and improve the stability of the LandMark points, so as to obtain 68 relatively stable LandMark points, and then extract the human eye area, so that the extraction result of the human eye area is more accurate.

[0101] Step S30: converting the first human eye region into the second image to obtain a second human eye region corresponding to the human eye in the second image;

[0102] The second camera has a high resolution, and pupil detection is performed directly on the second image, which requires a large amount of calculation. Moreover, in the embodiment where the second camera is an infrared camera, there are many interference factors in the second image taken by the infrared camera. If pupil detection is performed directly in the second image, these interference factors will affect the detection results of pupil detection, causing false pupil detection, thereby resulting in a large error in the calculation of the pupil center coordinates. Therefore, after obtaining the first human eye area in step S20, the first human eye area can be converted to the second image to obtain a second human eye area corresponding to the human eye in the second image, and then the pupil is detected in the second human eye area to obtain the pupil center coordinates.

[0103] However, since the internal parameter matrix and the external parameter matrix of the first camera are different from those of the second camera, after the first eye area is calculated in the first image, it cannot be directly applied to the second image. It is necessary to perform coordinate transformation on the first eye area relative to the coordinates of the first image and transform it to the coordinate system corresponding to the second image to obtain the second eye area corresponding to the eye area in the coordinate system of the second image.

[0104] The coordinate transformation between the first image and the second image is realized by the inter-camera coordinate transformation parameters, and the inter-camera coordinate transformation parameters can be determined according to the internal parameter matrix and the external parameter matrix of the first camera and the second camera. The internal parameter matrix refers to the parameters that reflect the properties of the camera itself, which are generally related to the focal length and distortion of the camera lens; the external parameter matrix refers to the coordinates of the camera relative to the world coordinate system, which can be considered as the position relationship of the camera in the world coordinate system, including the X, Y, and Z coordinates of the camera and the angle of the camera optical axis relative to the world coordinate system.

[0105] In a possible implementation, the first camera and the second camera are placed adjacent to each other, and the two cameras have the same field of view FOV, then the coordinate transformation relationship between the first camera and the second camera is:

[0106]

[0107] The resolution of the first camera is W1*H1, and the resolution of the second camera is W2*H2.

[0108] It can be seen that using two cameras with the same FOV can simplify the coordinate transformation between cameras.

[0109] Step S40: Detecting the pupil in the second person's eye area and obtaining the pupil center coordinates;

[0110] A pupil detection algorithm is used to extract pupil information in a second human eye region corresponding to the human eye region in the second image, and pupil center coordinates can be calculated.

[0111] In the method for obtaining pupil center coordinates in the above-mentioned embodiment, a first image with a relatively low resolution is used for face recognition and a first eye region is obtained, and then the first eye region is mapped to the second image through coordinate conversion to obtain a second eye region corresponding to the eye in the second image, and then pupil detection and pupil center coordinates are calculated in the second eye region; since the second image has a relatively high resolution, the calculation result of the pupil center coordinates has a relatively high accuracy, and face recognition and eye region extraction through the first image with a relatively low resolution have a relatively small amount of calculation and a relatively fast calculation speed compared to directly performing face recognition and eye region extraction in the second image. In other words, the method for obtaining pupil center coordinates provided in the embodiment of the present application can improve the calculation speed while improving the calculation accuracy, and can adapt to scenarios with high requirements for pupil center coordinate calculation accuracy and real-time performance.

[0112] Optionally, the internal parameter matrices of the first camera and the second camera can be obtained by Zhang's calibration method, and the acquisition process includes: placing a calibration plate within a set distance range in front of the first camera and the second camera to ensure that each camera can capture a clear image of the calibration plate; then rotating or twisting the calibration plate, and after each rotation or twisting, the two cameras collect an image as a calibration image; after shooting multiple times at different angles, each camera obtains multiple calibration images; each camera calculates its own internal parameter matrix through a program based on its own multiple calibration images.

[0113] The above camera calibration process only needs to be performed once, and there is no need to repeat it after obtaining the camera's internal parameter matrix.

[0114] Optionally, a method for obtaining pupil center coordinates provided in an embodiment of the present application further includes:

[0115] Perform face detection on the first image and obtain LandMark points;

[0116] The acquisition of LandMark points can refer to the description in step S20, which will not be repeated here.

[0117] The LandMark points are compared with the standard face model to obtain the spatial coordinate transformation parameters.

[0118] From the description of step S20, it can be known that 68 LandMark points can be obtained by performing face recognition on the first image. The 68 LandMark points are subjected to PnP operation with the coordinates of the 68 LandMark points in the standard face model, and the translation matrix T and the rotation matrix R of the first image coordinate system to the three-dimensional space coordinate system can be obtained. The space coordinate transformation parameters include the translation matrix T and the rotation matrix R. The distance from the face to the first camera can also be obtained through the PnP operation.

[0119] After obtaining the inter-camera coordinate conversion parameters and the space coordinate conversion parameters, the pupil center coordinates obtained in step S40 can be converted into three-dimensional space coordinates according to design requirements, such as Figure 3 As shown, a method for obtaining pupil center coordinates provided in an embodiment of the present application further includes, after step S40:

[0120] Step S50: converting the pupil center coordinates into a second coordinate value with reference to the first picture according to the inter-camera coordinate conversion parameter;

[0121] The pupil center coordinates calculated in step S40 are based on the second image as a reference, that is, based on the coordinate system of the second camera. According to the description in step S30, the conversion of coordinates between the first camera and the second camera can be achieved through the inter-camera coordinate conversion parameters. Therefore, the pupil center coordinates obtained in step S40 can be converted through the inter-camera coordinate conversion parameters to obtain the second coordinate value corresponding to the pupil center coordinates in the first image. The second coordinate value is based on the first image as a reference, that is, based on the coordinate system of the first camera.

[0122] Exemplarily, in an embodiment where the focal lengths f of the first camera and the second camera are the same, the conversion relationship between the pupil center coordinate and the second coordinate value is:

[0123]

[0124] Among them, (u c2 ,v c2 ) represents the pupil center coordinates obtained in step S40, l x Indicates the horizontal distance between the center of the first camera lens and the center of the second camera lens, l y represents the vertical distance between the center of the first camera lens and the center of the second camera lens, and f represents the focal length of the two cameras.

[0125] Step S60: converting the second coordinate value into a three-dimensional coordinate value according to the spatial coordinate conversion parameter.

[0126] The conversion formula between the three-dimensional coordinate value and the second coordinate value in the world coordinate system is as follows:

[0127]

[0128] in, represents the pupil center coordinates based on the first camera coordinate system, that is, the second coordinate value obtained in step S50;

[0129] Z C1 Indicates the distance from the face to the RGB camera. This parameter can be obtained through the PnP operation of the LandMark point.

[0130] represents the internal parameter matrix of the first camera, which has been obtained through the camera calibration step;

[0131] Represents the translation matrix T and rotation matrix R, that is, the spatial coordinate transformation parameters, which have been obtained through the PnP operation of the LandMark point.

[0132] Represents the three-dimensional coordinate value of the pupil center in the world coordinate system;

[0133] In this step, the known data is substituted into formula (1) to obtain the three-dimensional coordinate value of the pupil center in the world coordinate system.

[0134] There are few methods for calculating the three-dimensional coordinates of the pupil in three-dimensional space in the related art. Usually, a model estimation method is used. For example, after performing face detection to obtain the eye area, the pupil is assumed to be located in the center of the eye to calculate the three-dimensional coordinates of the pupil in three-dimensional space; or after obtaining the face coordinates through face detection, the relative difference between the center of the eye and the center of the face in a standard face model is directly added to the face coordinates to obtain the three-dimensional coordinates of the pupil in three-dimensional space. Both of the above methods cannot obtain accurate three-dimensional coordinates and cannot be used in some scenarios where the calculation accuracy of the pupil center coordinates is high.

[0135] According to the above description, the method for obtaining the pupil center coordinates provided in the embodiment of the present application can quickly and accurately obtain the three-dimensional coordinate value of the pupil center in the three-dimensional space through steps S10-S60, which has great application value for products such as SmartView and naked-eye 3D display.

[0136] Optional, such as Figure 4 As shown, in a method for obtaining pupil center coordinates provided in an embodiment of the present application, step S40 includes:

[0137] Step S41: converting the second image in the second eye area into a grayscale image;

[0138] This step usually also includes steps such as Gaussian filtering, and the pixel value of each pixel after conversion into a grayscale image is the grayscale value;

[0139] Step S42: Calculate and obtain the image gradient matrix of the grayscale image;

[0140] Each gradient value in the image gradient matrix is:

[0141]

[0142] Among them, X g is the gradient in the X direction, X gObtained by the following formula:

[0143]

[0144] Y g is the gradient in the Y direction, Y g Obtained by the following formula:

[0145]

[0146] P x,y is the pixel value of the pixel point (x, y).

[0147] Step S43: Calculate the pupil center coordinates according to the image gradient matrix.

[0148] According to the pupil grayscale characteristics and the pupil gradient characteristics, the pupil center coordinates can be calculated after obtaining the image gradient matrix.

[0149] According to the description and working principle of steps S41 to S43, the method for obtaining the pupil center coordinates calculates the pupil center coordinates based on the pupil grayscale characteristics and the pupil gradient characteristics. It is an acquisition method based on the gradient proper field, and the pupil center coordinates can be calculated on a color image or an infrared image, thereby solving the problem of difficulty in calculating the pupil center coordinates in complex environments, at long distances or in dark environments.

[0150] In order to improve the accuracy of the pupil center coordinate calculation result, in the embodiment of the present application, the second eye area is calculated and processed by an adaptive gradient threshold, and the pupil center coordinate is obtained, such as Figure 5 As shown, step S43 includes:

[0151] Step S431: Calculate the adaptive gradient threshold corresponding to the image gradient matrix;

[0152] The method for obtaining the adaptive gradient threshold can be:

[0153] Calculate the mean M of the image gradient matrix arg and standard deviation M dev ;

[0154] Mean M arg Calculated by the following formula:

[0155]

[0156] Among them, We and He represent the width and height of the single eye area in the second eye area respectively;

[0157] Standard Deviation M dev Calculated by the following formula:

[0158]

[0159] Adaptive threshold M threshold Calculated by the following formula:

[0160]

[0161] In the above formula, M gth M represents the average grayscale value of the pupil area. The pupil diameter is generally no more than 1 / 8 of the horizontal and 1 / 4 of the vertical of the human eye area. Therefore, in the embodiment of the present application, the human eye area is divided into 32*16 rectangular grids, and the average grayscale value of each rectangular grid is calculated, and the minimum average grayscale value is taken as M. gth .

[0162] Step S432: normalizing the gradient values ​​in the image gradient matrix according to the adaptive threshold to obtain an updated image gradient matrix;

[0163] In the normalization process, each gradient value M in the image gradient matrix is g The adaptive threshold M obtained in step S431 threshold Make a comparison;

[0164] If M g >M threshold ,but

[0165] If M g <M threshold ,but

[0166] For each gradient value M in the image gradient matrix g After the above normalization processing, the updated X-direction gradient matrix is ​​obtained And the Y-direction gradient matrix That is to say, the updated image gradient matrix is ​​obtained.

[0167] Step S433: creating a weight matrix;

[0168] Since the grayscale value of the pupil area is small and the grayscale values ​​of other areas are large, such grayscale distribution characteristics are not conducive to the identification and calculation of the pupil. In view of this, in the method for obtaining the pupil center coordinates provided in the embodiment of the present application, a weight rectangle is created as W, and the weight matrix W is MAX-P x，y , MAX is the maximum grayscale value corresponding to the color bit depth. For example, when the image is 8-bit color bit depth, MAX is 255. x，y is the pixel value in the grayscale image. It can be understood that in a grayscale image, the pixel value is the grayscale value.

[0169] By creating the weight matrix W, the weight of the pupil area can be increased and the weights of other areas can be reduced, which is beneficial to the detection of the pupil and the calculation of the center coordinates and reduces the difficulty of calculation.

[0170] Step S434: select any point in the image gradient matrix as the target point;

[0171] Step S435: Calculate the vector dot product sum of the target point and the remaining points in the image gradient matrix in combination with the weight vector matrix;

[0172] Step S436: Select all target points and repeat the above steps in sequence to obtain multiple vector dot product sums;

[0173] Step S437: compare all vector dot product sums, and the coordinates of the target point corresponding to the maximum vector dot product sum are the pupil center coordinates.

[0174] In steps S434 to S437, the pupil center coordinates are calculated based on the weight matrix W and the updated image gradient matrix.

[0175] In the updated image gradient matrix, any point O(x, y) is selected as the target point, and any point C(Xc, Yc) is selected as the second point from the remaining points. Then the normalized X-direction vector and Y-direction vector of OC are:

[0176]

[0177]

[0178] The X-direction gradient and Y-direction gradient at point C are: and

[0179] There is such a rule for the gradient around the pupil: Assuming that the pupil is a perfect circle, when O is the center of the pupil and C is the boundary of the pupil, the gradient of the vector OC and the point C are in the same direction, and the dot product value of the two is the largest, that is, The value of is the largest.

[0180] Based on the above characteristics of the pupil gradient, all target points are selected and the above steps are repeated in sequence to obtain multiple vector dot product sums corresponding to all target points; all vector dot product sums are compared, and by judging whether the vector dot product sum is the largest, it is judged whether the target point is the pupil center.

[0181] Vector dot product and OC sum The calculation formula is as follows:

[0182]

[0183] Select all target points O and repeat the above steps to calculate the dot products and OC of multiple vectors corresponding to all target points.sum , perform multiple vector dot products and OC sum The maximum value is selected by comparison, and the target point O corresponding to the maximum value is the pupil center, and the coordinates of the target point O are the coordinates of the pupil center.

[0184] From the above description and calculation formula, it can be seen that the vector dot product and OC corresponding to the target point O with a gradient value of 0 sum There will not be a maximum value, so in order to simplify the calculation process, a point whose gradient value is not 0 is usually selected as the target point in step S434.

[0185] The above-mentioned method for obtaining the pupil center coordinates is a method for obtaining the pupil center coordinates based on an adaptive threshold gradient vector field. This method calculates the pupil center coordinates based on the pupil grayscale characteristics and the pupil gradient characteristics. The pupil center coordinates can be calculated on a color image or an infrared image, solving the problem of difficulty in calculating the pupil center coordinates in complex environments, at long distances or in dark environments.

[0186] In the steps of the above embodiment, by setting the weight matrix, the weight of the pupil area can be increased and the weight of other areas can be reduced, which is conducive to highlighting the pupil features, thereby facilitating the detection and calculation of the pupil and reducing the difficulty of calculation. However, the embodiments of the present application are not limited to this. For example, in a possible implementation, the method for obtaining the pupil center coordinates includes:

[0187] In the image gradient matrix, any point with a non-zero gradient value is selected as the target point;

[0188] Calculate the vector dot product sum of the target point and the remaining points in the image gradient matrix;

[0189] Select all target points and repeat the above steps in sequence to obtain multiple vector dot product sums;

[0190] Compare the dot product sums of all vectors, and the coordinates of the target point corresponding to the minimum dot product sum of the vectors are the coordinates of the pupil center.

[0191] The working principle and working process of the acquisition method in this embodiment can be found in the description of steps S431-S437, which will not be repeated here.

[0192] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0193] In addition, the above description of specific embodiments of the present specification is provided. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0194] In the pupil center coordinate acquisition system of the embodiment of the present application, the processor has the beneficial technical effects of the acquisition method in the above embodiment by executing the acquisition method in the above embodiment, and can improve the calculation speed of the pupil center coordinates while improving the calculation accuracy, and can adapt to scenarios with high requirements on the calculation accuracy and real-time performance of the pupil center coordinates.

[0195] Based on the same inventive concept, an embodiment of the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions. The stored computer instructions are used to enable a processor in an acquisition system to execute the acquisition method in the above-mentioned implementation example.

[0196] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0197] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the acquisition method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments and system embodiments, which will not be repeated here.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A system for acquiring pupil center coordinates, characterized in that: The device comprises a first camera, a second camera, a memory and a processor, wherein the first camera captures a first image including a face and outputs the image to the processor, the second camera captures a second image including a pupil and outputs the image to the processor, and the resolution of the first camera is lower than that of the second camera; the memory is used to store processing data, and the processor is operable to: Acquire the first image and the second image; Extracting a first human eye region corresponding to a human eye from the first image; Converting the first human eye region to the second image to obtain a second human eye region corresponding to the human eye in the second image; In the second human eye region, detecting the pupil and obtaining the pupil center coordinates; Detecting the pupil and obtaining the pupil center coordinates includes: Converting the second image in the second eye area into a grayscale image; Calculate and obtain an image gradient matrix of the grayscale image; Calculate pupil center coordinates according to the image gradient matrix; Calculating pupil center coordinates according to the image gradient matrix comprises: Create a weight matrix; Select any point in the image gradient matrix as the target point; In combination with the weight matrix, calculating the sum of vector dot products of the target point and the remaining points in the image gradient matrix; Select all the target points and repeat the above steps in sequence to obtain a plurality of vector dot product sums; Compare all the vector dot product sums, and the coordinates of the target point corresponding to the maximum vector dot product sum are the pupil center coordinates; Wherein, the weight matrix is ​​MAX-P x，y , MAX is the maximum grayscale value corresponding to the color bit depth, P x，y is the pixel value in the grayscale image.

2. The acquisition system according to claim 1, characterized in that: Calculating pupil center coordinates according to the image gradient matrix comprises: Calculating an adaptive threshold corresponding to the image gradient matrix; Normalizing the gradient values ​​in the image gradient matrix according to the adaptive threshold to obtain an updated image gradient matrix; The pupil center coordinates are calculated according to the updated image gradient matrix.

3. The acquisition system according to claim 2, characterized in that: The calculating of the adaptive threshold corresponding to the image gradient matrix comprises: Calculate the mean M of the image gradient matrix arg and standard deviation M dev ; Calculate the grayscale average M of the pupil area gth ; The adaptive threshold M is calculated by the following formula threshold : W e , H e Respectively represent the width and height of the single eye area in the second eye area.

4. The acquisition system according to any one of claims 1 to 3, characterized in that: Calculating pupil center coordinates according to the image gradient matrix comprises: Select any point in the image gradient matrix as the target point; Calculate the vector dot product sum of the target point and the remaining points in the image gradient matrix; Select all the target points and repeat the above steps in sequence to obtain a plurality of vector dot product sums; All the vector dot product sums are compared, and the coordinates of the target point corresponding to the minimum vector dot product sum are the pupil center coordinates.

5. The acquisition system according to claim 1, characterized in that: The processor is further configured to perform the following steps: Perform camera calibration on the first camera and the second camera to obtain inter-camera coordinate conversion parameters between the first camera and the second camera.

6. The acquisition system according to claim 5, characterized in that: The processor is further configured to perform the following steps: Performing face detection on the first image and obtaining LandMark points; The LandMark points are compared with a standard face model to obtain space coordinate transformation parameters.

7. The acquisition system according to claim 6, characterized in that: After detecting the pupil and obtaining the pupil center coordinates, the method further includes: Converting the pupil center coordinates into a second coordinate value based on the first image according to the inter-camera coordinate conversion parameters; The second coordinate value is converted into a three-dimensional coordinate value according to the spatial coordinate conversion parameter.

8. The acquisition system according to any one of claims 5 to 7, characterized in that: The first camera is a color camera, and the first image is a color image taken by the first camera; the second camera is an infrared camera, and the second image is an infrared image taken by the second camera; the acquisition system also includes an infrared light source for infrared fill light under the control of the processor.

9. A method for obtaining pupil center coordinates, characterized in that: include: Acquire a first image containing a face and a second image containing a pupil; Extracting a first human eye region corresponding to a human eye from the first image; Converting the first human eye region to the second image to obtain a second human eye region corresponding to the human eye in the second image; In the second human eye region, detecting the pupil and obtaining the pupil center coordinates; wherein the resolution of the first image is smaller than the resolution of the second image; The step of detecting the pupil and obtaining the pupil center coordinates comprises: Converting the second image in the second eye area into a grayscale image; Calculate and obtain an image gradient matrix of the grayscale image; Calculate pupil center coordinates according to the updated image gradient matrix; The step of calculating pupil center coordinates according to the updated image gradient matrix comprises: Create a weight matrix; Select any point in the image gradient matrix as the target point; In combination with the weight matrix, calculating the sum of vector dot products of the target point and the remaining points in the image gradient matrix; Select all the target points and repeat the above steps in sequence to obtain a plurality of vector dot product sums; Compare all the vector dot product sums, and the coordinates of the target point corresponding to the maximum vector dot product sum are the pupil center coordinates; Wherein, the weight matrix is ​​MAX-P x，y , MAX is the maximum grayscale value corresponding to the color bit depth, P x，y is the pixel value in the grayscale image.

10. The acquisition method according to claim 9, characterized in that: Calculating pupil center coordinates according to the image gradient matrix comprises: Calculating an adaptive threshold corresponding to the image gradient matrix; Normalizing the gradient values ​​in the image gradient matrix according to the adaptive threshold to obtain an updated image gradient matrix; The pupil center coordinates are calculated according to the updated image gradient matrix.

11. The acquisition method according to claim 9, characterized in that: After the step of detecting the pupil and obtaining the pupil center coordinates, the following steps are further included: Converting the pupil center coordinates into a second coordinate value with reference to the first image according to an inter-camera coordinate conversion parameter; Converting the second coordinate value into a three-dimensional coordinate value according to a spatial coordinate conversion parameter; The inter-camera coordinate conversion parameters are obtained by calibrating the first camera and the second camera; The spatial coordinate transformation parameters are obtained by the following steps: Performing face detection on the first image and obtaining LandMark points; The LandMark points are compared with a standard face model to obtain space coordinate transformation parameters.

12. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the processor to perform the steps described in any one of claims 1-8.

Images