A smartphone control method based on eye gaze and facial micro-movements

Abstract

The present invention provides a method for controlling a smart phone based on eye gaze and facial micro-movements, and belongs to the field of smart phone control. The technical solution of the present invention is to pre-train a gaze point estimation model through a convolutional neural network by presetting multiple calibration points, use FaceNet to process the user's facial image, and output a first image set and a second image set. The first image set includes key values ​​of the facial triangle area as the convolution layer input of the gaze point estimation model, and the second image set, user distance, and gaze point boundary value are used as the full connection layer input of the gaze point estimation model. The gaze point estimation coordinates are output, the gaze point image is displayed, and then the user's facial features are obtained by using a dot matrix projector. Compared with the preset features, the execution operation is determined to meet the confidence interval. The method of the present invention has the real-time tracking of eye gaze, and realizes more humanized eye control. At the same time, only the front camera and sensor of the mobile phone are required, and no external equipment is required, which is low in cost.

Description

Technical Field

[0001] The invention belongs to the field of smart phone control, and in particular relates to a smart phone control method based on eye gaze and facial micro-movements. Background Art

[0002] With the rapid development of smart phones, smart phones have gradually become a necessity in modern people's lives with their strong functionality, high portability, and wide adaptability. Studies have shown that the average person interacts with their mobile phone more than 150 times a day. Smart phones currently on the market use capacitive touch screens. This touch operation requires the operator to tap and touch the screen to complete the interactive application. In addition to touch operations, some mobile phones use gesture recognition with the help of front cameras and sensors to realize the up and down page turning function under specific applications. For example, Huawei Mate series can perform air gesture recognition, and realize the sliding of short videos or the turning of e-books by recognizing the up and down or left and right waving of the palm. But in either case, the current smart phone interactive applications require the participation of hands, and there is a problem that it is inconvenient to operate the mobile phone (such as answering calls, entering or exiting APP) during the execution of tasks with both hands.

[0003] CN110191234B discloses a smart terminal unlocking method based on gaze point analysis. The feature labeling method is used to obtain the 68 feature point matrix of the face, the eye picture, the face area and the head posture angle, and the convolutional neural network model is used to predict the position of the human eye observing the screen. The user's intention and the unlocking function are judged by time, but it does not solve the extreme special situation that the human eye is at the edge of the mobile phone, which is likely to cause misjudgment and waste of resources.

[0004] CN108171152A discloses a deep learning method, device, system and readable storage medium for estimating human eye sight lines, which establishes a database by receiving human head images taken by multiple terminals, and establishes a human eye estimation model by deep learning. However, the images are obtained by taking pictures by multiple terminals, so the number of images obtained at the same time is limited, which may lead to insufficient facial data, and multiple terminal devices are also needed, which lacks simplicity; secondly, in order to take pictures from multiple angles, the user needs to frequently rotate the terminal screen, which increases the complexity of the operation. Summary of the invention

[0005] In view of the problems in the prior art, the present invention proposes a smart phone control method based on eye gaze and facial micro-movements, which realizes the operation control of the smart phone by real-time decoding of the operator's eye gaze position and coordinating facial micro-movements.

[0006] To achieve the above objectives, the present invention adopts the following technical scheme: a method for controlling a smart phone based on eye gaze and facial micro-movements, comprising the following steps: S1, pre-determining the user's operation intention, starting the front camera and infrared camera of the smart phone to obtain the user's facial image; S2, graying the user's facial image and inputting it into FaceNet, outputting a real-time first image set and a second image set, wherein the first image set is to segment the facial triangle area of ​​each image, obtain the distance from the center of the eyeball to the center of the face, the distance from the center of the eyeball to the inner corner of the eye, and the angle from the center of the eyeball to the center of the face and the inner corner of the eye, wherein the eyeball includes the left eyeball and the right eyeball; the second image set is the facial area of ​​each image , head angle, and eye corner position values; S3, using the first image set as the input of the convolution layer of the pre-trained gaze point estimation model, using the second image set and the real-time user distance value and gaze boundary value obtained by the depth sensor as the input of the full connection layer of the pre-trained gaze point estimation model, and outputting the real-time gaze point estimation coordinates; S4, judging the gaze point estimation coordinates, judging that the mobile phone application is not being gazed at, and the smart phone enters the standby state; judging that the mobile phone application being gazed at is an interactive object, starting the dot matrix projector (4) to obtain the user's facial features, and at the same time comparing them with the pre-set facial features through the discrimination index, and when the features match and fall within the confidence interval, judging to execute the operation.

[0007] Furthermore, the pre-determination method in S1 includes: starting the depth sensor (2) of the smartphone to measure the user distance value, and when the user distance value exceeds a threshold, determining it as non-operation intention, and the smartphone enters a standby state; if the user distance value does not exceed the threshold, determining it as operation intention.

[0008] Furthermore, the pre-training method of the gaze point estimation model in S3 comprises the following steps: S11, starting the front camera (1), the depth sensor (2) and the infrared camera (3) to capture the facial image of the user when gazing at different calibration points in the interactive area of ​​the mobile phone and recording the user distance value from the user's face to the mobile phone screen; S22, graying the facial image as the input of FaceNet, and using the first image set and the second image set at different calibration points as the output; S33, using the first image set at different calibration points as the input of the convolutional layer of the convolutional neural network, using the second image set at different calibration points and the user distance value and the gaze boundary value as the input of the fully connected layer of the convolutional neural network, using the two-dimensional pixel coordinates of the different calibration points as the solution of the model, and iterating repeatedly to obtain the gaze point estimation model.

[0009] Furthermore, the pre-training method of the gaze point estimation model also includes: presetting a plurality of calibration points in the smart phone, and the gaze point estimation coordinates corresponding to each calibration point are manually confirmed by the user.

[0010] Furthermore, the threshold of the user distance value is 20-50 cm.

[0011] Furthermore, the gaze boundary value is a boundary value of the interactive area of ​​the smartphone expanded to all sides by an allowable error value, and the allowable error value is the diameter of a gaze point image.

[0012] Furthermore, the method for determining the estimated coordinates of the gaze point in S4 includes: when the overlap area between the gaze point image and the application icon is calculated to be greater than half of the gaze point image, determining that the application is an interactive object.

[0013] Furthermore, the facial features described in S5 are "pouting" corresponding to the tap function, "pursing lips" corresponding to return / exit operation and non-operation, and the discrimination indicators are dot matrix offset and distortion value.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] (1) By outputting the gaze point of the interactive area of ​​the mobile phone through the key information of the facial triangle area, the real-time tracking of eye gaze can be ensured; the user's operation intention is determined by using the user distance value and the gaze point boundary value, and the user's operation intention has a high degree of determination accuracy; it is determined whether the gaze point is on the mobile phone application by comparing the overlapping area of ​​the gaze point image and the mobile phone application icon to determine whether it is an interactive object, which will not cause misjudgment and waste of resources, and is more humane;

[0016] (2) Only the front camera and sensor of the mobile phone are needed, no external equipment is required, and the cost is low. The present invention can effectively solve the problem of inconvenience in operating the mobile phone when performing tasks with both hands;

[0017] (3) In view of the possible errors that may occur in the boundary of the interactive area, the boundary value of the gaze point is limited, and the requirements for the error range of the gaze point are reduced. The boundary value of the gaze point is used as a constant value, which can reduce the amount of calculation of the output gaze point estimated coordinates and complete real-time operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a schematic diagram of the components of a smartphone;

[0019] Figure 2 This is a flowchart of the smart phone control method of the present invention;

[0020] Figure 3 Schematic diagram of facial triangle under different gaze points;

[0021] Figure 4 Schematic diagram of facial micro-movements;

[0022] Figure 5This is a flow chart of the gaze point estimation model training method of the present invention;

[0023] Figure 6 A schematic diagram of calibration points in the interactive area of ​​the mobile phone;

[0024] Figure 7 This is a schematic diagram of the eye angle for fixating the calibration point;

[0025] Figure 8 Build a flow chart for the gaze estimation model;

[0026] Fig. 9 Schematic diagram of gaze point estimation;

[0027] Fig.10 Schematic diagram of the estimation of the gaze point of the head and the interactive area;

[0028] Among them, 1 is the front camera, 2 is the depth sensor, 3 is the infrared camera, and 4 is the dot matrix projector. DETAILED DESCRIPTION

[0029] The following is further described in conjunction with the embodiments and drawings.

[0030] The present invention takes the iPhone X smartphone as an example, and the control method involves a front camera (1), a depth sensor (2), an infrared camera (3) and a dot matrix projector (4) of the smartphone, such as Figure 1 shown.

[0031] like Figure 2 As shown, the present invention provides a smart phone control method based on eye gaze and facial micro-movements, comprising the following steps:

[0032] S1, pre-determine the user's operation intention, start the front camera and infrared camera of the smartphone to obtain the user's head image;

[0033] S2. Grayscale the user's facial image and input it into FaceNet to generate a real-time first image set and a second image set. The first image set is to segment the facial triangle area of ​​each image to obtain the distance from the center of the eyeball to the center of the face, the distance from the center of the eyeball to the inner corner of the eye, and the angle from the center of the eyeball to the center of the face and the inner corner of the eye. The eyeball includes the left eyeball and the right eyeball. The second image set is the numerical values ​​of the facial area, head angle, and eye corner position of each image.

[0034] S3, using the first image set as the input of the convolutional layer of the pre-trained gaze point estimation model, using the second image set and the real-time user distance value and the gaze boundary value obtained by the depth sensor as the input of the fully connected layer of the pre-trained gaze point estimation model, and outputting the real-time gaze point estimation coordinates;

[0035] S4, judging the estimated coordinates of the gaze point, judging that it is not the gazed mobile phone application, and the smart phone enters the standby state; judging that the gazed mobile phone application is an interactive object, starting the dot matrix projector (4) to obtain the user's facial features, and comparing them with the pre-set facial features through the discrimination index, when the features match and fall within the confidence interval, it is judged that the operation is executed.

[0036] The pre-determination method in S1 is to start the depth sensor (2) of the smartphone to measure the user distance value. When the user distance value exceeds the threshold, it is determined as non-operation intention, and the smartphone enters the standby state; if the user distance value does not exceed the threshold, it is determined as operation intention. The threshold of the user distance value in the pre-determination method is 20 to 50 cm, that is, the user's operable distance to the smartphone is 20 to 50 cm. The real-time user distance value obtained by the depth sensor in S3 ranges from 20 to 50 cm, and the real-time user distance value is used as the input of the gaze point estimation model, and the real-time gaze point estimation coordinates are output.

[0037] The method for processing the user's facial image in S2 is to segment and grayscale the image frame by frame. The facial triangle is obtained by segmenting each image using FaceNet. The facial triangle is a triangular area formed by the right eyebrow tail, the left eyebrow tail and the nose tip. The center of gravity of the triangle is used as the facial center. When the user looks at different positions of the mobile phone interactive area, the positions of the facial center and the inner eye corner point on the face remain fixed, while the eyeball angle formed between the eyeball center and the facial center and the inner eye corner point changes, such as Figure 3 shown.

[0038] In S3, the gaze boundary value is the boundary value of the interactive area of ​​the smartphone, which is expanded by an allowable error value in all directions, and the allowable error value is the diameter of a gaze point image. The diameter of the gaze point image is determined by the operation accuracy of the mobile phone, the size of the interactive area of ​​the smartphone, the size of the application icon, and the distance between adjacent application icons. The present invention uses the gaze boundary value as the fully connected layer of the gaze point estimation model, which can be used to determine whether the user is gazing at the interactive area. If it is determined that the user is not gazing at the interactive area, the gaze point estimation model will not output the gaze point estimation coordinates, and the smartphone enters the standby state, which has a high judgment accuracy on the user's operation intention and will not cause resource waste.

[0039] In S4, the method for determining the estimated coordinates of the gaze point includes: when the overlap area between the gaze point image and the application icon is greater than half of the gaze point image, determining that the application is an interactive object. By determining the estimated coordinates of the gaze point, misjudgment of the user's operation intention is reduced, which is more humane.

[0040] The facial features in S5 are "pouting" corresponding to the tap function, "pursing lips" corresponding to the return / exit operation and non-operation, and the discrimination indexes are the dot matrix offset and distortion value, such as Figure 4 As shown. After the dot matrix projector (4) is started, countless light dots invisible to the naked eye are projected onto the user's face, and the user's facial features at this time are recorded. When the user makes a subtle facial movement, the dot matrix light dots on the face will be offset and distorted, and the current facial features can be obtained at this time. The present invention converts the tapping action of the mobile phone into the facial micro-movement "pouting", and converts the return / exit operation of the mobile phone into the facial micro-movement "pursing the lips". The dot matrix offset and distortion values ​​of "pouting" and "pursing lips" of the user are obtained in advance as discrimination indicators, and a 95% confidence interval is set. As an implementation method, a labeled "one-to-many" classification algorithm is used to perform pattern recognition and classification on facial micro-movements, and three feature values ​​of "pouting", "pursing lips" and non-operation are obtained. When the classification feature falls within the confidence interval, it can be determined that the operation is performed.

[0041] Before operation, users need to pre-train the gaze point estimation model, such as Figure 5 As shown, the method comprises the following steps:

[0042] S11, starting the front camera (1), the depth sensor (2) and the infrared camera (3) to capture a facial image of the user when the user is gazing at different calibration points in the interactive area of ​​the mobile phone and recording a user distance value from the user's face to the mobile phone screen;

[0043] S22, graying the facial image as the input of FaceNet, and using the first image set and the second image set of different calibration points as output;

[0044] S33. Using the first image set at different calibration points as the input of the convolutional layer of the convolutional neural network, using the second image set at different calibration points and the user distance value and the gaze boundary value as the input of the fully connected layer of the convolutional neural network, using the two-dimensional pixel coordinates of the different calibration points as the solution of the model, and repeatedly iterating to obtain the gaze point estimation model.

[0045] In S22, if the input recognized by FaceNet is a complete face, the first image set and the second image set are output. If not, it returns to S11 to shoot again. In S33, the user distance value of different calibration points is obtained by the depth sensor, and the threshold of the user distance value of different calibration points is 20 to 50 cm. The gaze boundary value is the boundary value of the interactive area of ​​the smartphone, which is expanded by an allowable error value in all directions. The allowable error value is the diameter of a gaze point image.

[0046] The pre-training method of the gaze point estimation model further includes: pre-setting a plurality of calibration points in the smart phone, and the estimated coordinates of the gaze point corresponding to each calibration point are manually confirmed by the user. The pre-training method of the gaze point estimation model of the present invention pre-sets 7 calibration points in the smart phone, such as Figure 6 As shown in the figure, each calibration of the estimated coordinates of a gaze point is completed by manual confirmation by the user, and each calibration point corresponds to a key information of the facial triangle. The key information of the facial triangle includes the distance from the center of the eyeball to the center of the face, the distance from the center of the eyeball to the inner corner of the eye, and the angle from the center of the eyeball to the center of the face and the inner corner of the eye. The eyeball includes the left eyeball and the right eyeball. When the user looks at the calibration point in the upper left corner of the interactive area of ​​the mobile phone, as shown in the figure, Figure 7 As shown, the line segment L from the center of the left eyeball to the center of the face α and the line segment l from the center point of the left eyeball to the inner corner of the eye α The left eyeball angle α is formed between them; the line segment L from the center point of the left eyeball to the inner corner point β and the line segment l from the center point of the left eyeball to the inner corner of the eye β The right eyeball angle β is formed between them.

[0047] In S22, the user's facial state is continuously photographed through the front camera of the smartphone, and the image is grayed out and input into FaceNet. FaceNet outputs α, L α , l α , β, L β , l β The values ​​of α, L and face area, head angle and eye corner position are also output. α , l α , β, L β , l β As the input of the convolutional layer of the convolutional neural network, the face area, head angle, eye corner position, user distance value, and gaze boundary value are used as the input of the fully connected layer, and the two-dimensional pixel coordinates of the calibration point are used as the supervision output of the convolutional neural network, such as Figure 8 As shown in the figure, the convolutional neural network will continuously modify parameters and variables according to the input conditions and supervision requirements to fit the closest discriminant model.

[0048] The present invention adopts convolutional neural network as a supervised learning model, and provides standard result-oriented training parameters to the convolutional neural network through calibration points. The key values ​​of the facial triangle area are input into the convolutional neural network, and a certain result is provided. The network pre-establishes the model, and uses cosine similarity correction and alignment orthogonal initialization algorithm to strengthen the model. After repeated iterative updates, the model parameters and variables are modified to fit the given results.

[0049] Since each person has different facial features, the positions of the face center and the inner eye corner are different, which leads to different distances from the eye center to the face center, the distance from the eye center to the inner eye corner, and the eye angle from the eye center to the face center and the inner eye corner. Therefore, the eye angle boundary value corresponding to the boundary of each person's gaze interactive area varies from person to person. Therefore, the present invention adopts a unified two-dimensional pixel coordinate boundary as the gaze boundary value to determine the user's gaze intention. The gaze boundary value is the boundary value of the smartphone's interactive area expanded by an allowable error value on all sides, and the allowable error value is the diameter of a gaze point image. Fig. 9 As shown, taking the iPhoneX smartphone as an example, the shaded area represents the interactive area of ​​the mobile phone, the center point of the interactive area of ​​the mobile phone is used as the origin of the two-dimensional pixel coordinate to establish an xy coordinate system, the interactive area size is 750*1468px, the x-axis value range of the interactive area in the coordinate system is [-375,375]px and the y-axis value range is [-734,734]px, the dark dot is the gaze point image, and the diameter is 80px. The gaze boundary value of the present invention is to expand the boundary value of the interactive area of ​​the smartphone to the surrounding areas by an allowable error value, and the error value is the diameter of a gaze point image, that is, the x-axis value range of the gaze point estimated coordinate in the coordinate system is [-375-80,375+80]px, that is, [-455,455]px, and the y-axis value range is [-734-80,734+80]px, that is, [-814,814]px, and the dotted box represents the gaze point estimated coordinate range output by the gaze point estimation model, that is, the gaze boundary. When the gaze point estimation coordinates output by the gaze point estimation model fall within the dotted box and outside the interactive area of ​​the mobile phone, the gaze point is still considered to be within the interactive area, and the image of the gaze point is adjusted to be displayed in the interactive area closest to the gaze point estimation coordinates. Fig.10 As shown in FIG. 1 , the user distance from the face to the interactive area of ​​the mobile phone, that is, the operable distance of the mobile phone, i.e., the value range of the z-axis is [20, 50] cm.

Claims

1. A method for controlling a smartphone based on eye gaze and facial micro-movements, characterized in that: The following steps are involved: S1. Pre-determine the user's operation intention, start the front camera and infrared camera of the smartphone to obtain the user's facial image; S2. Grayscale the user's facial image and input it into FaceNet, output the real-time first image set and the second image set, the first image set is to segment the facial triangle area of ​​each image, obtain the distance from the center of the eyeball to the center of the face, the distance from the center of the eyeball to the inner corner of the eye, and the angle from the center of the eyeball to the center of the face and the inner corner of the eye, the eyeball includes the left eyeball and the right eyeball; the second image set is the numerical value of the facial area, head angle, and eye corner position of each image; S3. Use the first image set as the input of the convolution layer of the pre-trained gaze point estimation model, use the second image set and the real-time user distance value and gaze boundary value obtained by the depth sensor as the input of the fully connected layer of the pre-trained gaze point estimation model, and output the real-time gaze point estimation coordinates; S4, determining the estimated coordinates of the gaze point, determining that the gaze point is not a mobile phone application, and the smart phone enters a standby state; determining that the gazed mobile phone application is an interactive object, starting the dot matrix projector (4) to obtain the user's facial features, and comparing them with the pre-set facial features through a discrimination index, and determining to execute the operation when the features match and fall within the confidence interval; The pre-determination method in S1 comprises: starting the depth sensor (2) of the smart phone to measure the user distance value, and when the user distance value exceeds a threshold, determining that there is no operation intention, and the smart phone enters a standby state; if the user distance value does not exceed the threshold, determining that there is an operation intention; The method for determining the estimated coordinates of the gaze point in S4 includes: when it is calculated that the overlap area between the gaze point image and the application icon is greater than more than half of the gaze point image, determining that the application is an interactive object.

2. The method for controlling a smart phone based on eye gaze and facial micro-movements according to claim 1, characterized in that: The pre-training method of the gaze point estimation model in S3 comprises the following steps: S11, starting the front camera (1), the depth sensor (2) and the infrared camera (3) to capture the facial image of the user when gazing at different calibration points in the interactive area of ​​the mobile phone and recording the user distance value from the user's face to the mobile phone screen; S22, graying the facial image as the input of FaceNet, and using the first image set and the second image set of different calibration points as the output; S33, using the first image set at different calibration points as the input of the convolutional layer of the convolutional neural network, using the second image set at different calibration points and the user distance value and the gaze boundary value as the input of the fully connected layer of the convolutional neural network, using the two-dimensional pixel coordinates of different calibration points as the solution of the model, and iterating repeatedly to obtain the gaze point estimation model.

3. The method for controlling a smart phone based on eye gaze and facial micro-movements according to claim 2, characterized in that: The pre-training method of the gaze point estimation model also includes: presetting a plurality of calibration points in the smart phone, and the gaze point estimation coordinates corresponding to each calibration point are manually confirmed by the user.

4. The method for controlling a smart phone based on eye gaze and facial micro-movements according to claim 2, characterized in that: The threshold of the user distance value is 20-50 cm.

5. The method for controlling a smart phone based on eye gaze and facial micro-movements according to claim 2, characterized in that: The gaze boundary value is obtained by expanding the boundary value of the interactive area of ​​the smartphone by an allowable error value in all directions, and the allowable error value is the diameter of a gaze point image.

6. The method for controlling a smart phone based on eye gaze and facial micro-movements according to claim 1, characterized in that: The facial features described in S5 are "pouting" corresponding to the tap function, "pursing lips" corresponding to return / exit operation and non-operation, and the discrimination indicators are dot matrix offset and distortion value.

Images