Method and device for determining the position of operator's gaze on terminal device screen

By generating virtual space on the terminal device and recording the corresponding relationship between the operator's human eye model and the terminal device screen, the problem of poor control effect of terminal devices in cold or noisy environments is solved, and the implementation of the Accurate control.

CN113760083BActive Publication Date: 2024-11-19张也弛
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010483284.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-06-01
Publication Date
2024-11-19
Estimated Expiration
2040-06-01

AI Technical Summary

Technical Problem

The control modes of existing terminal devices (capacitor screen sensing, sound commands and front-facing camera-based gesture control) are affected in cold or noisy environments, resulting in poor user experience.

Method used

By generating a virtual space including the operator's human eye model and the terminal device model, recording the corresponding relationship between the operator's human eye model when paying attention to the terminal device screen, and using the human eye feature information to determine the landing point of the operator's eyes on the terminal device screen. position, generate operation instructions to achieve control.

Benefits of technology

Without manual operation, precise control of terminal equipment is suitable for various environments and improves user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113760083B_ABST
    Figure CN113760083B_ABST
Patent Text Reader

Abstract

This article provides a method and device for determining the position of the operator's gaze on the terminal device screen, which is suitable for terminal device control scenarios, wherein the method includes: generating a virtual space including the operator's eye model and the terminal device model based on the collected image of the operator; recording the correspondence between different focused positions of the terminal device model screen and the characteristic information of the operator's eye model when the operator's eye model pays attention to the terminal device model screen in the virtual space; determining the position of the operator's gaze on the terminal device screen based on the eye characteristic information in the operator's image and the corresponding relationship, so that the terminal device generates a control instruction based on the position of the gaze. This article can have low requirements on the performance of the image acquisition device of the terminal device, can be unaffected by the environment, and can use the operator's gaze to achieve precise control of the terminal device without manual operation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a method for controlling a device, and more particularly to a method and device for determining the position of an operator's gaze on a screen of a terminal device. Background Art

[0002] With the development of society, terminal devices play an important role in people's lives and work, and the latest terminal devices are mainly controlled by capacitive screen sensing control, voice command (voice assistant) control, gesture control based on the front camera, etc. Although the above control methods can control terminal devices, they all have certain limitations, which are specifically reflected in:

[0003] 1. For capacitive screen sensing control: when the weather is cold or your hands are occupied, the control effect of the terminal device will be affected;

[0004] 2. Regarding the voice command control method: the command recognition is not accurate in noisy environments, it is not suitable for use in places that require quietness, and many functions need to be set in advance;

[0005] 3. For gesture control based on the front camera: In special circumstances, such as cold weather, the control effect of the terminal device will be affected. Summary of the invention

[0006] This article is used to solve the problem in the prior art that the landing point position on the terminal device screen and the control of the terminal device are achieved by the operator's manual control or voice control, and there is a problem that the user experience will be affected by the environment. For example, the environment is too cold, making it impossible for the operator to control the terminal device by hand.

[0007] In order to solve the above technical problem, a first aspect of this document provides a method for determining the position of an operator's gaze on a screen of a terminal device, comprising:

[0008] Generate a virtual space including an operator's eye model and a terminal device model according to the collected image of the operator;

[0009] Recording the correspondence between different focused positions on the terminal device model screen and feature information of the operator's human eye model when the operator's human eye model focuses on the terminal device model screen in the virtual space;

[0010] The location of the operator's gaze on the screen of the terminal device is determined according to the human eye feature information in the image of the operator and the corresponding relationship.

[0011] In a further embodiment, generating a virtual space including the operator's eye model and the terminal device model according to the collected image of the operator includes:

[0012] Determining a spatial position relationship between the operator and the terminal device according to the image of the operator;

[0013] According to the spatial position relationship between the operator and the terminal device, a virtual space including the operator's eye model and the terminal device model is generated.

[0014] In a further embodiment, recording the correspondence between different focused positions of the terminal device model screen and feature information of the operator's human eye model when the operator's human eye model focuses on the terminal device model screen in the virtual space includes:

[0015] According to a preset rule, the operator's human eye model in the virtual space is controlled to focus on a predetermined position of the terminal device model screen, characteristic information of the operator's human eye model is determined, and a corresponding relationship between the focused position of the terminal device screen and the characteristic information of the human eye model is established;

[0016] The correspondence between each focused position on the terminal device screen and the corresponding feature information of the human eye model is collected to obtain the correspondence between different focused positions on the terminal device model screen and the feature information of the operator's human eye model.

[0017] In a further embodiment, determining the location of the operator's gaze on the screen of the terminal device according to the human eye feature information in the image of the operator and the corresponding relationship includes:

[0018] From the corresponding relationship, searching for eye feature information of the operator;

[0019] The focused position on the terminal device model screen corresponding to the found human eye feature information is used as the landing point of the operator's gaze on the terminal device screen.

[0020] In a further embodiment, the operator's eye model is pre-established, including: an iris edge curve and eye surrounding features, and the eye surrounding features are used to locate the operator's eye model.

[0021] In a further embodiment, the process of determining the operator's eye model includes:

[0022] Searching and extracting eye feature information of the operator in each three-dimensional dot matrix model of the operator's face, wherein the three-dimensional dot matrix model of the operator's face is pre-scanned by a three-dimensional scanner;

[0023] The operator's eye model is constructed according to the extracted eye feature information.

[0024] A second aspect of the present invention provides a terminal device control method, including:

[0025] Acquiring images of the operator;

[0026] Generate a virtual space including an eye model of the operator and a terminal device model according to the collected image of the operator;

[0027] Recording the correspondence between different focused positions on the terminal device model screen and feature information of the operator's human eye model when the operator's human eye model focuses on the terminal device model screen in the virtual space;

[0028] Determining the location of the operator's gaze on the screen of the terminal device according to the human eye feature information in the image of the operator and the corresponding relationship;

[0029] An operation instruction of the terminal device is generated according to the location of the operator's gaze on the screen of the terminal device.

[0030] A third aspect of the present invention provides a device for determining a location of an operator's gaze on a screen of a terminal device, comprising:

[0031] A virtual space construction module, used to generate a virtual space including an eye model of the operator and a terminal device model according to the collected image of the operator;

[0032] A human eye movement simulation module, used for recording the correspondence between different focused positions of the terminal device model screen and feature information of the operator's human eye model when the operator's human eye model focuses on the terminal device model screen in the virtual space;

[0033] The operator target positioning module is used to determine the location of the operator's gaze on the screen of the terminal device according to the human eye feature information in the operator's image and the corresponding relationship.

[0034] A fourth aspect of the present invention provides a terminal device control device, comprising:

[0035] An image acquisition module, used for acquiring an image of an operator;

[0036] A virtual space construction module, used to generate a virtual space including an eye model of the operator and a terminal device model according to the collected image of the operator;

[0037] A human eye movement simulation module, used for recording the correspondence between different focused positions of the terminal device model screen and feature information of the operator's human eye model when the operator's human eye model focuses on the terminal device model screen in the virtual space;

[0038] An operator target positioning module, used to determine the location of the operator's gaze on the screen of the terminal device according to the human eye feature information in the operator's image and the corresponding relationship;

[0039] The control module is used to generate an operation instruction of the terminal device according to the location of the operator's gaze on the screen of the terminal device.

[0040] The fifth aspect of this document provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for determining the position of the operator's gaze on the screen of the terminal device as described in any of the aforementioned embodiments is implemented.

[0041] A sixth aspect of the present invention provides a computer-readable storage medium storing an executable computer program, which, when executed by a processor, implements the method for determining the position of the operator's gaze on the screen of a terminal device as described in any of the aforementioned embodiments.

[0042] The method and device for determining the position of the operator's gaze on the terminal device screen, the terminal device control method, the computer device and the computer device storage medium provided herein, generate a virtual space including the operator's human eye model and the terminal device model according to the collected image of the operator through the step of, so as to simulate the operator and the terminal device itself and their positional relationship in the virtual space; record the correspondence between different focused positions of the terminal device model screen and the characteristic information of the operator's human eye model when the operator's human eye model focuses on the terminal device model screen in the virtual space through the step of, so as to simulate the situation that the operator focuses on various positions of the terminal device screen, and determine the correspondence between the focused positions of the terminal device screen and the characteristic information of the operator's human eye model; determine the position of the operator's gaze on the terminal device screen according to the human eye characteristic information in the operator's image and the corresponding relationship, so as to accurately determine the position of the operator's gaze on the terminal device screen; generate the operation instruction of the terminal device according to the position of the operator's gaze on the terminal device screen through the step of, so as to realize the control of the terminal device by using the operator's gaze. This article can be applied to common terminal devices. It has low requirements on the performance of the image acquisition device of the terminal device and is not affected by the environment. It can achieve precise control of the terminal device by using the operator's eyes without manual operation.

[0043] In order to make the above and other purposes, features and advantages of this article more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of this article or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of this article. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1A A flow chart showing a method for determining the position of an operator's gaze on a terminal device screen according to an embodiment of the present invention;

[0046] Figure 1B A flow chart showing a terminal device control method according to an embodiment of this invention;

[0047] Figure 2 A schematic diagram showing a first vector diagram of an embodiment of this invention is shown;

[0048] Figure 3 A schematic diagram of a three-dimensional dot matrix model of an operator in the embodiment of this article is shown;

[0049] Figure 4 A schematic diagram of eye feature information of an operator in the embodiment of this invention is shown;

[0050] Figure 5 A schematic diagram of the process of establishing a three-dimensional lattice model by an operator in the embodiment of this invention is shown;

[0051] Figure 6 A flowchart showing a virtual space generation process according to an embodiment of the present invention is shown;

[0052] Figure 7 and Figure 8 A schematic diagram showing different perspectives of the virtual space of the embodiment of this article;

[0053] Fig. 9 A flowchart showing a process in which an operator's eye model simulates an operator's eye gaze at a terminal device in a virtual space in an embodiment of this invention;

[0054] Fig.10 A schematic diagram showing screen partitioning of a terminal device model according to an embodiment of this invention is shown;

[0055] Fig.11 A schematic diagram showing a second vector diagram of the embodiment of this invention is shown;

[0056] Fig.12 A schematic diagram showing the overlap of the second vector diagram of the embodiment of this article is shown;

[0057] Fig.13 A flow chart of determining the location of the operator's gaze on the terminal device screen according to an embodiment of the present invention is shown;

[0058] Fig.14 A flow chart of a terminal device control method according to a specific embodiment of this invention is shown;

[0059] Fig.15 Shows Fig.14 A schematic diagram of a comparison process between a first vector diagram and a second vector diagram obtained in a specific embodiment of FIG.

[0060] Fig.16 A structural diagram of a device for determining the position of an operator's gaze on a terminal device screen in an embodiment of this invention is shown;

[0061] Fig.17 The structure diagram of the terminal device control device in the embodiment of this invention is shown;

[0062] Fig.18 A structural diagram of a computer device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0063] In order to make the technical features and effects of this article more obvious, the technical solution of this article is further explained below in conjunction with the accompanying drawings. This article may also have other different specific examples to illustrate or implement it. Any equivalent changes made by any technical personnel in this field within the scope of the claims belong to the protection scope of this article.

[0064] The terminal device described in this article is a device with image acquisition and touch functions, such as a smart phone, PAD, etc. This article does not limit the specific type of terminal device. As long as the device has image acquisition and touch functions, it belongs to the terminal device described in this article.

[0065] like Figure 1A As shown, Figure 1A The flowchart of the method for determining the position of the operator's gaze on the terminal device screen in the embodiment of this article is shown. This embodiment is used to solve the problem that in the prior art, the control of the point position on the terminal device screen is achieved by the operator's manual control, which is affected by the environment and affects the user experience. For example, the environment is too cold, so that the operator cannot control the terminal device by hand.

[0066] Specifically, the method for determining the position of the operator's gaze on the terminal device screen includes:

[0067] Step 110: Generate a virtual space including an operator's eye model and a terminal device model based on the collected image of the operator.

[0068] Specifically, the image of the operator is captured by an image capture device (e.g., a camera) on the terminal device. In the specific implementation of this step, the image of the operator can also be recognized by face recognition software pre-stored on the terminal device to obtain the eye feature information in the image of the operator. In the specific implementation, in order to facilitate the storage and management of these feature points, the eye feature information in the operator image obtained based on the captured operator image can be represented in the form of a vector diagram, which is called the first vector diagram, such as Figure 2 As shown, the first vector image 200 includes human eye feature information, which at least includes a human eye iris edge curve 201. In order to accurately locate the human eye feature information, the human eye feature information also includes eye surrounding feature information 202-204, such as the outer corners of the left and right eyes 202, the nose tip 203, the junction of the forehead and the hair corner shape 204, etc., and can also include the pupil center point and a point next to the pupil center point. Specifically, the human eye iris edge curve 201 can be formed by connecting the iris edge features, and the iris edge in the operator's image can also be directly outlined with a vector curve to form the human eye iris edge curve 201.

[0069] The operator's eye model is used to simulate the operator's eye movement. It is a collection of eye feature information when the operator's eyeball moves in various directions. This collection of eye feature information constitutes the three-dimensional shape and movement law of the human eye. In order to avoid interference from multiple information and to be applicable to ordinary terminal devices, ordinary terminal devices in this article refer to image acquisition devices with low performance, such as low pixels, and the eye feature information in the eye model is simplified.

[0070] In some embodiments, the operator's eye model can be established based on the currently collected image of the operator and / or the historically collected image of the operator. The specific establishment process can refer to the process of using images to establish a three-dimensional model in the prior art, which is not limited in this article.

[0071] In other implementations, the process of determining the operator's eye model includes:

[0072] S1, on each face three-dimensional dot matrix model of the operator (such as Figure 3 ), find and extract the operator's eye feature information 401. It should be noted that the extracted eye feature 401 is the same as the feature recognized by the face recognition software in the operator's real image (such as Figure 4 As shown, it at least includes a dot matrix of the edge of the human eye iris, and these dot matrices are connected into a curve to form an iris edge curve), wherein the three-dimensional dot matrix model of each face of the operator is pre-scanned by the three-dimensional scanner 503 during the process of the operator 501 looking at the calibration plate 502 one by one (such as Figure 5 shown).

[0073] S2, constructing an operator's eye model according to the eye feature information extracted in step S1, that is, constructing an operator's eye model by combining the extracted eye feature information.

[0074] The terminal device model has a virtual screen and a virtual camera for simulating the terminal device. Its size ratio is the same as that of the terminal device. The virtual space including the operator's human eye model and the terminal device model is used to simulate the operator, terminal device and their positional relationship in the real scene.

[0075] Step 120, recording the correspondence between different focused positions on the terminal device model screen and feature information of the operator's eye model when the operator's eye model focuses on the terminal device model screen in the virtual space. The focused position described herein is usually a region.

[0076] In specific implementation, the terminal device model screen is divided into multiple areas, for example, 24 areas, each area has a different number, and in the virtual space, the operator's human eye model focuses on each area of ​​the terminal device model screen one by one.

[0077] The correspondence between different focused positions on the terminal device model screen and the characteristic information of the operator's human eye model can be stored in the database in the form of a data list, as shown in Table 1:

[0078] Table 1

[0079] Terminal device model screen area number Feature information of the human eye model when it focuses on each area 1001 A1 1002 A2 … …

[0080] Step 130, determining the location of the operator's gaze on the screen of the terminal device based on the eye feature information in the operator's image and the corresponding relationship, wherein the location can be a point on the screen or an area range.

[0081] The human eye feature information in the operator's image is the same as the feature information of the operator's human eye model. That is to say, when the human eye feature information in the operator's image can only be extracted as simplified human eye feature information due to the performance limitations of the image acquisition device of the terminal device, the feature information of the operator's human eye model is also simplified human eye model feature information. The simplified human eye model feature information is the same as the simplified human eye feature information of the operator image. For example, the human eye feature information in the operator's image includes the outer corners of the left and right eyes, the tip of the nose, the junction of the forehead and the hair corner shape, and the dot matrix of the edge of the human eye iris. The feature information in the human eye model also includes the outer corners of the left and right eyes, the tip of the nose, the junction of the forehead and the hair corner shape, and the dot matrix of the edge of the human eye iris.

[0082] This embodiment is applicable to the scenario of terminal device control. After determining the location of the operator's gaze on the terminal device screen, it will generate and respond to operation instructions based on the location of the operator's gaze on the terminal device screen to complete the operator's control of the terminal device. For example, after identifying the location of the operator's gaze, it stays for a predetermined time, then triggering a click operation at that location on the terminal device screen, or triggering the control function at that location (or area) on the screen. In specific implementation, it is also possible to count the duration of the operator's gaze on the terminal device screen, and trigger different operations at that location on the terminal device screen according to different durations.

[0083] This embodiment is also applicable to the scenario where the server side obtains the content of interest to the user. After determining the location of the operator's gaze on the terminal device screen, the terminal device will send the content corresponding to the location of the operator's gaze on the terminal device screen to the corresponding server side, so that the server side can analyze the content of interest to the user based on the received content. In other embodiments, the terminal device can also send the location of the operator's gaze on the terminal device screen (or information such as the location and the gaze dwell time) to the corresponding server side, so that the server side can analyze the content of interest to the user based on the received content.

[0084] When this embodiment is applied to a scenario where a terminal device is controlled, such as Figure 1B As shown, the above step 130 further includes: step 140, generating an operation instruction of the terminal device according to the location of the operator's gaze on the screen of the terminal device.

[0085] When this embodiment is applied to a scenario where the server side obtains the content of interest to the user, the above step 130 further includes: sending the location of the operator's gaze on the screen of the terminal device to the server side.

[0086] Of course, this embodiment can also be applied to other scenarios during implementation, which is not limited herein.

[0087] In one embodiment of this invention, Figure 6 As shown, the above step 110 generates a virtual space including the operator's eye model and the terminal device model according to the operator's image, including:

[0088] Step 610: Determine the spatial position relationship between the operator and the terminal device according to the image of the operator.

[0089] In this step, for example, laser positioning can be used to set a laser distance measuring unit on the front camera of the terminal device, so as to obtain the spatial position relationship between the terminal device and the operator. Alternatively, the operator image obtained by terminal device image acquisition can be used to analyze the image, for example, according to the size of the operator in the image or the arm length of the general operator, to obtain the spatial position relationship between the operator and the terminal device through calculation.

[0090] Step 620: Generate a virtual space including the operator's eye model and the terminal device model according to the spatial position relationship between the operator and the terminal device.

[0091] like Figure 7 and Figure 8 As shown, the virtual space includes an operator eye model 710, a terminal device model 720 and the positional relationship between them, which is used to simulate the real operator's eyes looking at a certain area of ​​the terminal device screen. The operator eye model 710 includes: right outer eye corner 711, left outer eye corner 712, nose tip 713, the junction of the right forehead and the hair corner shape 714, and the iris edge curves 715 and 716 of the operator's left and right eyes.

[0092] In one embodiment of this invention, Fig. 9 As shown, the above step 120 records the correspondence between different focused positions of the terminal device model screen and the feature information of the operator's human eye model when the operator's human eye model focuses on the terminal device model screen in the virtual space, including:

[0093] Step 910, according to a preset rule, control the operator's human eye model in the virtual space to focus on a predetermined position of the terminal device model screen, determine the characteristic information of the operator's human eye model, and establish a corresponding relationship between the focused position of the terminal device screen and the characteristic information of the human eye model.

[0094] In order to ensure that every position in the terminal device model is noticed by the operator's human eye model, during implementation, Fig.10 As shown, the terminal device model screen is divided into multiple areas, for example, 24 areas. The specific number of areas 1001 to 1024 depends on the screen size and touch range. Each area has a unique number. Usually, the areas are numbered from left to right and from top to bottom. In the specific operation of this step, according to the number of each area, the operator's human eye model in the virtual space is controlled to focus on the area with the corresponding number in the terminal device model screen.

[0095] In this step, the process of determining the characteristic information of the operator's human eye model includes: when the operator's human eye model in the virtual space focuses on a position on the terminal device model screen, the virtual camera on the terminal device model (used to simulate the image acquisition device on the real terminal device) collects the image of the operator's human eye model, which can be used as the characteristic information of the operator's human eye model, referred to as the second vector diagram, wherein the second vector diagram has the same format as the first vector diagram. In addition, the terminal device model screen is divided into multiple areas (24), and 24 second vector diagrams are obtained by simulated shooting, that is, the number of the second vector diagram is the same as the number of areas divided into the terminal device model screen. The rendering imaging rule of the second vector diagram is consistent with the imaging rule of the image acquisition device of the terminal device. Because of the imitation shooting, the focal length of the rendering may not be consistent with the image acquisition device of the real terminal device, and the image acquisition device of the real terminal device shoots the image, and the image edge imaging has a stretched effect, for example, the person at the edge of the image in the image of the person will appear fatter. Therefore, only by adjusting the imaging rules of the terminal device model, including the focal length, the effect of the image edge being stretched, etc., to be consistent with the imaging rules of the image acquisition device of the real terminal device, the effect of the image edge being stretched, etc., can the operator's gaze on the terminal device screen be analyzed more accurately, so as to more accurately control the terminal device. Among them, the focal length of the image acquisition device of the real terminal device can be obtained according to the aforementioned captured character image, or obtained from the setting parameters of the image acquisition device of the terminal device when capturing the character image, and other setting parameters of the image acquisition device when capturing the character image can be obtained from the terminal device or from the captured character image, and these imaging parameters are used to configure the parameters of the virtual camera on the terminal device model, so that the imaging rules of the virtual camera on the terminal device model can be consistent with the imaging rules of the image acquisition device of the real terminal device, and an image of the human eye model with a high similarity to the character image captured by the image acquisition device of the real terminal device is obtained.

[0096] In a specific implementation, Fig.11 As shown in FIG. 1 , the second vector diagram includes the right outer corner of the eye 1101, the left outer corner of the eye 1102, the nose tip 1103, the junction of the right forehead and the hair corner 1104, and the iris edge curves 1105 and 1106 of the operator's left and right eyes. If the terminal device model screen is divided into 24 areas, 24 second vector diagrams will be obtained (the second vector diagrams are numbered 1502 to 1525 respectively). The effect of the 24 second vector diagrams overlapping is as follows: Fig.12 shown.

[0097] Step 920, collect the correspondence between each focused position on the terminal device screen and the corresponding feature information of the human eye model, and obtain the correspondence between different focused positions on the terminal device model screen and the feature information of the operator's human eye model.

[0098] In one embodiment of this invention, Fig.13 As shown, the above step 130 determines the location of the operator's gaze on the screen of the terminal device according to the human eye feature information in the image of the operator and the corresponding relationship, including:

[0099] Step 1310 , searching for the operator's human eye feature information from the correspondence between different focused positions on the terminal device model screen determined in step 920 and the feature information of the operator's human eye model.

[0100] Step 1320: The focused position on the terminal device model screen corresponding to the found human eye feature information is used as the landing position of the operator's gaze on the terminal device screen.

[0101] In order to explain the technical solution of this article more clearly, the following is a detailed description of using the operator's gaze to control the terminal device as an example. Fig.14 and Fig.15 As shown, the terminal device control method includes:

[0102] Step 1410: The front camera of the terminal device captures an image of the operator.

[0103] Step 1420, the operator image is recognized by using the face recognition software pre-stored in the terminal device to obtain the operator's eye feature information, and the first vector image 1501 is rendered according to the operator's eye feature information. The operator's eye feature information includes the left outer corner of the eye, the right outer corner of the eye, the tip of the nose, the junction of the right forehead and the hair corner shape, and the iris edge curve.

[0104] Step 1430, based on the operator's eye feature information, determine the angle between the light entering the front camera lens of the terminal device at the eye feature pixel and the normal line of the front camera lens of the terminal device, and the angle is used to represent the positional relationship between the operator and the terminal device.

[0105] Step 1440, based on the angle obtained in step 1430, a virtual space including the operator's eye model and the terminal device model is generated to simulate the positional relationship between the operator and the terminal device.

[0106] Step 1450, control the operator's human eye model in the virtual space to focus on predetermined positions on the terminal device model screen one by one, determine the second vector diagrams 1502-1525 containing the characteristic information of the operator's human eye model, and establish a corresponding relationship between the focused position on the terminal device screen and the second vector diagrams 1502-1525, as shown in Table 2. The characteristic information of the operator's human eye model includes the left outer corner of the eye, the right outer corner of the eye, the tip of the nose, the junction of the right forehead and the hair corner shape, and the iris edge curve.

[0107] Table 2:

[0108] The number of the focused position on the terminal device screen Second vector diagram 1001 Second vector image 1502 1002 Second vector image 1503 … …

[0109] Step 1460, compare the first vector diagram 1501 and the second vector diagrams 1502-1525, find the second vector diagram that is most similar to the first vector diagram 1501. In this embodiment, the second vector diagram 1505 is most similar to the first vector diagram, and the focused position on the terminal device screen corresponding to the second vector diagram 1505 is used as the landing position of the operator's gaze on the terminal device screen.

[0110] The process of comparing the first vector graph with any second vector graph includes:

[0111] (1) First, the left outer corner of the eye, the right outer corner of the eye, the nose tip, the right forehead and the hair corner shapes in the first vector diagram are compared with the second vector diagram. Because there is an error between the spatial position relationship between the terminal device model and the operator's human eye model and the terminal device and the operator in reality, the left outer corner of the eye, the right outer corner of the eye, the nose tip, the right forehead and the hair corner shapes in the first vector diagram 1501 cannot completely overlap with the left outer corner of the eye, the right outer corner of the eye, the nose tip, the right forehead and the hair corner shapes in the second vector diagrams 1502-1525. Therefore, when this step is implemented specifically, it is only necessary to ensure that the distance a between the left eye corner of the second vector diagram and the left eye corner in the first vector diagram, the distance b between the right eye corner of the second vector diagram and the right eye corner in the first vector diagram, the distance c between the nose tip of the second vector diagram and the nose tip of the first vector diagram, and the distance d between the forehead of the second vector diagram and the forehead of the first vector diagram are as equal as possible (the difference is within a predetermined range), and then it is considered that the left outer eye corner, the right outer eye corner, the nose tip, the right forehead and the hair corner shape in the first vector diagram match the second vector diagram.

[0112] (2) After the left outer corner of the eye, the right outer corner of the eye, the nose tip, the right forehead and the hair corner shapes in the first vector image and the second vector image are matched, the iris edge curves in the first vector image and the second vector image are compared (including the shape and position of the iris edge curve). If the shape and position of the iris edge curves in the first vector image and the second vector image are similar, then the first vector image and the second vector image are considered to be the most similar.

[0113] In specific implementation, if only one of the second vector diagrams 1502 to 1525 has an iris edge curve that is similar in position and shape to the iris edge curve in the first vector diagram, the area of ​​the terminal device model screen corresponding to the second vector diagram is used as the location of the operator's gaze on the terminal device screen.

[0114] If, among the second vector diagrams 1502-1525, the iris edge curves in two or more of the second vector diagrams are similar in position and shape to the iris edge curves in the first vector diagram, it means that the operator's gaze falls between two adjacent upper screen areas, then the process returns to step 1450, further subdivides and numbers the screen area where the operator falls, and continues to execute step 1450 and subsequent steps.

[0115] Step 1470, generating an operation instruction according to the operator's landing point on the terminal device screen, and the terminal device executes the operation instruction to complete the operator's control over the terminal device.

[0116] The terminal device control method provided in this embodiment is applicable to ordinary terminal devices, has low requirements on the performance of the image acquisition device of the terminal device, is not affected by the environment, and can achieve precise control of the terminal device by using the operator's eyes without manual operation.

[0117] Based on the same inventive concept, this article also provides a device for determining the position of the operator's gaze on the terminal device screen, as described in the following embodiments. Since the principle of the device to solve the problem is similar to the method for determining the position of the operator's gaze on the terminal device screen, the implementation of the device can refer to the implementation of the method for determining the position of the operator's gaze on the terminal device screen, and the repeated parts will not be repeated.

[0118] like Fig.16 As shown, the device for determining the position of the operator's gaze on the terminal device screen includes:

[0119] The virtual space construction module 1610 is used to generate a virtual space including the operator's eye model and the terminal device model according to the collected image of the operator.

[0120] The human eye movement simulation module 1620 is used to record the correspondence between different focused positions of the terminal device model screen and feature information of the operator's human eye model when the operator's human eye model focuses on the terminal device model screen in the virtual space.

[0121] The operator target positioning module 1630 is used to determine the location of the operator's gaze on the screen of the terminal device according to the human eye feature information in the image of the operator and the corresponding relationship.

[0122] In one embodiment of this invention, Fig.17 As shown, a terminal device control device is also provided, including:

[0123] The image acquisition module 1710 is used to acquire the image of the operator.

[0124] The virtual space construction module 1720 is used to generate a virtual space including the operator's eye model and the terminal device model according to the collected image of the operator.

[0125] The human eye movement simulation module 1730 is used to record the correspondence between different focused positions of the terminal device model screen and feature information of the operator's human eye model when the operator's human eye model focuses on the terminal device model screen in the virtual space.

[0126] The operator target positioning module 1740 is used to determine the location of the operator's gaze on the screen of the terminal device according to the human eye feature information in the image of the operator and the corresponding relationship.

[0127] The control module 1750 is used to generate an operation instruction of the terminal device according to the location of the operator's gaze on the screen of the terminal device.

[0128] The terminal device control device provided in this embodiment is set on the terminal device in the form of software or hardware, and can be unaffected by the environment. It can use the operator's eyes to achieve precise control of the terminal device without manual operation.

[0129] In one embodiment of the present invention, a computer device is also provided, such as Fig.18As shown, the computer device 1802 may include one or more processors 1804, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The computer device 1802 may also include any memory 1806, which is used to store any kind of information such as code, settings, data, etc. Non-limitingly, for example, the memory 1806 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 1802. The memory 1806 stores a computer program that can be run on the processor 1804, and when the processor 1804 executes the computer program, the method for determining the position of the operator's gaze on the terminal device screen or the terminal device control method described in any of the aforementioned embodiments is implemented. In one embodiment, when the processor 1804 executes the associated instructions stored in any memory or combination of memories, the computer device 1802 can perform any operation of the associated instructions. The computer device 1802 also includes one or more drive mechanisms 1808 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0130] The computer device 1802 may also include an input / output module 1810 (I / O) for receiving various inputs (via input device 1812) and for providing various outputs (via output device 1814). A specific output mechanism may include a presentation device 1816 and an associated graphical user interface (GUI) 1818. In other embodiments, the input / output module 1810 (I / O), input device 1812, and output device 1814 may not be included, and the computer device 1802 may be used as a computer device in a network. The computer device 1802 may also include one or more network interfaces 1820 for exchanging data with other devices via one or more communication links 1822. One or more communication buses 1824 couple the components described above together.

[0131] The communication link 1822 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 1822 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0132] The embodiments of the present invention further provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for determining the position of the operator's gaze on the screen of the terminal device described in any of the above embodiments is executed.

[0133] The embodiments of this document also provide a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to execute the method for determining the position of the operator's gaze on the screen of the terminal device as described in any of the above embodiments.

[0134] The embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the terminal device control method described in any of the above embodiments is executed.

[0135] The embodiments of this document also provide a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to execute the terminal device control method described in any of the above embodiments.

[0136] It should be understood that in the various embodiments of this document, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0137] It should also be understood that in the embodiments of this article, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0138] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this article.

[0139] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0140] In the several embodiments provided herein, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0141] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this article.

[0142] In addition, each functional unit in each embodiment of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.

[0143] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0144] Specific embodiments are used in this article to illustrate the principles and implementation methods of this article. The description of the above embodiments is only used to help understand the methods and core ideas of this article. At the same time, for general technicians in this field, according to the ideas of this article, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on this article.

Claims

1. A method for determining the position of an operator's gaze on a terminal device screen, characterized in that: include: The terminal device collects the operator's image; Determining a spatial position relationship between the operator and the terminal device according to the image of the operator; generating a virtual space including an operator's eye model and a terminal device model according to a spatial position relationship between the operator and the terminal device; The operator's eye model is pre-established to simulate the operator's eye movement, and is a collection of eye feature information when the operator's eyeballs rotate to various directions; The terminal device model has a virtual screen and a virtual camera for simulating the terminal device; Recognizing the image of the operator to obtain first eye feature information; According to a preset rule, the operator's human eye model in the virtual space is controlled to focus on a predetermined position of the terminal device model screen, second human eye feature information is determined through a virtual camera, and a corresponding relationship between the focused position of the terminal device screen and the second human eye feature information is established; Collecting the correspondence between each focused position on the terminal device screen and the corresponding second human eye feature information, to obtain the correspondence between different focused positions on the terminal device model screen and the second human eye feature information; From the corresponding relationship, searching for second human eye feature information that is most similar to the first human eye feature information; The focused position on the terminal device model screen corresponding to the found second human eye feature information is used as the landing point of the operator's gaze on the terminal device screen.

2. The method according to claim 1, characterized in that The operator's eye model includes: an iris edge curve and eye surrounding features, and the eye surrounding features are used to locate the operator's eye model.

3. The method according to claim 1, characterized in that The operator's eye model is predetermined by the following steps: Searching and extracting eye feature information of the operator in each three-dimensional dot matrix model of the operator's face, wherein the three-dimensional dot matrix model of the operator's face is pre-scanned by a three-dimensional scanner; A human eye model of the operator is constructed according to the extracted human eye feature information.

4. A terminal device control method, characterized in that: include: Using the method described in any one of claims 1 to 3, determining the location of the operator's gaze on the screen of the terminal device; An operation instruction of the terminal device is generated according to the location of the operator's gaze on the screen of the terminal device.

5. A device for determining the position of an operator's gaze on a terminal device screen, characterized in that: The terminal device collects the image of the operator, and the device includes: A virtual space construction module is used to determine the spatial position relationship between the operator and the terminal device according to the image of the operator; and to generate a virtual space including an operator eye model and a terminal device model according to the spatial position relationship between the operator and the terminal device; the operator eye model is pre-established to simulate the operator's eye movement, and is a collection of eye feature information when the operator's eyeballs rotate to various directions; the terminal device model has a virtual screen and a virtual camera to simulate the terminal device; The human eye movement simulation module is used to identify the image of the operator and obtain first human eye feature information; according to a preset rule, control the operator's human eye model in the virtual space to focus on a predetermined position of the terminal device model screen, determine second human eye feature information through a virtual camera, and establish a corresponding relationship between the focused position of the terminal device screen and the second human eye feature information; collect the corresponding relationship between each focused position of the terminal device screen and the corresponding second human eye feature information, and obtain the corresponding relationship between different focused positions of the terminal device model screen and the second human eye feature information; The operator target positioning module is used to find the second human eye feature information that is most similar to the first human eye feature information from the corresponding relationship; and use the focused position of the terminal device model screen corresponding to the found second human eye feature information as the landing position of the operator's gaze on the terminal device screen.

6. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for determining the position of the operator's gaze on the screen of the terminal device according to any one of claims 1 to 3 is implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an executable computer program, and when the computer program is executed by a processor, the method for determining the position of the operator's gaze on the screen of the terminal device according to any one of claims 1 to 3 is implemented.