Computer interface control method, device, equipment and storage medium

Abstract

The present invention discloses a computer interface control method, device, equipment and storage medium. Specifically, the method is to obtain the image area and eye control activity range of the user, determine the control mode according to the image area and eye control activity range of the user, wherein the control mode is any one of the head control mode and the eye control mode, and then control the computer interface according to the determined control mode. In this way, when only one device is needed, the user can flexibly switch between the head control mode and the eye control mode according to the actual use situation to assist in controlling the computer interface.

Description

Technical Field

[0001] The embodiments of the present invention relate to information processing technology, and more particularly to a computer interface control method, device, equipment and storage medium. Background Art

[0002] For some special groups of people who have lost their ability to speak or whose limbs cannot move, they usually need to use instruments to assist in controlling a computer to communicate, for example, using an eye-control device to assist in controlling a computer, or using a head-control device to assist in controlling a computer.

[0003] However, the use of a head-controlled device requires the movement of the neck to control the computer. When the user's neck is not flexible, it will affect the control of the head-controlled device by the head movement. The use of an eye-controlled device requires ensuring that the calibration position and the usage position remain relatively consistent for a long time to obtain better use effects. However, most users are unable to maintain a relatively fixed position for a long time, resulting in deviation from the calibration position, which affects the use effect. Summary of the invention

[0004] The present invention provides a computer interface control method, device, equipment and storage medium, which can enable a user to flexibly switch between a head control mode and an eye control mode according to actual usage conditions to assist in controlling a computer interface, while only requiring one device.

[0005] In a first aspect, an embodiment of the present invention provides a computer interface control method, the method comprising:

[0006] Obtain the user's image area and eye control activity range;

[0007] Determine a control mode according to the image area and eye control activity range of the user, the control mode being any one of a head control mode and an eye control mode;

[0008] Control the computer interface according to the control method.

[0009] In a second aspect, an embodiment of the present invention provides a computer interface control device, the device comprising:

[0010] An acquisition module, used to acquire the image area and eye control activity range of the user;

[0011] A determination module, used to determine a control mode according to the image area and eye control activity range of the user, where the control mode is any one of a head control mode and an eye control mode;

[0012] The control module is used for controlling the computer interface according to the control mode.

[0013] In a third aspect, an embodiment of the present invention further provides a device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a computer interface control method as provided in an embodiment of the present invention is implemented.

[0014] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer interface control method provided in the embodiment of the present invention is implemented.

[0015] The embodiments of the present invention provide a computer interface control method, device, equipment and storage medium, specifically, after obtaining the image area and eye control activity range of the user, a control mode is determined according to the image area and eye control activity range of the user, wherein the control mode is any one of the head control mode and the eye control mode, and then the computer interface is controlled according to the determined control mode. In this way, when only one device is needed, the user can flexibly switch between the head control mode and the eye control mode according to the actual usage to assist in controlling the computer interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a computer interface control method in an embodiment of the present invention;

[0017] Figure 2 is a schematic diagram of the range of head movement in an embodiment of the present invention;

[0018] Figure 3 is a schematic diagram of a user's image area exceeding the head movement range by a first preset distance to the left in an embodiment of the present invention;

[0019] Figure 4 is a schematic diagram of an image area of ​​a user in an embodiment of the present invention that exceeds the head movement range by a first preset distance to the right;

[0020] Figure 5 is a schematic diagram of a user's image area extending upward beyond the head activity range by a first preset distance in an embodiment of the present invention;

[0021] Figure 6 is a schematic diagram of a user's image area extending downward beyond the head movement range by a first preset distance in an embodiment of the present invention;

[0022] Figure 7 is a schematic diagram of an image area of ​​a user close to a camera in an embodiment of the present invention;

[0023] Figure 8 is a schematic diagram of an image area of ​​a user in an embodiment of the present invention being far away from a camera;

[0024] Fig. 9 is a schematic diagram of a user leaning to the left in an embodiment of the present invention;

[0025] Fig.10 is a schematic diagram of a user leaning to the right in an embodiment of the present invention;

[0026] Fig.11 is a schematic diagram of a computer interface control device in an embodiment of the present invention;

[0027] Fig.12 It is a schematic diagram of the structure of the device in the embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0029] In addition, in the embodiments of the present application, words such as "optionally" or "exemplarily" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "optionally" or "exemplarily" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "optionally" or "exemplarily" is intended to present related concepts in a specific way.

[0030] The embodiments of the present invention can be applied to scenarios where both eye control devices and head control devices are not required. Flexible switching between eye control mode and head control mode can be achieved through only one interactive device according to the actual usage of the user to assist in controlling the computer interface.

[0031] Figure 1 A flowchart of a computer interface control method provided by an embodiment of the present invention, the method can be applied to an interactive device capable of switching between a head control mode and an eye control mode to assist in controlling a computer interface, such as Figure 1 As shown, the method specifically comprises the following steps:

[0032] S101. Obtain an image area and an eye control activity range of a user.

[0033] In this embodiment, the image area of ​​the user can be understood as an image area captured by a camera that includes the user's head and in which the user's head can move arbitrarily within a certain range.

[0034] The eye control activity range can be understood as a specific area in the user's image area where the user's pupil can move arbitrarily.

[0035] It should be noted that the area involved in the eye control activity range is smaller than the image area of ​​the user, that is, the eye control activity range is included in the image area of ​​the user.

[0036] In one example, the method for obtaining the user's image area and eye control activity range in the above step S101 can be that after turning on the device, the camera captures an image including the user's head area, and performs eye control calibration at the same time. Then, the image area including the user's head captured by the camera is the user's image area, and after performing eye control calibration in the user's image area, a specific area obtained is the user's eye control activity range.

[0037] Exemplarily, the method for obtaining the eye control activity range is that after turning on the device, the camera can capture an image area including the location of the user's pupil. When the camera detects that the user's pupil maintains a constant coordinate position within a preset time (such as 5 seconds), a specific area including the user's pupil captured by the camera is used as the eye control activity range.

[0038] Optionally, the camera may be a camera on a camera.

[0039] For example, the eye control activity range may be a polygonal area with the midpoint of the line connecting the left and right pupils of the user as the center point and the foot line as a preset length, or an ellipse area with the left and right pupils of the user as the focus. That is, as long as the user's pupils are within the area, it is the user's eye control activity range.

[0040] In this embodiment, if Figure 2 As shown, the area formed by a rectangle with the midpoint of the line connecting the left and right pupils of the user as the center point and the foot line as the preset length is used as the eye control activity range. Of course, the eye control activity range can also be other shapes.

[0041] It should be noted that the camera may be a part of the interactive device or a separate device independent of the interactive device.

[0042] S102: Determine a control method according to the image area and eye control activity range of the user.

[0043] Exemplarily, the above control mode can be any one of a head control mode and an eye control mode. Further, the head control mode can be understood as a mode in which the user controls by moving the head with the neck, and the eye control mode can be understood as a mode in which the user controls by moving the pupil.

[0044] In this step, the user's eye position is identified from the user's image area, and a control mode can be selected from the head control mode and the eye control mode to control the computer interface according to the position of the user's eye position in the eye control activity range.

[0045] For example, when the user's eye position is within the eye control activity range, the control mode is determined to be the eye control control mode; when the user's eye position is not within the eye control activity range, that is, the user's eyes are outside the eye control activity range, the control mode is determined to be the head control control mode.

[0046] S103, controlling the computer interface according to the control method.

[0047] After determining a control mode between the head control mode and the eye control mode based on the above step S102, the computer interface is controlled according to the control mode.

[0048] Through the above method, after obtaining the image area and eye control activity range of the user, the control mode is determined according to the image area and eye control activity range of the user, wherein the control mode is any one of the head control mode and the eye control mode, and then the computer interface is controlled according to the determined control mode. In this way, when only one device is needed, the user can flexibly switch between the head control mode and the eye control mode according to the actual usage to assist in controlling the computer interface.

[0049] In addition, in the process of obtaining the user's eye control activity range in step S101, after the device is turned on and the eye control activity range is initially determined, the eye control activity range can also be updated in real time based on the image area of ​​the user captured by the camera while the user is using the interactive device. That is, as the user's usage habits change, the user maintains a fixed comfortable way during the use of the interactive device, and the user's pupil position captured by the camera does not change within a preset time, then the image area captured by the camera is used as the eye control activity range.

[0050] Furthermore, in the above step S102, the method of determining whether the user's eye position is within the eye control activity range can be to identify the user's eye position from the user's image area, and when the user's eye position does not meet a preset condition, determine that the user's eye position is within the eye control activity range.

[0051] For example, the preset condition may be that the user's eye position exceeds the eye control activity range by a first preset distance, such as Figure 3 — Figure 6 As shown, when the user moves left or right or up or down and causes the position of his or her eye to exceed the eye control activity range by a first preset distance to the left or right, or up or down, causing one of the user's pupils to exceed the eye control activity range, and the camera cannot accurately obtain the coordinates of the pupil that is out of range, it is determined that the user's eye position meets the preset conditions and the user's eye position is not within the eye control activity range.

[0052] Alternatively, the above-mentioned preset condition may be that the pupil distance of the user in the user's image area and the pupil distance of the user calibrated in the eye control activity range satisfy a first judgment condition, wherein the first judgment condition is that the absolute value of the length difference between the pupil distance of the user in the user's image area and the pupil distance of the user calibrated in the eye control activity range is greater than or equal to a first threshold, or that the length ratio of the pupil distance of the user in the user's image area to the pupil distance of the user calibrated in the eye control activity range is greater than or equal to a second threshold or less than or equal to a third threshold.

[0053] like Figure 7 As shown, when the user's activity causes his face to approach the camera, the user's pupil distance in the image area of ​​the user captured by the camera becomes larger, and the pupil distance is different from the user's pupil distance calibrated in the eye control activity range (such as Figure 2 If the length difference between the pupil distance and the user's pupil distance calibrated in the eye control activity range is too large, for example, greater than or equal to the first threshold, or the length ratio of the pupil distance to the user's pupil distance calibrated in the eye control activity range exceeds a certain ratio range, for example, the ratio is greater than or equal to the second threshold or less than or equal to the third threshold, then it can be determined that the user's eye position is beyond the eye control activity range, that is, the user's image area meets the preset conditions.

[0054] like Figure 8 As shown, when the user moves away from the camera, the pupil distance of the user in the image area of ​​the user captured by the camera becomes smaller, resulting in a large difference between the pupil distance and the length of the user's pupil distance calibrated in the eye control activity range, and the absolute value of the difference is greater than or equal to the first threshold, or the ratio of the pupil distance to the length of the user's pupil distance calibrated in the eye control activity range exceeds a certain ratio range, and the ratio is greater than or equal to the second threshold or less than or equal to the third threshold, then it is determined that the image area of ​​the user captured by the camera at this time meets the preset condition.

[0055] It should be noted that when obtaining the above ratio, the pupil distance of the user in the user's image area can be ratioed with the pupil distance of the user calibrated in the eye control activity range, or the pupil distance of the user calibrated in the eye control activity range can be ratioed with the pupil distance of the user in the user's image area.

[0056] For example, assuming that the pupil distance of the user in the image area of ​​the user is compared with the pupil distance of the user calibrated in the eye control activity range, then Figure 7 In the case shown, when the user approaches the camera, the pupil distance of the user in the image area of ​​the user captured by the camera is larger, resulting in a larger ratio of the pupil distance to the pupil distance of the user calibrated in the eye control activity range. When the ratio is greater than or equal to the second threshold, it indicates that the user's eye position is beyond the eye control activity range and the user's image area meets the preset conditions.

[0057] Similarly, in Figure 8In the case shown, when the user is far away from the camera, the pupil distance of the user in the image area of ​​the user captured by the camera is relatively small, resulting in a relatively small ratio between the pupil distance and the user pupil distance calibrated in the eye control activity range. When the ratio is less than or equal to the third threshold, it means that the pupil in the captured image area of ​​the user may be relatively small or even unclear. At this time, the user's eye position also meets the preset condition. In other words, whether the user is far away from the camera or close to the camera, as long as the ratio between the user's pupil distance in the image area of ​​the user and the user pupil distance calibrated in the eye control activity range satisfies any one of the conditions of being greater than or equal to the second threshold or less than or equal to the third threshold, it can be determined that the user's eye position meets the preset condition.

[0058] In another example, the preset condition may be that the user's pupil coordinates in the user's image area and the user's pupil coordinates calibrated in the eye control activity range meet a second judgment condition, wherein the second judgment condition is that the absolute value of the angle between the line connecting the pupil coordinates in the user's image area and the line connecting the user's pupil coordinates calibrated in the eye control activity range is greater than or equal to a fourth threshold.

[0059] like Fig. 9 , Fig.10 As shown in FIG. 1 , when the user's head tilts to the left or right, the line connecting the user's pupil coordinates in the image area of ​​the user captured by the camera and Figure 2 If the angle between the lines connecting the calibrated user pupil coordinates in the middle head movement range exceeds a certain range, it can be considered that the user's eye position meets the preset conditions.

[0060] Among them, the absolute value of the above-mentioned angle can be understood as that when the user's head is tilted to the left, the angle between the line between the user's pupil coordinates in the image area and the line between the user's pupil coordinates calibrated in the eye control activity range is positive; when the user's head is tilted to the right, the angle between the line between the user's pupil coordinates in the user's image area and the line between the user's pupil coordinates calibrated in the eye control activity range is negative; regardless of whether the user's head is tilted to the left or to the right, as long as the absolute value of the angle is greater than or equal to the fourth threshold, it can be considered that the user's image area meets the preset conditions.

[0061] Of course, the positive and negative values ​​of the above-mentioned angle can be set by those skilled in the art according to actual needs, and this embodiment does not limit this. Similarly, the specific values ​​of the first threshold, the second threshold, the third threshold, and the fourth threshold involved in this embodiment can also be set by those skilled in the art according to actual needs, and this embodiment does not limit this.

[0062] By combining the above preset conditions to determine the eye position of the user in the image area, the control method to be adopted by the user can be effectively determined. For example, when the eye position of the user meets the above preset conditions, it is considered that the eye position of the user is not within the eye control activity range, and the head control method is used to operate the computer interface. On the contrary, when the eye position of the user does not meet the above preset conditions, it is considered that the eye position of the user is within the eye control activity range, and the eye control method is used to operate the computer interface. In this way, the two control methods can be flexibly switched according to the actual usage of the user, so that the user can effectively operate the computer interface through one interactive device without the need for two devices, an eye control device and a head control device.

[0063] Fig.11 A computer interface control device is provided in one embodiment, such as Fig.11 As shown, the device includes: an acquisition module 1101, a determination module 1102, and a control module 1103;

[0064] The acquisition module is used to acquire the image area and eye control activity range of the user;

[0065] A determination module, used to determine a control mode according to the image area and eye control activity range of the user, wherein the control mode is any one of a head control mode and an eye control mode;

[0066] The control module is used for controlling the computer interface according to the control mode.

[0067] Among them, the above-mentioned eye control activity range is an area composed of a polygon with the midpoint of the line connecting the user's left and right pupil coordinates as the center point and a diagonal line of a preset length, or an area composed of an ellipse with the user's left and right pupil coordinates as the focus.

[0068] Furthermore, the above-mentioned determination module is used to identify the user's eye position from the user's image area, and when the user's eye position is within the eye control activity range, the control mode is determined to be the eye control control mode; when the user's eye position is not within the eye control activity range, the control mode is determined to be the head control mode.

[0069] The above-mentioned determination module is also used to determine that the eye position of the user is within the eye control activity range when the eye position of the user does not meet the preset conditions.

[0070] The preset condition is that the user's eye position exceeds the eye control activity range by a first preset distance;

[0071] Alternatively, the preset condition is that the pupil distance of the user in the image area of ​​the user and the pupil distance of the user calibrated in the eye control activity range meet the first judgment condition;

[0072] Alternatively, the preset condition is that the user pupil coordinates in the image area of ​​the user and the user pupil coordinates calibrated in the eye control activity range meet the second judgment condition.

[0073] The above-mentioned first judgment condition is that the absolute value of the difference in length between the pupil distance of the user in the user's image area and the pupil distance of the user calibrated in the eye control activity range is greater than or equal to the first threshold, or the ratio of the length of the pupil distance of the user in the user's image area to the pupil distance of the user calibrated in the eye control activity range is greater than or equal to the second threshold or less than or equal to the third threshold.

[0074] The second judgment condition is that the absolute value of the angle between the pupil coordinate line in the user's image area and the user's pupil coordinate line calibrated in the eye control activity range is greater than or equal to a fourth threshold.

[0075] Fig.11 The computer interface control device provided can execute Figure 1 The provided computer interface control method has functional modules and beneficial effects corresponding to the execution method.

[0076] Fig.12 A device is provided in an embodiment, such as Fig.12 As shown, the device includes: a processor 1201, a memory 1202, an input device 1203, and an output device 1204; the number of processors 1201 in the device can be one or more. Fig.12 A processor 1201 is taken as an example; the processor 1201, the memory 1202, the input device 1203, and the output device 1204 in the device can be connected by a bus or other means. Fig.12 The example of connecting through bus is taken in the following.

[0077] The memory 1202 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the computer interface control method in the embodiment of the present invention (for example, the acquisition module 1101, the determination module 1102, and the control module 1103 in the computer interface control device). The processor 1201 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 1202, that is, implements the above-mentioned computer interface control method.

[0078] The memory 1202 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 1202 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 1202 may further include a memory remotely arranged relative to the processor 901, and these remote memories may be connected to the device / terminal / server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0079] The input device 1203 may be used to receive input digital or character information and generate key signal input related to user settings and function control. The output device 1204 may include display devices such as a display screen.

[0080] An embodiment of the present invention further provides a storage medium containing computer executable instructions, wherein the computer executable instructions are used to execute a computer interface control method when executed by a computer processor, the method comprising:

[0081] Obtain the user's image area and eye control activity range;

[0082] Determining a control mode according to the image area and eye control activity range of the user, wherein the control mode is any one of a head control mode and an eye control mode;

[0083] Control the computer interface according to the control method.

[0084] Of course, the computer executable instructions of a storage medium containing computer executable instructions provided in an embodiment of the present invention are not limited to the operations of the method described above, and can also execute related operations in the computer interface control method provided in any embodiment of the present invention.

[0085] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0086] It is worth noting that in the embodiment of the above-mentioned computer interface control device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0087] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A computer interface control method, characterized in that: include: Acquire the image area and eye control activity range of the user; wherein the image area of ​​the user is an image area captured by a camera that includes the user's head and where the user's head moves arbitrarily within a certain range; the eye control activity range is a specific area in the user's image area where the user's pupil moves arbitrarily; Determining a control mode according to the image area and eye control activity range of the user, the control mode being any one of a head control mode and an eye control mode; controlling the computer interface according to the control method; The determining of the control mode according to the image area and eye control activity range of the user includes: identifying the eye position of the user from the image area of ​​the user, and determining that the control mode is the eye control mode when the eye position of the user is within the eye control activity range; When the eye position of the user is not within the eye control activity range, the control mode is determined to be the head control mode.

2. The method according to claim 1, characterized in that The eye control activity range is an area formed by a polygon with the midpoint of the line connecting the coordinates of the left and right pupils of the user as the center point and the diagonal line being a preset length; Alternatively, the eye control activity range is an elliptical area with the left and right pupil coordinates of the user as foci.

3. The method according to claim 1, characterized in that Determining that the eye position of the user is within the eye control activity range includes: When the eye position of the user does not meet a preset condition, it is determined that the eye position of the user is within the eye control activity range.

4. The method according to claim 3, characterized in that The preset condition is that the eye position of the user exceeds the eye control activity range by a first preset distance; Alternatively, the preset condition is that the pupil distance of the user in the image area of ​​the user and the pupil distance of the user calibrated in the eye control activity range meet the first judgment condition; Alternatively, the preset condition is that the user pupil coordinates in the image area of ​​the user and the user pupil coordinates calibrated in the eye control activity range meet the second judgment condition.

5. The method according to claim 4, characterized in that The first judgment condition is that an absolute value of a difference between a user's pupil distance in the image area of ​​the user and a user's pupil distance calibrated in the eye control activity range is greater than or equal to a first threshold; Alternatively, the first judgment condition is that the ratio of the pupil distance of the user in the image area of ​​the user to the pupil distance of the user calibrated in the eye control activity range is greater than or equal to a second threshold or less than or equal to a third threshold.

6. The method according to claim 4, characterized in that The second judgment condition is that the absolute value of the angle between the line connecting the pupil coordinates in the image area of ​​the user and the line connecting the pupil coordinates of the user calibrated in the eye control activity range is greater than or equal to a fourth threshold.

7. A computer interface control device, characterized in that: include: The acquisition module is used to acquire the image area and eye control activity range of the user; wherein the image area of ​​the user is an image area captured by a camera that includes the user's head and where the user's head moves arbitrarily within a certain range; the eye control activity range is a specific area in the user's image area where the user's pupil moves arbitrarily; A determination module, used to determine a control mode according to the image area and eye control activity range of the user, wherein the control mode is any one of a head control mode and an eye control mode; A control module, used for controlling the computer interface according to the control method; The determination module is used to identify the user's eye position from the user's image area, and when the user's eye position is within the eye control activity range, determine that the control mode is the eye control control mode; when the user's eye position is not within the eye control activity range, determine that the control mode is the head control mode.

8. A device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the computer interface control method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer interface control method according to any one of claims 1 to 6 is implemented.

Images