Method and apparatus for operating computer device, computer device and storage medium

Abstract

The application relates to the field of artificial intelligence, and particularly discloses a computer device control method and device, a computer device and a storage medium. First, the effective eyeballs of an operator are acquired, and a corresponding focus point track recognition algorithm is called based on the number of effective eyeballs. Specifically, if the effective eyeballs of the operator are one, a first focus point track recognition algorithm for recognizing the focus point track of one eyeball is called; if the effective eyeballs of the operator are two, a second focus point track recognition algorithm for recognizing the focus point track of two eyeballs is called. Second, the focus point track of the gaze focus point of the operator on the display screen is acquired based on the called focus point track recognition algorithm. Finally, the change information of the focus point track is calculated, and a control command for controlling the computer device is generated based on the change information, so that the operation of the computer device is controlled through the eyes, and the hands of the operator are not required for the operation of the computer device.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular, to a computer device control method and device, computer device, and storage medium. BACKGROUND

[0002] The inventor has found that if a user's hands are in a non-idle state, or the user is a person with one hand or both hands disabled and unable to normally operate a computer device, the user may not be able to normally operate a computer device such as a mobile phone. SUMMARY

[0003] The main purpose of the present application is to provide a computer device control method and device, computer device, and storage medium, aiming to solve the technical problem of difficulty in operating a computer device when a user's hands cannot operate the computer device.

[0004] In order to achieve the above-mentioned purpose of the application, the present application provides a computer device control method applied to a computer device having a camera and a display screen, the method comprising the steps of:

[0005] acquiring binocular image information of an operator through the camera, and identifying effective eyeballs of the operator and acquiring corresponding effective eyeball information based on the binocular image information through a preset eyeball image recognition model;

[0006] calling a corresponding focal point trajectory recognition algorithm according to the effective eyeball information, wherein the focal point trajectory recognition algorithm is an algorithm for calculating a focal point trajectory on the display screen when the effective eyeballs gaze at the display screen, and includes a first focal point trajectory recognition algorithm for identifying a focal point trajectory of one eyeball and a second focal point trajectory recognition algorithm for identifying a focal point trajectory of two eyeballs;

[0007] calculating a focal point trajectory of a gaze focal point of the operator on the display screen through the focal point trajectory recognition algorithm;

[0008] acquiring change information of the focal point trajectory, and generating a control command for controlling the computer device according to the change information.

[0009] The present application also provides a computer device control device applied to a computer device having a camera and a display screen, the device comprising:

[0010] an acquisition and judgment unit for acquiring binocular image information of an operator through the camera, and identifying effective eyeballs of the operator and acquiring corresponding effective eyeball information based on the binocular image information through a preset eyeball image recognition model;

[0011] The calling unit is configured to call a corresponding focal point trajectory identification algorithm according to the effective eyeball information, wherein the focal point trajectory identification algorithm is an algorithm for calculating a focal point trajectory of an effective eyeball gaze on the display screen, and the focal point trajectory identification algorithm includes a first focal point trajectory identification algorithm for identifying a focal point trajectory of one eyeball and a second focal point trajectory identification algorithm for identifying focal point trajectories of two eyeballs.

[0012] The trajectory calculation unit is configured to calculate the focal point trajectory of the gaze focal point of the operator on the display screen by using the focal point trajectory identification algorithm.

[0013] The generation unit is configured to acquire change information of the focal point trajectory and generate a control command for controlling the computer device according to the change information.

[0014] The present application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method according to any one of the preceding embodiments when executing the computer program.

[0015] The present application also provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the method according to any one of the preceding embodiments when executed by a processor.

[0016] The computer device control method according to the present application first acquires the effective eyeball of the operator and calls a corresponding focal point trajectory identification algorithm based on the number of effective eyeballs, specifically, if the effective eyeball of the operator is one, the first focal point trajectory identification algorithm for identifying the focal point trajectory of one eyeball is called, and if the effective eyeball of the operator is two, the second focal point trajectory identification algorithm for identifying the focal point trajectories of two eyeballs is called; secondly, the focal point trajectory of the gaze focal point of the operator on the display screen is acquired based on the called focal point trajectory identification algorithm; finally, the change information of the focal point trajectory is calculated, and the control command for controlling the computer device is generated based on the change information, thereby completing the operation of the computer device by the eyes without the need of the operator's hands to operate the computer device. Further, the present application first determines the effective eyeball of the operator and then calls a corresponding focal point trajectory identification algorithm, so that the focal point trajectory of the operator can be accurately acquired, and the accuracy of the operation of the computer device by the eyes is improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The flowchart of the computer device control method according to an embodiment of the present application;

[0018] Figure 2 The structural schematic block diagram of the computer device control apparatus according to an embodiment of the present application;

[0019] Figure 3The structural schematic block diagram of the computer device of an embodiment of the present application.

[0020] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0021] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not used to limit the present application.

[0022] Referring to Figure 1 The embodiment of the present application provides a control method of a computer device, which is applied to a computer device having a camera and a display screen, and the method comprises the following steps:

[0023] S1, acquiring the binocular image information of an operator through the camera, and identifying the effective eyeballs of the operator and acquiring the corresponding effective eyeball information through a preset eyeball image recognition model based on the binocular image information;

[0024] S2, calling the corresponding focal point trajectory recognition algorithm according to the effective eyeball information, wherein the focal point trajectory recognition algorithm is an algorithm for calculating the focal point trajectory on the display screen when the effective eyeballs gaze at the display screen, and comprises a first focal point trajectory recognition algorithm for identifying the focal point trajectory of one eyeball and a second focal point trajectory recognition algorithm for identifying the focal point trajectory of two eyeballs;

[0025] S3, calculating the focal point trajectory of the gaze focal point of the operator on the display screen through the focal point trajectory recognition algorithm;

[0026] S4, acquiring the change information of the focal point trajectory, and generating a control command for controlling the computer device according to the change information.

[0027] The computer device described above is generally a smart phone, a computer, an electronic book, a tablet computer and the like, which has a camera and a display screen and has the ability to run computer programs. The positions of the camera and the display screen are relatively fixed.

[0028] As described in the above step S1, each person has two eyes, but due to various reasons, only one of the two eyes can be normally used (such as one eye is blind, the myopia degree of one eye is much higher than that of the other eye, etc.), and the eyeball of the normally used eye is the effective eyeball. The effective eyeball can be identified by a preset eyeball image recognition model, and the effective eyeball information of one effective eyeball or both effective eyeballs can be obtained, etc. The eyeball image recognition model is a pre-trained neural network model, which can be a CNN model, etc. The pre-trained neural network model is obtained by training a large number of samples, and the samples are picture samples (the images of the blind eye are different from those of the normal eye), including a large number of positive samples representing effective eyeballs and a large number of negative samples representing invalid eyeballs. The training process is a supervised learning process, specifically: obtaining sample pictures, wherein the sample data includes positive sample pictures marked as effective eyeballs and negative sample pictures marked as invalid eyeballs; dividing the sample data into training sample data and test sample data according to a ratio of 8:2; inputting the training sample data into the preset CNN model in sequence for training to obtain a temporary CNN model; inputting the test sample data into the temporary CNN model in sequence for testing; if the accuracy of the test result reaches a preset requirement (such as the accuracy rate reaches 85% to be considered as reaching the preset requirement), the temporary CNN model is used as the above-mentioned pre-trained neural network model (eyeball image recognition model). Further, the activation function of the convolution layer of the pre-trained neural network model is the ReLU function (Rectified Linear Unit), which can improve the training speed. After determining the effective eyeball information of the operator, the focal point trajectory of the operator when gazing at the display screen can be accurately obtained in the subsequent steps.

[0029] As described in the above step S2, the focal point trajectory is the trajectory of the effective eyeball gazing at the focal point of the display screen. Because the number of effective eyeballs is different, the called focal point trajectory recognition algorithm is also different, so as to improve the recognition accuracy of the focal point. Further, if it is determined that both eyeballs of the operator are not effective eyeballs, a voice reminder is sent to inform the operator that the computer device cannot be operated in the focal point trajectory mode, etc.

[0030] As described in step S3 above, a specific focus trajectory recognition algorithm is used to obtain the operator's focus trajectory. Specifically, if both eyes are valid, the algorithm used is a second focus trajectory recognition algorithm for both eyes. This algorithm combines the positional relationship between the camera and the display screen, the deflection angle of each eye relative to the camera, the distance between each eye and the camera, and the distance between the two eyes to calculate the focus of both eyes on the display screen. The distance between each eye and the camera can be calculated using image recognition methods, or by obtaining the distance between the eye and the operator using a preset distance sensor, and then combining this with the image to calculate the distance of each eye from the display screen. The calculation of the deflection angle of each eye relative to the camera can also be achieved using existing smart camera technology. If only one eye is valid, it is only necessary to calculate the deflection angle and distance of that valid eye relative to the camera to obtain the focus trajectory of that single eye.

[0031] As described in step S4 above, the change information of the focus trajectory includes prolonged staring at a point, or the focus moving up and down or left and right. Different changes correspond to different commands; that is, when a preset change in focus trajectory is detected, a corresponding control command is generated. In this embodiment, a list is set up, which contains focus trajectory change information and commands with a one-to-one mapping relationship. The computer device continuously matches the focus trajectory change information in this list. If a match is found, a corresponding command to control the computer device is generated.

[0032] In one embodiment, before step S1, which involves acquiring binocular image information of the operator through the camera, identifying the operator's valid eyeballs based on the binocular image information using a preset eyeball image recognition model, and obtaining the corresponding valid eyeball information, the procedure includes:

[0033] The operator's lip shape information is obtained through the camera;

[0034] Determine whether the mouth shape information is a preset start-up judgment information and whether both of the operator's eyeballs are valid eyeballs;

[0035] If so, then proceed with step S1 as described above.

[0036] In the embodiment, the mouth shape information refers to information represented by a series of actions made by the operator through the mouth. In some special scenarios (quiet scenarios, noisy scenarios), the operator cannot use limbs such as hands to contact the computer device to control the computer device, and at the same time, the scenario requires quiet or noise, so the computer device cannot be controlled by using voice or other auxiliary means to enter the state of controlling the computer device by the focal point track. Therefore, the mouth shape information of the operator is used to realize the state of controlling the computer device by the focal point track. Further, the mouth shape information includes information generated by combination of multiple mouth shapes, such as combination of multiple opening and closing actions.

[0037] In one embodiment, before the step S1 of acquiring, by the camera, image information of eyes of the operator, and identifying, based on the image information of the eyes, an effective eye of the operator by a preset eye image recognition model, and acquiring corresponding effective eye information, the computer device is provided with a microphone, and the step S1 includes:

[0038] receiving first vibration information collected by the microphone, wherein the first vibration information is vibration information generated by air flow acting on the microphone;

[0039] determining whether the first vibration information is preset information for starting to identify the effective eye of the operator;

[0040] If yes, the step S1 is performed.

[0041] In the embodiment, the vibration information is vibration information generated by air flow acting on the microphone, that is, the operator can make the microphone collect corresponding vibration information by blowing air to the microphone. If the air is blown for a short time, vibration information for a short time is generated, if the air is blown for a long time, vibration information for a long time is generated, and so on. By alternately blowing air for a long time and a short time, vibration information with certain meaning is generated, so that the command of acquiring, by the camera, image information of eyes of the operator, and identifying, based on the image information of the eyes, an effective eye of the operator is generated in an active control manner. The embodiment is also applied in a scenario in which the operator cannot use limbs such as hands to contact the computer device to control the computer device, and at the same time, the scenario requires quiet or noise.

[0042] In one embodiment, before the step of generating a control command for controlling the computer device according to the change information, the method includes:

[0043] receiving second vibration information collected by the microphone;

[0044] determining, according to the second vibration information, a current operation type of the computer device; and

[0045] The step of generating a control command for operating the computer device according to the change information comprises:

[0046] calling a change information-control command mapping list corresponding to the operation type;

[0047] searching for a control command corresponding to the change information in the change information-control command mapping list, and generating the control command.

[0048] In this embodiment, when operating the computer device, different operations such as page turning and screen sliding are required. If no specific classification is made, misoperation may occur in the control process. For example, the change information of the focus track is moving from top to bottom, and the corresponding control command can be a command for sliding the screen upward or a command for turning the page upward. If no distinction is made, operation confusion may occur. If a function (sliding the screen upward) is limited, another function (turning the page upward) cannot be implemented. Therefore, the operation type is given in this embodiment, which can include a screen sliding operation type and a page turning operation type. The screen sliding operation type can include a click operation. Different operation types are provided with a corresponding change information-control command mapping list. For example, when browsing a webpage, it is generally required to slide the screen upward and downward or click the interested deep information. Therefore, when browsing the webpage, the operation type at this time is limited to the screen sliding operation type. The second vibration information is the same type of vibration information as the first vibration information, and is mainly applied in a quiet environment or a scene with relatively large noise, such as a quiet hospital room. A relatively small sound can be generated through the second vibration information to reduce the influence on others and effectively input a command. The computer device recognizes the second vibration signal, mainly through the combination of long and short signals to recognize the intention (which operation type), which has small calculation amount and high accuracy, and is relatively simple compared with picture recognition or voice recognition.

[0049] In one embodiment, before the step S1 of acquiring the image information of the operator's eyes through the camera and identifying the effective eye of the operator and acquiring the corresponding effective eye information through the preset eye image recognition model based on the image information of the eyes, the method comprises:

[0050] acquiring the facial image of the operator;

[0051] determining whether the operator wears glasses according to the facial image;

[0052] if yes, generating a prompt information requiring the operator to take off the glasses.

[0053] In the present embodiment, the judgment of whether the glasses are worn can be realized by the existing image recognition technology, which is not described herein. The accuracy of identifying whether the glasses are effective for the eyeball is high, so in the present application, it is first determined whether the operator wears glasses, and if so, the operator is reminded to take off the glasses. The above-mentioned reminding information is preferably text information or vibration information, which can be normally used in a quiet environment and will not affect others.

[0054] In one embodiment, the above-mentioned step of generating the reminding information requiring the operator to "take off the glasses" is preceded by:

[0055] obtaining brightness information of a plurality of regions specified in the face image, wherein the plurality of regions include a lens region;

[0056] judging, based on the plurality of brightness information, whether the glasses worn by the operator are glasses of a preset type;

[0057] if so, the step of generating the reminding information requiring the operator to "take off the glasses" is executed

[0058] In the embodiment, the preset type of glasses includes colored glasses, such as common sunglasses and the like. In the embodiment, the colored glasses are distinguished from conventional vision correction glasses, including myopia correction glasses, glare correction glasses and the like. If it is determined that the glasses worn by the operator are the preset type of glasses, such as sunglasses, it has a great influence on the identification of the effective eye, and at this time, the reminding information requiring the operator to "take off the glasses" is generated. If it is the conventional vision correction glasses, the reminding information of "taking off the glasses" does not need to be sent. The specific process of identifying whether the glasses worn by the operator are the preset type of glasses is as follows: obtaining first brightness information of a glasses lens area in the face image, second brightness information of a forehead area and third brightness information of a nose tip area; obtaining a first difference value between the brightness of the first brightness information and the second brightness information, and a second difference value between the brightness of the second brightness information and the third brightness information; calculating a third difference value between the first difference value and the second difference value; comparing the third difference value with a preset brightness threshold value; and if the third difference value is greater than the brightness threshold value, it is determined that the glasses worn by the operator are the preset type of glasses. The brightness information of the forehead area and the nose tip area in the face image is basically the same when there is no obstruction, and if the glasses worn by the operator are non-colored glasses, the brightness information of the lens area is also basically the same as that of the forehead area. Therefore, when the third difference value is greater than the preset brightness threshold value, it means that the brightness information difference between the first brightness information, the second brightness information and the third brightness information is large, and at this time, it is considered that the glasses worn by the operator are the preset type of glasses. The brightness threshold value is an empirical value obtained by the developer through a large number of experiments. In the embodiment, whether the glasses are the preset type of glasses is determined by brightness comparison, which has smaller calculation amount and saves calculation resources compared with the identification of the type of glasses by the CNN neural network model.

[0059] In one embodiment, a light emitting source and a light sensor are arranged on one side of the display screen of the computer device; before the step of acquiring the image information of the eyes of the operator by the camera, and identifying the effective eye of the operator based on the image information of the eyes by a preset eye image recognition model, and acquiring the corresponding effective eye information, the method further comprises:

[0060] identifying whether a command of "identifying the effective eye of the operator" is received;

[0061] if the command is identified, acquiring the light intensity value of the ambient light collected by the light sensor;

[0062] judging whether the light intensity value is less than a preset light intensity threshold value;

[0063] if the light intensity value is less than the light intensity threshold value, the light emitting source is turned on.

[0064] In the embodiment, the application is mainly applied to the scene where the environment is relatively dark, such as at night, and the operator needs to control the computer device by the focus track. Since the ambient light is relatively weak, the recognition of the focus track is affected, so the application provides a light source. When receiving the command of "recognizing the effective eyeball of the operator", the ambient light is collected. If it is determined that the ambient light is less than the preset light intensity threshold, the light source is turned on to provide auxiliary light, so that the camera can collect a clear binocular image. It should be noted that the light emitted by the light source is soft light to prevent stimulating the eyes of the operator. Further, if the light intensity value is less than the light intensity threshold, the step of turning on the light source includes: calling a preset light source light list, wherein the light source list records a plurality of light intensity value ranges and output light intensity values corresponding to different light intensity value ranges; matching the light intensity range of the light intensity value in the light source light list, and then controlling the light source to emit the output light intensity value corresponding to the light intensity range.

[0065] In one embodiment, the camera described above includes an infrared camera, and the light source is an infrared light source. The clarity of the binocular image information in a dark environment can be improved. The camera can include an infrared camera and a visible light camera, and the infrared camera can improve the clarity of the image in the dark.

[0066] The computer device control method of the application first acquires the effective eyeball of the operator, and calls the corresponding focus track recognition algorithm based on the number of effective eyeballs. Specifically, if the effective eyeball of the operator is one, the first focus track recognition algorithm for recognizing the focus track of one eyeball is called, and if the effective eyeball of the operator is two, the second focus track recognition algorithm for recognizing the focus track of two eyeballs is called. Secondly, the focus track of the gaze focus of the operator on the display screen is acquired based on the called focus track recognition algorithm. Finally, the change information of the focus track is calculated, and the control command for controlling the computer device is generated based on the change information, so as to complete the operation of controlling the computer device by the eyes without the need of the operator to operate the computer device with both hands. Further, the application first determines the effective eyeball of the operator, and then calls the corresponding focus track recognition algorithm, so as to accurately acquire the focus track of the operator and improve the accuracy of operating the computer device by the eyes.

[0067] Reference Figure 2 The application also provides a computer device control device applied to a computer device, wherein the computer device has a camera and a display screen, and the device includes:

[0068] The acquisition judgment unit 10 is configured to acquire the binocular image information of the operator through the camera, and identify the effective eyeballs of the operator based on the binocular image information through a preset eyeball image recognition model, and acquire corresponding effective eyeball information.

[0069] The calling unit 20 is configured to call a corresponding focal point trajectory recognition algorithm according to the effective eyeball information, wherein the focal point trajectory recognition algorithm is an algorithm for calculating the focal point trajectory on the display screen when the effective eyeball gazes at the display screen, and includes a first focal point trajectory recognition algorithm for identifying the focal point trajectory of one eyeball and a second focal point trajectory recognition algorithm for identifying the focal point trajectory of two eyeballs.

[0070] The trajectory calculation unit 30 is configured to acquire the effective eyeball information of the operator through the camera, and calculate the focal point trajectory of the gaze focal point of the operator on the display screen by using the called focal point trajectory recognition algorithm.

[0071] The acquisition generation unit 40 is configured to acquire the change information of the focal point trajectory, and generate a control command for operating the computer device according to the change information.

[0072] Further, the operating device of the computer device further comprises:

[0073] The mouth shape acquisition unit is configured to acquire the mouth shape information of the operator through the camera.

[0074] The first judgment unit is configured to judge whether the mouth shape information is preset information for starting to identify the effective eyeballs of the operator.

[0075] The first execution unit is configured to start the acquisition judgment unit 10 to acquire the binocular image information of the operator through the camera, and identify the effective eyeballs of the operator based on the binocular image information through a preset eyeball image recognition model, and acquire corresponding effective eyeball information.

[0076] Further, a microphone is arranged on the computer device, and the operating device of the computer device further comprises:

[0077] The first receiving unit is configured to receive first vibration information collected by the microphone, wherein the first vibration information is vibration information generated by air flow acting on the microphone.

[0078] The second judgment unit is configured to judge whether the first vibration information is preset information for starting to identify the effective eyeballs of the operator.

[0079] The second execution unit is configured to, if the first vibration information is preset start information of "identifying the valid eye of the operator", start the acquisition judgment unit 10 to acquire the binocular image information of the operator through the camera, and identify the valid eye of the operator based on the binocular image information through a preset eye image recognition model, and acquire the corresponding valid eye information.

[0080] Further, the operation device of the computer device further comprises:

[0081] The second receiving unit is configured to receive second vibration information collected by the microphone.

[0082] The type determination unit is configured to determine the current operation type of the computer device according to the second vibration information.

[0083] The acquisition generation unit 40 comprises:

[0084] The calling module is configured to call a change information-control command mapping list corresponding to the operation type.

[0085] The searching generation module is configured to search for a control command corresponding to the change information in the change information-control command mapping list, and generate the control command.

[0086] Further, the operation device of the computer device further comprises:

[0087] The image acquisition unit is configured to acquire a face image of the operator.

[0088] The third judgment unit is configured to judge whether the operator wears glasses according to the face image.

[0089] The generation reminding unit is configured to generate reminding information requiring the operator to "take off the glasses" if the operator wears glasses.

[0090] Further, the operation device of the computer device further comprises:

[0091] The brightness acquisition unit is configured to acquire brightness information of a plurality of regions specified in the face image, wherein the plurality of regions include a lens region.

[0092] The preset judgment unit is configured to judge whether the glasses worn by the operator are glasses of a preset type based on a plurality of the brightness information.

[0093] The reminding execution unit is configured to start the generation reminding unit to generate reminding information requiring the operator to "take off the glasses".

[0094] Further, the display screen of the computer device is provided with a light source and a photosensor; the control device of the computer device further comprises:

[0095] a command identifying unit configured to identify whether a command of "identifying valid eye balls of the operator" is received;

[0096] a light intensity acquiring unit configured to acquire a light intensity value of ambient light collected by the photosensor if the command is identified;

[0097] a light intensity judging unit configured to judge whether the light intensity value is less than a preset light intensity threshold value;

[0098] a light source starting unit configured to start the light source if the light intensity value is less than the light intensity threshold value.

[0099] The control device of the computer device comprises units and modules for executing the control method of the computer device, which will not be described one by one.

[0100] Referring to Figure 3 , the embodiment of the present application further provides a computer device which can be a server, and the internal structure of the computer device can be as shown in Figure 3 . The computer device comprises a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store data such as a focus trajectory recognition algorithm and a change information-control command mapping list. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a control method of a computer device.

[0101] The processor executes the control method of the computer device, and is applied to a computer device having a camera and a display screen, and the method comprises the steps of:

[0102] acquiring image information of eyes of an operator through the camera, and identifying valid eye balls of the operator and acquiring corresponding valid eye ball information based on the image information of the eyes through a preset eye ball image recognition model;

[0103] According to the effective eye information, a corresponding focus point track recognition algorithm is called, wherein the focus point track recognition algorithm is an algorithm for calculating a focus point track on the display screen when the effective eye gazes at the display screen, and the focus point track recognition algorithm includes a first focus point track recognition algorithm for recognizing a focus point track of one eye and a second focus point track recognition algorithm for recognizing focus point tracks of two eyes;

[0104] A focus point track of a gaze focus point of the operator on the display screen is calculated through the focus point track recognition algorithm;

[0105] Change information of the focus point track is acquired, and a control command for operating the computer device is generated according to the change information.

[0106] The embodiment of the application further provides a computer readable storage medium, which has a computer program stored thereon, and the computer program is executed by a processor to implement a computer device operating method, and is applied to a computer device having a camera and a display screen, and the method comprises the steps of:

[0107] Double eye image information of an operator is acquired through the camera, and effective eyes of the operator are recognized and corresponding effective eye information is acquired through a preset eye image recognition model based on the double eye image information;

[0108] According to the effective eye information, a corresponding focus point track recognition algorithm is called, wherein the focus point track recognition algorithm is an algorithm for calculating a focus point track on the display screen when the effective eye gazes at the display screen, and the focus point track recognition algorithm includes a first focus point track recognition algorithm for recognizing a focus point track of one eye and a second focus point track recognition algorithm for recognizing focus point tracks of two eyes;

[0109] A focus point track of a gaze focus point of the operator on the display screen is calculated through the focus point track recognition algorithm;

[0110] Change information of the focus point track is acquired, and a control command for operating the computer device is generated according to the change information.

[0111] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiment methods can be included. Any reference to memory, storage, databases, or other media in this application and in examples provided herein, unless specifically stated otherwise, can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0112] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, device, article, or method that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, device, article, or method. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, device, article, or method that includes the element.

[0113] The above description is only the preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A method for controlling a computer device, applied to a computer device having a camera and a display screen, characterized in that, The method comprises the steps of: obtaining image information of the operator's eyes through the camera, and identifying the effective eyes of the operator and obtaining corresponding effective eye information based on the image information of the eyes through a preset eye image recognition model, wherein the effective eyes are eyes that can be normally used; calling a corresponding focus point trajectory recognition algorithm according to the effective eye information, wherein the focus point trajectory recognition algorithm is an algorithm for calculating the focus point trajectory on the display screen when the effective eyes are gazing at the display screen, and the focus point trajectory recognition algorithm includes a first focus point trajectory recognition algorithm for identifying the focus point trajectory of one eye and a second focus point trajectory recognition algorithm for identifying the focus point trajectories of two eyes; the first focus point trajectory recognition algorithm includes calculating the focus point trajectory of the one eye through the deflection angle and distance parameters of the effective eye and the camera; and the second focus point trajectory recognition algorithm includes calculating the focus points of the two eyes on the display screen by combining the interpupillary distance, the respective deflection angles and the distance parameters from the screen; calculating the focus point trajectory of the gazing focus point of the operator on the display screen through the focus point trajectory recognition algorithm; obtaining change information of the focus point trajectory, and generating a control command for controlling the computer device according to the change information.

2. The manipulation method of a computer device according to claim 1, wherein, Before the step of obtaining image information of the operator's eyes through the camera, and identifying the effective eyes of the operator and obtaining corresponding effective eye information based on the image information of the eyes through a preset eye image recognition model, the method comprises the steps of: obtaining mouth shape information of the operator through the camera; determining whether the mouth shape information is preset information for starting the identification of the effective eyes of the operator; if yes, performing the step of obtaining image information of the operator's eyes through the camera, and identifying the effective eyes of the operator and obtaining corresponding effective eye information based on the image information of the eyes through a preset eye image recognition model.

3. The method of claim 1, wherein, The computer device is provided with a microphone; before the step of obtaining image information of the operator's eyes through the camera, and identifying the effective eyes of the operator and obtaining corresponding effective eye information based on the image information of the eyes through a preset eye image recognition model, the method comprises the steps of: receiving first vibration information collected by the microphone, wherein the first vibration information is vibration information generated by air flow acting on the microphone; determining whether the first vibration information is preset information for starting the identification of the effective eyes of the operator; if yes, performing the step of obtaining image information of the operator's eyes through the camera, and identifying the effective eyes of the operator and obtaining corresponding effective eye information based on the image information of the eyes through a preset eye image recognition model.

4. The method of claim 3, wherein, Before the step of generating a control command for controlling the computer device according to the change information, the method comprises the steps of: receiving second vibration information collected by the microphone; determining the current operation type of the computer device according to the second vibration information; and The step of generating a control command for controlling the computer device according to the change information comprises the steps of: retrieve a change information-control command mapping list corresponding to the operation type; find a control command corresponding to the change information in the change information-control command mapping list, and generate the control command.

5. The method of claim 1, wherein, Before the step of acquiring the binocular image information of the operator through the camera, and identifying the valid eyeballs of the operator based on the binocular image information through a preset eyeball image recognition model, and acquiring corresponding valid eyeball information, the method further comprises: acquiring a face image of the operator; determining whether the operator wears glasses according to the face image; if so, generating a prompt information requiring the operator to "take off the glasses".

6. The method of claim 5, wherein, Before the step of generating the prompt information requiring the operator to "take off the glasses", the method further comprises: acquiring brightness information of a plurality of regions specified in the face image, wherein the plurality of regions include a lens region; determining whether the glasses worn by the operator are glasses of a preset type based on the plurality of brightness information; if so, performing the step of generating the prompt information requiring the operator to "take off the glasses".

7. The method according to any one of claims 1-6, wherein, The display screen of the computer device is provided with a light source and a photosensitive receptor; before the step of acquiring the binocular image information of the operator through the camera, and identifying the valid eyeballs of the operator based on the binocular image information through a preset eyeball image recognition model, and acquiring corresponding valid eyeball information, the method further comprises: identifying whether a command of "identifying the valid eyeballs of the operator" is received; if the command is identified, acquiring a light intensity value of ambient light collected by the photosensitive receptor; determining whether the light intensity value is less than a preset light intensity threshold value; if the light intensity value is less than the light intensity threshold value, turning on the light source.

8. A manipulation device of a computer device, applied to a computer device, the computer device having a camera and a display screen, characterized in that, The device comprises: an acquisition determining unit configured to acquire binocular image information of an operator through a camera, and identify valid eyeballs of the operator based on the binocular image information through a preset eyeball image recognition model, and acquire corresponding valid eyeball information, wherein the valid eyeballs are eyeballs that can be normally used; a calling unit configured to call a corresponding focal point trajectory recognition algorithm according to the valid eyeball information, wherein the focal point trajectory recognition algorithm is an algorithm for calculating a focal point trajectory on a display screen when the valid eyeballs gaze at the display screen, and the focal point trajectory recognition algorithm comprises a first focal point trajectory recognition algorithm for identifying a focal point trajectory of one eyeball and a second focal point trajectory recognition algorithm for identifying focal points of two eyeballs on the display screen; the first focal point trajectory recognition algorithm comprises calculating the focal point trajectory of the one eyeball through a deflection angle and a distance parameter of the valid eyeball and the camera; and the second focal point trajectory recognition algorithm comprises calculating the focal points of the two eyeballs on the display screen by combining a binocular pupil distance, respective deflection angles and a distance parameter from the screen; a trajectory calculating unit configured to calculate a focal point trajectory of a gaze focal point of the operator on the display screen through the focal point trajectory recognition algorithm; an acquisition generating unit configured to acquire change information of the focal point trajectory, and generate a control command for operating the computer device according to the change information. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images