Interaction control method, electronic equipment and storage medium

Abstract

The invention relates to the field of computers, and provides an interaction control method, electronic equipment and a storage medium. The method comprises the steps of determining a first coordinate of an eyeball of a user on a camera coordinate system based on an eye image of the user, and determining a deviation angle of the eyeball; determining a second coordinate of the eyeball on a world coordinate system based on the first coordinate, the deviation angle and the distance between the user and the terminal device; according to the second coordinate, the sight line image of the eyeball and an imaging area of the terminal equipment in the sight line image, determining a sight line drop point position of the eyeball; and executing control operation on the terminal equipment according to the eye movement characteristic parameters of the user on the sight drop point position. The method can improve the interaction control efficiency and accuracy.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to an interaction control method, an electronic device, and a storage medium. Background Art

[0002] Currently, the interaction control of a terminal device can be achieved through Augmented Reality (AR) technology. In related technologies, the operation of the terminal device is usually controlled by voice commands, gesture recognition, etc. However, the execution efficiency of voice commands is difficult to meet the needs of users, and the gesture recognition method is easily interfered by the outside world, resulting in inaccurate control of the terminal device, thereby affecting the fluency of user operations. Summary of the Invention

[0003] This application provides an interaction control method, an electronic device, and a storage medium to solve the technical problems of low interaction control efficiency and low accuracy.

[0004] In a first aspect of an embodiment of this application, an interaction control method is provided, which is applied to a near-eye display device. The near-eye display device communicates with a terminal device. The method includes: determining a first coordinate of the user's eyeball in a camera coordinate system based on the user's eye image, and determining the deviation angle of the eyeball; determining a second coordinate of the eyeball in a world coordinate system based on the first coordinate, the deviation angle, and the distance between the user and the terminal device; determining the sight landing position of the eyeball according to the second coordinate, the sight image of the eyeball, and the imaging area of the terminal device in the sight image; and performing a control operation on the terminal device according to the eye movement feature parameters of the user at the sight landing position.

[0005] According to an embodiment of this application, the deviation angle includes a first deviation angle of the eyeball in the horizontal direction and a second deviation angle of the eyeball in the vertical direction. The determining the second coordinate of the eyeball in the world coordinate system based on the first coordinate, the deviation angle, and the distance between the user and the terminal device includes: constructing a unit vector according to the first deviation angle and the second deviation angle; and calculating the second coordinate according to the first coordinate, the distance, and the unit vector.

[0006] According to an embodiment of this application, the determining the sight landing position of the eyeball according to the second coordinate, the sight image of the eyeball, and the imaging area of the terminal device in the sight image includes: determining the deviation distance of the eyeball according to the sight image and the imaging area; determining a scaling ratio based on the image size of the sight image and a preset size; and determining the sight landing position based on the second coordinate, the deviation distance, and the scaling ratio.

[0007] According to an embodiment of the present application, the offset distance includes a first offset distance of the eyeball in the horizontal direction and a second offset distance in the vertical direction. Determining the offset distance of the eyeball according to the line-of-sight image and the imaging area includes: determining the first offset distance based on the distance between the left boundary of the line-of-sight image and the left boundary of the imaging area and / or the distance between the right boundary of the line-of-sight image and the right boundary of the imaging area; determining the second offset distance based on the distance between the upper boundary of the line-of-sight image and the upper boundary of the imaging area and / or the distance between the lower boundary of the line-of-sight image and the lower boundary of the imaging area.

[0008] According to an embodiment of the present application, the eye image is acquired by a first photographing device installed on the near-eye display device. The light-emitting diode of the first photographing device is an infrared light-emitting diode, and the first photographing device does not include an infrared filter; the line-of-sight image is acquired by a second photographing device installed on the near-eye display device, and the photographing angle of the second photographing device is different from that of the first photographing device.

[0009] According to an embodiment of the present application, determining the first coordinate of the user's eyeball in the camera coordinate system based on the user's eye image includes: extracting feature points from the eye image; determining the third coordinate of the feature points in the camera coordinate system based on the first three-dimensional eye model and the actual positions of the feature points in the eye image, where the first three-dimensional eye model is constructed based on the initial coordinates of the eyeball in the camera coordinate system; predicting the theoretical positions of the feature points in the eye image based on the third coordinates and the initial coordinates; adjusting the first three-dimensional eye model based on the actual positions and the theoretical positions to obtain a second three-dimensional eye model, where the second three-dimensional eye model is constructed based on the first coordinate.

[0010] According to an embodiment of the present application, adjusting the first three-dimensional eye model based on the actual positions and the theoretical positions to obtain a second three-dimensional eye model includes: determining the loss value of the first three-dimensional eye model based on the actual positions and the theoretical positions; adjusting the initial coordinates in the first three-dimensional eye model until the loss value meets a preset condition to obtain the second three-dimensional eye model.

[0011] According to an embodiment of the present application, performing a control operation on the terminal device according to the eye movement characteristic parameters of the user at the line-of-sight landing position includes: if the line-of-sight landing position is not on the display interface of the terminal device, releasing the control authority of the user over the terminal device; if the line-of-sight landing position is on the display interface of the terminal device, performing a control operation on the terminal device according to the eye movement characteristic parameters, where the eye movement characteristic parameters include at least one of the following parameters: the number of blinks, blink duration, and blink frequency of the user.

[0012] A second aspect of the embodiments of the present application provides an interactive control device running on a near-eye display device, where the near-eye display device communicates with a terminal device. The device includes: a determination unit configured to determine a first coordinate of the user's eyeball in a camera coordinate system based on the user's eye image, and determine an offset angle of the eyeball; the determination unit is further configured to determine a second coordinate of the eyeball in a world coordinate system based on the first coordinate, the offset angle, and the distance between the user and the terminal device; the determination unit is further configured to determine the line-of-sight landing position of the eyeball according to the second coordinate, the line-of-sight image of the eyeball, and the imaging area of the terminal device in the line-of-sight image; a control unit configured to perform a control operation on the terminal device according to the eye movement characteristic parameters of the user at the line-of-sight landing position.

[0013] A third aspect of the embodiments of the present application provides an electronic device, where the electronic device includes: a memory storing computer-readable instructions; and a processor configured to execute the computer-readable instructions stored in the memory to implement the interactive control method.

[0014] A fourth aspect of the embodiments of the present application provides a computer-readable storage medium storing computer-readable instructions, where the computer-readable instructions are executed by a processor in an electronic device to implement the interactive control method.

[0015] In multiple embodiments of the present application, through the user's eye image, the first coordinate and the offset angle of the user's eyeball in the camera coordinate system can be determined. Furthermore, through the first coordinate, the offset angle, and the distance between the user and the terminal device, the conversion relationship between the camera coordinate system and the world coordinate system can be dynamically established, so that the second coordinate of the eyeball in the world coordinate system can be accurately determined. Through the second coordinate, the line-of-sight image of the eyeball, and the imaging area of the terminal device in the line-of-sight image, the line-of-sight landing position of the eyeball can be accurately located. Through the eye movement characteristic parameters of the user at the line-of-sight landing position, a control operation on the terminal device can be quickly and accurately performed, improving the fluency of user operations. In addition, the embodiments of the present application can realize the intelligent linkage between the near-eye display device and an external terminal device, expanding the application scenarios of the eye movement tracking technology. Brief Description of the Drawings

[0016] Figure 1 It is a schematic three-dimensional structure diagram of a near-eye display device provided by an embodiment of the present application.

[0017] Figure 2 It is an application scenario diagram of an interaction control method provided by an embodiment of the present application.

[0018] Figure 3 It is a flowchart of an interaction control method provided by an embodiment of the present application.

[0019] Figure 4 It is a schematic diagram of an eye image provided by an embodiment of the present application.

[0020] Figure 5 It is another schematic diagram of an eye image provided by an embodiment of the present application.

[0021] Figure 6 It is a flowchart of a method for determining a first coordinate provided by an embodiment of the present application.

[0022] Figure 7 It is a schematic diagram of a second imaging device collecting a line-of-sight image provided by an embodiment of the present application.

[0023] Figure 8 It is a schematic diagram of a line-of-sight image provided by an embodiment of the present application.

[0024] Figure 9 It is a flowchart of a method for determining the line-of-sight landing position of an eyeball provided by an embodiment of the present application.

[0025] Figure 10 It is a schematic diagram of a line-of-sight landing position provided by an embodiment of the present application.

[0026] Figure 11 It is another schematic diagram of a line-of-sight landing position provided by an embodiment of the present application.

[0027] Figure 12 It is a flowchart of an interaction control method provided by another embodiment of the present application.

[0028] Figure 13 It is a functional module diagram of an interaction control device provided by an embodiment of the present application.

[0029] Figure 14 It is a schematic structural diagram of an electronic device for implementing an interaction control method provided by an embodiment of the present application. Detailed Description of the Embodiments

[0030] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] It should be noted that in this application, "at least one" means one or more, and "a plurality" means two or more than two. "And / or" describes the association relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent: A exists alone, both A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims, and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0032] In the embodiments of this application, words such as "exemplary" or "for example" are used to give examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0033] Currently, interactive control of a terminal device can be achieved through Augmented Reality (AR) technology. In related technologies, the operation of a terminal device is usually controlled by means such as voice commands and gesture recognition. However, the execution efficiency of voice commands is difficult to meet the needs of users, and the gesture recognition method is easily interfered by the outside world, resulting in the inability to accurately control the terminal device, thereby affecting the fluency of user operations.

[0034] With the development of computer technology, eye-tracking technology can be applied in fields such as virtual reality and human-computer interaction. This technology mainly realizes efficient human-computer interaction by tracking the eye movements of users and locating the focus of their attention. However, in related technical solutions, the human-computer interaction method based on eye tracking is still limited to the internal operation of a single device, making it difficult to achieve intelligent linkage between a near-eye display device and external devices (such as computers, TVs, etc.), thus hindering the application of AR technology in scenarios such as multi-screen collaboration and cross-device control.

[0035] Based on the above problems, an embodiment of the present application provides an interaction control method. Through the user's eye image, the first coordinate and the offset angle of the user's eyeball in the camera coordinate system can be determined. Furthermore, through the first coordinate, the offset angle, and the distance between the user and the terminal device, the conversion relationship between the camera coordinate system and the world coordinate system can be dynamically established, so that the second coordinate of the eyeball in the world coordinate system can be accurately determined. Through the second coordinate, the line-of-sight image of the eyeball, and the imaging area of the terminal device in the line-of-sight image, the line-of-sight landing position of the eyeball can be accurately located. Through the eye movement feature parameters of the user at the line-of-sight landing position, the control operation of the terminal device can be quickly and accurately executed, improving the fluency of the user operation. In addition, the embodiment of the present application can not only realize the intelligent linkage between the near-eye display device and the external terminal device, but also realize the interaction between the near-eye display device and multiple external devices, expanding the application scenarios of the eye movement tracking technology.

[0036] As Figure 1 shown, it is a schematic three-dimensional structure diagram of a near-eye display device provided by an embodiment of the present application.

[0037] Please refer to Figure 1 , the near-eye display device 100 in the embodiment of the present application may be a virtual reality (VR) helmet, VR glasses, etc. In the embodiment of the present application, the near-eye display device 100 may include, but is not limited to: a first shooting device 10 and a second shooting device 20. The shooting angle of the second shooting device 20 is different from that of the first shooting device 10. The first shooting device 10 may be used to shoot the user's eye image, and the second shooting device 20 may be used to shoot the line-of-sight image of the user's eyeball. In one example, the first shooting device 10 may be an infrared camera. In another example, the light-emitting diode of the first shooting device 10 is an infrared light-emitting diode, and the first shooting device 10 does not include an infrared filter.

[0038] As Figure 2 shown, it is an application scenario diagram of the interaction control method provided by an embodiment of the present application.

[0039] In the embodiment of the present application, the interaction control method may be applied to one or more near-eye display devices 100. The near-eye display device 100 may include intelligent wearable devices. For example, the near-eye display device 100 may include, but is not limited to: virtual reality (VR) devices, augmented reality (AR) devices, and mixed reality (MR) devices. Among them, VR devices may include, but are not limited to: VR helmets and VR glasses; AR devices may include, but are not limited to: AR helmets and AR glasses.

[0040] In an embodiment of the present application, the near-eye display device 100 can communicate with the terminal device 200. The terminal device 200 can be any electronic product capable of performing human-computer interaction with a user. For example, a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an Internet Protocol Television (IPTV), a smart wearable device, etc.

[0041] In an embodiment of the present application, the near-eye display device 100 and the terminal device 200 can be directly or indirectly communicatively connected through one or more networks. The network can be a wired network or a wireless network. For example, the wireless network can be a mobile cellular network or a Wireless-Fidelity (WIFI) network. Of course, it can also be other possible networks, and the embodiments of the present application do not limit this. It should be noted that Figure 2 The above is only an example, and actually the number of terminal devices and near-eye display devices is not limited and is not specifically defined in the embodiments of the present application.

[0042] Referring to Figure 2 As shown, the near-eye display device 100 in the embodiment of the present application is exemplified by a VR helmet. When a user wears the VR helmet, the user can perform control operations on the terminal device 200 through the VR helmet. For example, the user can control the terminal device 200 to start an application program through the VR helmet, so that the terminal device 200 plays a video and / or displays an image.

[0043] In another example, when the user wears the VR helmet, the user can also superimpose voice commands to perform control operations on the terminal device 200. For example, the user can control the terminal device 200 to start an application program through the VR helmet. At this time, the user can input text corresponding to the voice command on the display interface of the application program through the voice command. Another example is that the user can control the terminal device 200 to start an application program through the VR helmet. At this time, the user can control the terminal device 200 to play a video and / or display an image on the display interface of the application program through the voice command. In actual applications, it is not limited to this.

[0044] Next, in combination with the above-described application scenarios, the interactive control method provided in the embodiments of the present application will be described with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown for the convenience of understanding the spirit and principle of the embodiments of the present application, and the embodiments of the present application are not limited in this regard.

[0045] As Figure 3As shown, it is a flowchart of an interaction control method provided by an embodiment of the present application. The interaction control method is applied to an electronic device. For example, Figure 1 a near-eye display device 100. The near-eye display device can communicate with a terminal device. According to different requirements, the order of steps in this flowchart can be changed, and some steps can be omitted.

[0046] S301, based on the user's eye image, determine the first coordinate of the user's eyeball in the camera coordinate system, and determine the deviation angle of the eyeball.

[0047] In at least one embodiment of the present application, the near-eye display device may be equipped with at least one first photographing device. In one example, the first photographing device may be an infrared camera. In another example, the light-emitting diode of the first photographing device is an infrared light-emitting diode, and the first photographing device does not include an infrared filter, and the infrared filter can be used to filter infrared light.

[0048] The near-eye display device can collect the user's eye image through the first photographing device. Refer to Figure 4 and Figure 5 As shown, the eye image collected by an ordinary camera can be as Figure 4 shown, and the eye image collected by the first photographing device of the embodiment of the present application can be as Figure 5 shown. In the Figure 4 eye image shown, the eyeball in the eye image contains interference information.

[0049] In the embodiment of the present application, the infrared light-emitting diode in the first photographing device can be used to control the first photographing device to emit an infrared light source. The first photographing device does not include an infrared filter, which can avoid the filtering of the infrared light emitted by the first photographing device by the infrared filter, and ensure the image quality of the eye image collected by the first photographing device.

[0050] In at least one embodiment of the present application, the manner in which the near-eye display device determines the first coordinate of the user's eyeball in the camera coordinate system based on the eye image can be referred to the following Figure 6 flowchart shown, Figure 6 including steps S3011 to S3014.

[0051] S3011, extract feature points from the eye image.

[0052] In some embodiments, the near-eye display device performs grayscale processing on the eye image to obtain a grayscale image. The near-eye display device performs convolution processing on the grayscale image through multiple preset filters to obtain a feature image corresponding to each preset filter. Among them, each preset filter has a corresponding Gaussian convolution kernel, and the sizes of the Gaussian convolution kernels corresponding to the multiple preset filters are different. Correspondingly, the scales of the multiple feature images are different.

[0053] For each feature image, the near-eye display device compares the gray value of any pixel point in the feature image with the gray values of other pixel points adjacent to the pixel point. If the gray value of any pixel point is a minimum value or a maximum value, the near-eye display device determines the pixel point as a key point.

[0054] The near-eye display device determines a neighborhood image corresponding to the key point from the feature image corresponding to the key point. Herein, the neighborhood image represents an image in the feature image centered on the key point, and the shape of the neighborhood image is not limited. For example, the shape of the neighborhood image can be circular, square, rectangular, etc. The size of the neighborhood image is smaller than the size of the feature image.

[0055] The near-eye display device calculates the contrast of the key point based on the neighborhood image. The calculation formula of the contrast can be expressed as: , where can represent the contrast of the key point, can represent the gray value of the key point, can represent the average value of the gray values of all pixel points in the neighborhood image, can represent the standard deviation of the gray values of all pixel points in the neighborhood image.

[0056] The near-eye display device determines the key points with a contrast greater than a preset contrast threshold as feature points.

[0057] S3012. Based on the first three-dimensional eye model and the actual position of the feature point in the eye image, determine the third coordinate of the feature point in the camera coordinate system.

[0058] In some embodiments, the near-eye display device can pre-construct a camera coordinate system. The near-eye display device uses the optical center of the first imaging device as the origin of the camera coordinate system. On the imaging plane of the first imaging device, the X-axis of the camera coordinate system is constructed based on the horizontal direction of the imaging plane, the Y-axis of the camera coordinate system is constructed based on the vertical direction of the imaging plane, and the direction perpendicular to the imaging plane is determined as the Z-axis direction of the camera coordinate system.

[0059] In some embodiments, the near-eye display device can preset the initial coordinates of the eyeball in the camera coordinate system, and the initial coordinates can be set and adjusted according to actual needs. For example, the initial coordinates can be set to ( ), where can represent the horizontal position of the eyeball on the imaging plane, can represent the vertical position of the eyeball on the imaging plane, can represent the distance between the eyeball and the first imaging device. The near-eye display device can construct a first three-dimensional eye model based on the initial coordinates of the eyeball in the camera coordinate system.

[0060] In some embodiments, the near-eye display device may construct an image coordinate system based on the eye image. Among them, the near-eye display device uses the upper left corner of the eye image as the origin of the image coordinate system, the upper boundary of the eye image as the X-axis of the image coordinate system, and the left boundary of the eye image as the Y-axis of the image coordinate system. The near-eye display device may determine the actual position of the feature point in the eye image based on the image coordinate system.

[0061] In some embodiments, the near-eye display device may determine the third coordinate of the feature point in the camera coordinate system based on the first three-dimensional eye model and the actual position of the feature point in the eye image. The determination formula of the third coordinate may be expressed as: , , ([[]] ) may represent the third coordinate of the th feature point, may represent the focal length of the first imaging device, ([[]] ) may represent the actual position of the th feature point in the eye image, ([[]] ) may represent the initial coordinate.

[0062] S3013. Predict the theoretical position of the feature point in the eye image based on the third coordinate and the initial coordinate in the first three-dimensional eye model.

[0063] In some embodiments, the determination formula of the theoretical position of the feature point in the eye image may be expressed as: , , where ([[]] ) may represent the theoretical position of the feature point in the eye image, may represent the focal length of the first imaging device, ([[]] ) may represent the third coordinate of the th feature point, ([[]] ) may represent the initial coordinate.

[0064] S3014. Adjust the first three-dimensional eye model based on the actual position and the theoretical position to obtain a second three-dimensional eye model.

[0065] In some embodiments, during the process of determining the first coordinate, the near-eye display device may determine the loss value of the first three-dimensional eye model based on the actual position and the theoretical position, and adjust the initial coordinate in the first three-dimensional eye model until the loss value meets the preset condition to obtain the first coordinate. The near-eye display device may construct a second three-dimensional eye model based on the first coordinate. Among them, the determination formula of the loss value may be expressed as: , where may represent the loss value of the first three-dimensional eye model, ([[]] can represent the theoretical position of the feature point in the eye image, ( can represent the th actual position of the feature point in the eye image, can represent the total number of all feature points in the eye image. The preset conditions may include at least one of the following conditions: the loss value is less than or equal to a preset loss threshold, the loss value no longer decreases, etc. In this embodiment, based on the actual position and the theoretical position of the feature point in the eye image, the loss value of the first three-dimensional eye model can be predicted. Since the first three-dimensional eye model is constructed based on the initial coordinates of the eyeball in the camera coordinate system, therefore, by adjusting the initial coordinates in the first three-dimensional eye model, the accuracy of the second three-dimensional eye model can be improved, and thus an accurate first coordinate can be obtained.

[0066] In the embodiment of the present application, by extracting feature points from the eye image and based on the initial coordinates of the eyeball in the camera coordinate system and the actual position of the feature point in the eye image, the theoretical position of the feature point in the eye image can be predicted. Based on the actual position and the theoretical position of the feature point in the eye image, the first three-dimensional eye model can be accurately optimized, the accuracy of the second three-dimensional eye model is improved, and thus the first coordinate of the eyeball in the camera coordinate system can be accurately determined.

[0067] S302. Based on the first coordinate, the offset angle, and the distance between the user and the terminal device, determine the second coordinate of the eyeball in the world coordinate system.

[0068] In at least one embodiment of the present application, the offset angle includes a first offset angle of the eyeball in the horizontal direction and a second offset angle of the eyeball in the vertical direction. The distance between the user and the terminal device may indicate the distance between the user and the display screen of the terminal device. The distance between the user and the terminal device may also indicate the distance between the near-eye display device worn by the user and the terminal device.

[0069] In at least one embodiment of the present application, in the process of determining the second coordinate of the eyeball in the world coordinate system, the near-eye display device constructs a unit vector according to the first offset angle and the second offset angle, and calculates the second coordinate according to the first coordinate, the distance, and the unit vector.

[0070] In some embodiments, the construction formula of the unit vector can be expressed as: , , . Wherein, can represent the unit vector, can represent the first offset angle, can represent the second offset angle.

[0071] In some embodiments, the calculation formula of the second coordinate can be expressed as: , , . Wherein, ( ) can represent the second coordinate, ( ) can represent the first coordinate, can represent the distance, can represent the unit vector.

[0072] In the embodiments of the present application, through the first offset angle and the second offset angle, a unit vector can be obtained. Furthermore, through the distance and the unit vector, a conversion relationship between the camera coordinate system and the world coordinate system can be established, so that based on the first coordinate, the second coordinate of the eyeball in the world coordinate system can be quickly determined.

[0073] S303. Determine the line-of-sight landing position of the eyeball according to the second coordinate, the line-of-sight image of the eyeball, and the imaging area of the terminal device in the line-of-sight image.

[0074] In at least one embodiment of the present application, the near-eye display device may be equipped with a second photographing device. The photographing angle of the second photographing device is different from that of the first photographing device. Through the second photographing device, the near-eye display device can collect a line-of-sight image, and the line-of-sight image may be an image captured by the user's eyeball. Refer to Figure 7 as shown in Figure 7 , Figure 7 which is a schematic diagram of the line-of-sight image collected by the second photographing device provided in the embodiments of the present application. As shown in

[0075] Figure 8 , taking the user's eyeball 300 as the photographing angle, the line-of-sight image 400 can be collected through the second photographing device in the near-eye display device. Figure 8 The imaging area is used to indicate the display area of the terminal device in the line-of-sight image. Refer to Figure 8 as shown in

[0076] Figure 9 , which is a schematic diagram of the line-of-sight image provided in the embodiments of the present application. As shown in

[0077]

[0078] S304. Perform a control operation on the terminal device according to the eye movement characteristic parameters of the user at the line-of-sight landing position.In at least one embodiment of the present application, if the line-of-sight landing position is not on the display interface of the terminal device, the near-eye display device releases the user's control authority over the terminal device. If the line-of-sight landing position is on the display interface of the terminal device, the near-eye display device performs a control operation on the terminal device according to the eye movement characteristic parameters.

[0079] Refer to Figure 10 as shown in Figure 10 a schematic diagram of the line-of-sight landing position provided by an embodiment of the present application. As Figure 10 shown, the line-of-sight landing position A is within the display interface of the terminal device, and the near-eye display device can perform a control operation on the line-of-sight landing position in the terminal device according to the eye movement characteristic parameters.

[0080] In at least one embodiment of the present application, the eye movement characteristic parameters include at least one of the following parameters: the number of user blinks, blink duration, and blink frequency.

[0081] In an example, if the number of blinks is less than a preset number threshold, the near-eye display device can control a first preset cursor corresponding to the line-of-sight landing position. If the number of blinks is greater than or equal to the preset number threshold, the near-eye display device can control a second preset cursor corresponding to the line-of-sight landing position. Among them, the preset number threshold can be set and adjusted according to actual needs. The control methods corresponding to the first preset cursor and the second preset cursor are different. The control method corresponding to the first preset cursor can correspond to the function of the left mouse button, and the control method corresponding to the second preset cursor can correspond to the function of the right mouse button. For example, if the number of blinks is less than the preset number threshold, the near-eye display device can open the application program corresponding to the line-of-sight landing position in the terminal device, and the near-eye display device can move the object at the line-of-sight landing position in the terminal device. Another example is that if the number of blinks is greater than or equal to the preset number threshold, the near-eye display device can pop up the corresponding shortcut menu at the line-of-sight landing position in the terminal device, etc.

[0082] In another example, if the user's blink duration is greater than a preset duration, it is determined that the user has control authority over the terminal device, and the preset duration can be set and adjusted according to actual needs.

[0083] Refer to Figure 11 as shown in Figure 11 another schematic diagram of the line-of-sight landing position provided by an embodiment of the present application. As Figure 11 shown, the terminal device includes display screen ①, display screen ②, and display screen ③. When the line-of-sight landing position is on display screen ② of the terminal device, the near-eye display device can determine that the user has control authority over display screen ②.

[0084] In another example, if the near-eye display device communicates with multiple terminal devices, and if terminal device 201 has a display screen 1, terminal device 202 has a display screen 2, and terminal device 203 has a display screen 3. When the line-of-sight landing position is on the display screen 2, the near-eye display device can determine that the user has the control authority over terminal device 202.

[0085] In multiple embodiments of the present application, through the user's eye image, the first coordinate and the offset angle of the user's eyeball in the camera coordinate system can be determined. Furthermore, through the first coordinate, the offset angle, and the distance between the user and the terminal device, the conversion relationship between the camera coordinate system and the world coordinate system can be dynamically established, so that the second coordinate of the eyeball in the world coordinate system can be accurately determined. Through the second coordinate, the line-of-sight image of the eyeball, and the imaging area of the terminal device in the line-of-sight image, the line-of-sight landing position of the eyeball can be accurately located. Through the eye movement characteristic parameters of the user at the line-of-sight landing position, the control operation on the terminal device can be quickly and accurately performed, improving the fluency of the user operation. In addition, the embodiments of the present application can realize the intelligent linkage between the near-eye display device and the external terminal device, expanding the application scenarios of the eye movement tracking technology.

[0086] As Figure 9 shown, it is a flowchart of the method for determining the line-of-sight landing position of the eyeball provided by the embodiments of the present application, including the following processes: S901, determine the offset distance of the eyeball according to the line-of-sight image and the imaging area.

[0087] In some embodiments, the offset distance may include a first offset distance of the eyeball in the horizontal direction. Among them, when constructing an image coordinate system with any point on the left boundary of the eye image as the origin, the near-eye display device can determine the first offset distance based on the distance between the left boundary of the line-of-sight image and the left boundary of the imaging area. For example Figure 8 shown, the first offset distance Xoffset can represent the straight-line distance between the left boundary of the line-of-sight image 400 and the left boundary of the imaging area 500.

[0088] When constructing an image coordinate system with any point on the right boundary of the eye image as the origin, the near-eye display device can determine the first offset distance based on the distance between the right boundary of the line-of-sight image and the right boundary of the imaging area.

[0089] The first central boundary of the eye image is the central boundary between the left boundary and the right boundary of the eye image. When constructing an image coordinate system with any point on the first central boundary of the eye image as the origin, the near-eye display device can determine a first offset distance based on the left boundary distance between the left boundary of the line-of-sight image and the left boundary of the imaging area, and the right boundary distance between the right boundary of the line-of-sight image and the right boundary of the imaging area. Among them, the first offset distance can be the average value of the left boundary distance and the right boundary distance.

[0090] In some other embodiments, the offset distance may further include a second offset distance of the eyeball in the vertical direction. Among them, when constructing an image coordinate system with any point on the upper boundary of the eye image as the origin, the near-eye display device can determine the second offset distance based on the distance between the upper boundary of the line-of-sight image and the upper boundary of the imaging area. For example Figure 8 As shown, the second offset distance Yoffset can represent the straight-line distance between the upper boundary of the line-of-sight image 400 and the upper boundary of the imaging area 500.

[0091] When constructing an image coordinate system with any point on the lower boundary of the eye image as the origin, the near-eye display device can determine the second offset distance based on the distance between the lower boundary of the line-of-sight image and the lower boundary of the imaging area.

[0092] The second central boundary of the eye image is the central boundary between the upper boundary and the lower boundary of the eye image. When constructing an image coordinate system with the second central boundary of the eye image as the origin, the near-eye display device can determine the second offset distance based on the upper boundary distance between the upper boundary of the line-of-sight image and the upper boundary of the imaging area, and the lower boundary distance between the lower boundary of the line-of-sight image and the lower boundary of the imaging area. Among them, the first offset distance can be the average value of the upper boundary distance and the lower boundary distance.

[0093] In some other embodiments, the offset distance may further include a third offset distance of the eyeball in the vertical direction. The third offset distance can be set and adjusted according to actual needs. Usually, the third offset distance can be set to 0.

[0094] S902. Determine a scaling ratio based on the image size of the line-of-sight image and a preset size.

[0095] In some embodiments, the preset size can be set and adjusted according to actual needs. For example, the preset size can be set to the size of the display screen of the terminal device. The preset size can include a first size in the horizontal direction and a second size in the vertical direction. The near-eye display device can determine the scaling ratio of the eyeball in the horizontal direction by the ratio of the image size of the line-of-sight image in the horizontal direction to the first size. The near-eye display device can determine the scaling ratio of the eyeball in the vertical direction by the ratio of the image size of the line-of-sight image in the vertical direction to the second size. The scaling ratio of the eyeball in the vertical direction can be set and adjusted according to actual needs. Usually, the third offset distance can be set to 1.

[0096] S903. Determine the line-of-sight landing position based on the second coordinate, the offset distance, and the scaling ratio.

[0097] In some embodiments, the formula for determining the line-of-sight landing position can be expressed as: , , , where, ( ) can represent the line-of-sight landing position of the eyeball, ( , ) can represent the second coordinate, can represent the first offset distance, can represent the second offset distance, can represent the third offset distance, can represent the scaling ratio of the eyeball in the horizontal direction, can represent the scaling ratio of the eyeball in the vertical direction, can represent the scaling ratio of the eyeball in the vertical direction.

[0098] In the embodiments of the present application, through the line-of-sight image and the imaging area, the offset distance of the eyeball can be accurately quantified. Through the image size of the line-of-sight image and the preset size, the scaling ratio can be determined. By adjusting the second coordinate with the offset distance and the scaling ratio, the line-of-sight landing position of the eyeball can be accurately determined.

[0099] As Figure 12 shown, it is a flowchart of an interaction control method provided by another embodiment of the present application. The interaction control method is applied to an electronic device. For example, Figure 1 the near-eye display device 100. The near-eye display device can communicate with the terminal device. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0100] S1201. Send a control permission request to the terminal device.

[0101] In at least one embodiment of the present application, when the near-eye display device and the terminal device are in the same local area network, the near-eye display device may send a control permission request to the terminal device. The control permission request may be used to request control of the terminal device.

[0102] S1202. In response to the consent response sent by the terminal device based on the control permission request, establish a communication connection with the terminal device.

[0103] S1203. Collect an eye image of the user through the near-eye display device, determine the first coordinate of the user's eyeball in the camera coordinate system, and determine the deviation angle of the eyeball.

[0104] S1204. Based on the first coordinate, the deviation angle, and the distance between the user and the terminal device, determine the second coordinate of the eyeball in the world coordinate system.

[0105] S1205. According to the second coordinate, the line-of-sight image of the eyeball, and the imaging area of the terminal device in the line-of-sight image, determine the line-of-sight landing position of the eyeball.

[0106] For the detailed content of steps S1203 - S1205, reference may be made to the detailed description of steps S301 - S303 above Figure 3 and the detailed description will not be repeated here.

[0107] S1206. Detect whether the line-of-sight landing position is on the display interface of the terminal device.

[0108] In at least one embodiment of the present application, the near-eye display device determines the device position of the terminal device in the world coordinate system, and detects whether the line-of-sight landing position is within the device position. If the line-of-sight landing position is within the device position, the near-eye display device determines that the line-of-sight landing position is on the display interface of the terminal device. If the line-of-sight landing position is outside the device position, the near-eye display device determines that the line-of-sight landing position is not on the display interface of the terminal device.

[0109] In at least one embodiment of the present application, if the line-of-sight landing position is not on the display interface of the terminal device, execute step S1207; if the line-of-sight landing position is on the display interface of the terminal device, execute step S1208.

[0110] S1207. Release the user's control permission for the terminal device.

[0111] S1208. Perform a control operation on the terminal device according to the eye movement characteristic parameters of the user at the line-of-sight landing position.

[0112] For the detailed content of steps S1207 - S1208, reference may be made to the detailed description of step S304 above Figure 3 and the detailed description will not be repeated here.

[0113] In multiple embodiments of the present application, based on the user's eye image, the first coordinates of the user's eyeball in the camera coordinate system and the offset angle can be determined. Furthermore, based on the first coordinates, the offset angle, and the distance between the user and the terminal device, the conversion relationship between the camera coordinate system and the world coordinate system can be dynamically established, so that the second coordinates of the eyeball in the world coordinate system can be accurately determined. Based on the second coordinates, the line-of-sight image of the eyeball, and the imaging area of the terminal device in the line-of-sight image, the line-of-sight landing position of the eyeball can be accurately located. Based on the eye movement feature parameters of the user at the line-of-sight landing position, the control operation of the terminal device can be quickly and accurately performed, improving the fluency of user operation. In addition, the embodiments of the present application can achieve the intelligent linkage between the near-eye display device and the external terminal device, expanding the application scenarios of the eye movement tracking technology.

[0114] As Figure 13 shown, it is a functional module diagram of the interaction control device provided by the embodiments of the present application. The interaction control device 11 runs on a near-eye display device, and the near-eye display device communicates with the terminal device. The interaction control device 11 includes a determination unit 110 and a control unit 111. The module / unit referred to in the present application means a series of computer-readable instruction segments that can be obtained by a processor (such as Figure 14 the processor 1401 shown) and can complete fixed functions, and are stored in a memory (such as Figure 14 the memory 1402 shown).

[0115] In one embodiment, the determination unit 110 is configured to determine the first coordinates of the user's eyeball in the camera coordinate system based on the user's eye image, and determine the offset angle of the eyeball; the determination unit 110 is further configured to determine the second coordinates of the eyeball in the world coordinate system based on the first coordinates, the offset angle, and the distance between the user and the terminal device; the determination unit 110 is further configured to determine the line-of-sight landing position of the eyeball according to the second coordinates, the line-of-sight image of the eyeball, and the imaging area of the terminal device in the line-of-sight image; the control unit 111 is configured to perform a control operation on the terminal device according to the eye movement feature parameters of the user at the line-of-sight landing position.

[0116] In one embodiment, the offset angle includes a first offset angle of the eyeball in the horizontal direction and a second offset angle of the eyeball in the vertical direction. The determination unit 110 is specifically configured to: construct a unit vector according to the first offset angle and the second offset angle; calculate the second coordinates according to the first coordinates, the distance, and the unit vector.

[0117] In one embodiment, the determination unit 110 is specifically further configured to: determine the offset distance of the eyeball according to the line-of-sight image and the imaging area; determine the scaling ratio based on the image size of the line-of-sight image and a preset size; determine the line-of-sight landing position based on the second coordinates, the offset distance, and the scaling ratio.

[0118] In one embodiment, the offset distance includes a first offset distance of the eyeball in the horizontal direction and a second offset distance in the vertical direction. The determining unit 110 is specifically further configured to include: determining the first offset distance based on the distance between the left boundary of the line-of-sight image and the left boundary of the imaging region, and / or the distance between the right boundary of the line-of-sight image and the right boundary of the imaging region; determining the second offset distance based on the distance between the upper boundary of the line-of-sight image and the upper boundary of the imaging region, and / or the distance between the lower boundary of the line-of-sight image and the lower boundary of the imaging region.

[0119] In one embodiment, the eye image is acquired by a first photographing device installed on the near-eye display device. The light-emitting diode of the first photographing device is an infrared light-emitting diode, and the first photographing device does not include an infrared filter; the line-of-sight image is acquired by a second photographing device installed on the near-eye display device, and the photographing angle of the second photographing device is different from that of the first photographing device.

[0120] In one embodiment, the determining unit 110 is specifically further configured to: extract feature points from the eye image; determine the third coordinate of the feature points in the camera coordinate system based on the first three-dimensional eye model and the actual positions of the feature points in the eye image, where the first three-dimensional eye model is constructed based on the initial coordinates of the eyeball in the camera coordinate system; predict the theoretical positions of the feature points in the eye image based on the third coordinate and the initial coordinates; and adjust the first three-dimensional eye model based on the actual positions and the theoretical positions to obtain a second three-dimensional eye model, where the second three-dimensional eye model is constructed based on the first coordinate.

[0121] In one embodiment, the determining unit 110 is specifically further configured to: determine the loss value of the first three-dimensional eye model based on the actual positions and the theoretical positions; and adjust the initial coordinates in the first three-dimensional eye model until the loss value meets a preset condition to obtain the second three-dimensional eye model.

[0122] In one embodiment, the control unit 111 is specifically configured to: release the control authority of the user over the terminal device if the line-of-sight landing position is not on the display interface of the terminal device; and perform a control operation on the terminal device according to the eye movement feature parameters if the line-of-sight landing position is on the display interface of the terminal device, where the eye movement feature parameters include at least one of the following parameters: the number of blinks, the blink duration, and the blink frequency of the user.

[0123] In multiple embodiments of the present application, based on the user's eye image, the first coordinates of the user's eyeball in the camera coordinate system and the offset angle can be determined. Furthermore, based on the first coordinates, the offset angle, and the distance between the user and the terminal device, the conversion relationship between the camera coordinate system and the world coordinate system can be dynamically established, thereby accurately determining the second coordinates of the eyeball in the world coordinate system. Based on the second coordinates, the line-of-sight image of the eyeball, and the imaging area of the terminal device in the line-of-sight image, the line-of-sight landing position of the eyeball can be accurately located. Based on the eye movement feature parameters of the user at the line-of-sight landing position, the control operation of the terminal device can be quickly and accurately performed, improving the fluency of user operation. In addition, the embodiments of the present application can achieve the intelligent linkage between the near-eye display device and the external terminal device, expanding the application scenarios of the eye tracking technology.

[0124] As Figure 14 shown, it is a schematic structural diagram of an electronic device according to a preferred embodiment of the interactive control method of the present application.

[0125] In one embodiment of the present application, the electronic device 1 includes, but is not limited to, a memory 1402, a processor 1401, and a computer program stored in the memory 1402 and executable on the processor 1401, such as an interactive control program.

[0126] Those skilled in the art can understand that the schematic diagram is only an example of the electronic device 1, and does not constitute a limitation on the electronic device 1. It may include more or fewer components than shown, or combine some components, or different components. For example, the electronic device 1 may also include input / output devices, network access devices, buses, etc.

[0127] The processor 1401 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may also be any conventional processor, etc. The processor 1401 is the operation core and control center of the electronic device 1, connecting various parts of the entire electronic device 1 through various interfaces and lines, and obtaining the operating system of the electronic device 1 and various installed application programs, program codes, etc.

[0128] The processor 1401 obtains the operating system of the electronic device 1 and various installed application programs. The processor 1401 obtains the application programs to implement the steps in the above-mentioned embodiments of each interaction control method. For example Figure 3 , Figure 6 , Figure 9 , Figure 12 the steps shown.

[0129] Exemplarily, the computer program can be divided into one or more modules / units. One or more modules / units are stored in the memory 1402 and obtained by the processor 1401 to complete the present application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the obtaining process of the computer program in the electronic device 1.

[0130] The memory 1402 can be used to store computer programs and / or modules. The processor 1401 realizes various functions of the electronic device 1 by running or obtaining the computer programs and / or modules stored in the memory 1402, and by calling the data stored in the memory 1402. The memory 1402 mainly includes a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the electronic device. In addition, the memory 1402 can include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0131] The memory 1402 can be an external memory and / or an internal memory of the electronic device 1. Further, the memory 1402 can be a memory in physical form, such as a memory stick, a TF card (Trans-flash Card), etc.

[0132] If the modules / units integrated in the electronic device 1 are implemented in the form of software functional units and sold or used as independent artifacts, they can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-mentioned embodiment methods of the present application, it can also be completed by instructing relevant hardware with computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium. When the computer-readable instructions are executed by the processor, the steps of the above-mentioned various method embodiments can be implemented.

[0133] Among them, the computer-readable instructions include computer-readable instruction codes, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable medium can include: any entity or device capable of carrying the computer-readable instruction code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM).

[0134] The memory 1402 can be used to store computer-readable instructions and / or modules. The processor 1401 realizes various functions of the electronic device 1 by running or executing the computer-readable instructions and / or modules stored in the memory 1402, and by calling the data stored in the memory 1402. The memory 1402 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the electronic device. The memory 1402 can include non-volatile and volatile memories, such as: hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one magnetic disk storage device, flash memory device, or other storage devices.

[0135] Exemplarily, the computer-readable instructions can be divided into one or more modules / units. One or more modules / units are stored in the memory 1402 and executed by the processor 1401 to complete this application. One or more modules / units can be a series of computer-readable instruction segments capable of completing specific functions, and the computer-readable instruction segments are used to describe the execution process of the computer-readable instructions in the electronic device 1. For example, the computer-readable instructions can be divided into a determination unit 110 and a control unit 111.

[0136] For the detailed content of the functions of each module / unit, reference can be made to the detailed description above Figure 3 、 Figure 6 、 Figure 9 、 Figure 12 , and the description will not be repeated here.

[0137] In several embodiments provided by this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division, and there can be other division methods in actual implementation.

[0138] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed over multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0139] In addition, in each embodiment of the present application, the functional modules can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of a combination of hardware and software functional modules.

[0140] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present application is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be regarded as limiting the claims involved.

[0141] In addition, it is obvious that the word "including" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices can also be implemented by one unit or device through software or hardware. The terms such as "first" and "second" are used to represent names and do not indicate any specific order.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Although the present application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. An interactive control method, characterized in that: Applied to a near-eye display device, the near-eye display device communicates with a terminal device, and the method includes: Based on the eye image of the user, determining a first coordinate of an eyeball of the user in a camera coordinate system, and determining an offset angle of the eyeball; Determine a second coordinate of the eyeball in a world coordinate system based on the first coordinate, the offset angle, and the distance between the user and the terminal device; Determine the location of the eye's sight line according to the second coordinate, the eye's sight line image, and the imaging area of ​​the terminal device in the sight line image; A control operation is performed on the terminal device according to the eye movement characteristic parameters of the user at the position where the sight line falls.

2. The interactive control method according to claim 1, characterized in that: The offset angle includes a first offset angle of the eyeball in the horizontal direction and a second offset angle of the eyeball in the vertical direction, and determining the second coordinate of the eyeball in the world coordinate system based on the first coordinate, the offset angle, and the distance between the user and the terminal device includes: constructing a unit vector according to the first offset angle and the second offset angle; The second coordinate is calculated according to the first coordinate, the distance and the unit vector.

3. The interactive control method according to claim 1, characterized in that: The determining the position of the sight point of the eyeball according to the second coordinate, the sight line image of the eyeball and the imaging area of ​​the terminal device in the sight line image includes: Determining the offset distance of the eyeball according to the sight line image and the imaging area; Determining a scaling ratio based on an image size of the sight line image and a preset size; The location of the sight point is determined based on the second coordinate, the offset distance and the scaling ratio.

4. The interactive control method according to claim 3, characterized in that: The offset distance includes a first offset distance of the eyeball in a horizontal direction and a second offset distance in a vertical direction, and determining the offset distance of the eyeball according to the sight line image and the imaging area includes: Determining the first offset distance based on a distance between a left boundary of the sight line image and a left boundary of the imaging area, and / or a distance between a right boundary of the sight line image and a right boundary of the imaging area; The second offset distance is determined based on a distance between an upper boundary of the sight line image and an upper boundary of the imaging area and / or a distance between a lower boundary of the sight line image and a lower boundary of the imaging area.

5. The interactive control method according to claim 1, characterized in that: The eye image is acquired by a first shooting device installed in the near-eye display device, the light emitting diode of the first shooting device is an infrared light emitting diode, and the first shooting device does not include an infrared filter; The line of sight image is acquired by a second shooting device installed on the near-eye display device, and a shooting angle of view of the second shooting device is different from a shooting angle of view of the first shooting device.

6. The interactive control method according to claim 1, characterized in that: The determining, based on the eye image of the user, a first coordinate of the eyeball of the user in the camera coordinate system comprises: Extracting feature points from the eye image; Determine, based on a first three-dimensional eye model and an actual position of the feature point in the eye image, a third coordinate of the feature point in the camera coordinate system, wherein the first three-dimensional eye model is constructed based on the initial coordinates of the eyeball in the camera coordinate system; Based on the third coordinate and the initial coordinate, predicting the theoretical position of the feature point in the eye image; Based on the actual position and the theoretical position, the first three-dimensional eye model is adjusted to obtain a second three-dimensional eye model, where the second three-dimensional eye model is constructed based on the first coordinates.

7. The interactive control method according to claim 6, characterized in that: The adjusting the first three-dimensional eye model based on the actual position and the theoretical position to obtain a second three-dimensional eye model includes: Determining a loss value of the first three-dimensional eye model based on the actual position and the theoretical position; The initial coordinates in the first three-dimensional eye model are adjusted until the loss value meets a preset condition, thereby obtaining the second three-dimensional eye model.

8. The interactive control method according to claim 1, characterized in that: The performing a control operation on the terminal device according to the eye movement characteristic parameters of the user at the position where the sight line falls, comprises: If the sight point is not located on the display interface of the terminal device, releasing the user's control authority over the terminal device; If the line of sight is located on the display interface of the terminal device, a control operation is performed on the terminal device according to the eye movement characteristic parameters, and the eye movement characteristic parameters include at least one of the following parameters: the number of blinks, blink duration and blink frequency of the user.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor implements the interactive control method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor of an electronic device, the interactive control method according to any one of claims 1 to 8 is implemented.

Images