UI element control method, head-mounted display device and storage medium

Abstract

The embodiment of the invention provides a UI element control method, head-mounted display equipment and a storage medium, and the method comprises the steps: obtaining UI elements displayed on a display interface of the head-mounted display equipment after the head-mounted display equipment starts an eye movement control function, the distance between the UI elements is greater than a first preset distance after the eye movement control function is started; obtaining eyeball movement data of a user wearing the head-mounted display device; and determining and controlling UI elements displayed in the display interface according to the eyeball movement data. The distance between the UI elements in the display interface is greater than the first preset distance, so that the false touch rate of the eye movement control operation is reduced; the user can control the UI element through eye movement, the user does not need to carry out secondary confirmation on the UI element determined through eyeball movement data through voice, gestures and the like, the two hands of the user are liberated, other users cannot find the control intention of the user wearing the head-mounted display device conveniently, and the user experience is improved. Therefore, the control privacy of the UI elements displayed by the head-mounted display equipment is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of display interface control, and in particular to a UI element control method, a head-mounted display device and a storage medium. BACKGROUND

[0002] With the development of smart wearable technology, head-mounted display devices such as augmented reality glasses, virtual reality glasses and AI (Artificial Intelligence) glasses have entered the lives of users. Since the head-mounted display device needs to be worn on the head of the user, in order to ensure the comfort of the user wearing and the use experience of the user, it is required that the head-mounted display device be light enough, so that the head-mounted display device is light. The operation on the head-mounted display device needs to be operated through a touch bar, a key, a ring or other remote controller that establishes a communication connection with the head-mounted display device, etc. on the head-mounted display device, which cannot free the hands of the user; the head-mounted display device can also be controlled through voice instructions, but the voice control has poor privacy. SUMMARY

[0003] Embodiments of the present application provide a UI element control method, a head-mounted display device and a storage medium, which aim to improve the privacy of controlling the UI elements displayed by the head-mounted display device.

[0004] In a first aspect, embodiments of the present application provide a UI element control method, which is applied to a head-mounted display device, and the method comprises:

[0005] After the head-mounted display device starts an eye movement control function, the UI elements displayed on the display interface of the head-mounted display device are acquired, wherein the distance between each UI element after starting the eye movement control function is greater than a first preset distance;

[0006] Eye movement data of a user wearing the head-mounted display device is acquired;

[0007] The UI elements displayed on the display interface are determined according to the eye movement data.

[0008] Optionally, the method further comprises:

[0009] The display mode of the UI elements in the display interface is determined;

[0010] The step of determining the UI elements displayed on the display interface according to the eye movement data comprises:

[0011] If the display mode is a first display mode, the UI elements displayed on the display interface are controlled according to at least one parameter in the eye movement data;

[0012] If the display mode is the second display mode, the UI element displayed in the display interface is controlled according to at least two parameters in the eye movement data.

[0013] Optionally, the step of determining the display mode of the UI element in the display interface comprises:

[0014] determining a display area of the UI element to be switched, wherein the UI element to be switched is at least partially displayed in the display interface;

[0015] If the display area is greater than a first preset value, it is determined that the display mode is the first display mode.

[0016] If the display area is less than or equal to a second preset value, it is determined that the display mode is the second display mode.

[0017] Or

[0018] determining a minimum value of display areas of all UI elements displayed in the display interface;

[0019] If the minimum value is greater than a second preset value, it is determined that the display mode is the first display mode.

[0020] If the minimum value is less than or equal to a second preset value, it is determined that the display mode is the second display mode.

[0021] Optionally, the UI elements displayed in the display interface at least include a first UI element and a second UI element corresponding to the first element, wherein the first UI element is used to display information, and the second UI element is a UI element corresponding to a function button.

[0022] The step of determining the control of the UI element displayed in the display interface according to the eye movement data comprises:

[0023] If it is determined according to the eye movement data that the user's line of sight is away from the first UI element within a first preset time length, and it is determined according to the eye movement data that the user's line of sight data matches the line of sight data corresponding to the second UI element, a control instruction corresponding to the second UI element is triggered, and the first UI element is controlled based on the control instruction.

[0024] Or

[0025] The step of determining the control of the UI element displayed in the display interface according to the eye movement data comprises:

[0026] If it is determined according to the eye movement data that the user's line of sight falls within the second preset time length of the first UI element, eye movement data meeting the preset condition is detected, a control instruction is triggered, and the first UI element is controlled based on the control instruction.

[0027] Optionally, the display interface displays a UI element to be switched, and the UI element to be switched is at least partially displayed in the display interface. The step of determining the UI element displayed in the display interface based on the eye movement data comprises the following steps.

[0028] If it is determined according to the eye movement data that the user's line of sight moves from a target UI element displayed in the display interface to the UI element to be switched, the UI element to be switched is completely displayed in the display interface, and a display area of the UI element to be switched is a display area of the target UI element.

[0029] Optionally, the UI elements displayed in the display interface at least include a first UI element and a second UI element corresponding to the first element, wherein the first UI element is used to display information, and the second UI element is used to be a UI element corresponding to a function button; the first UI element is displayed in the middle of the display interface, and the second UI element is displayed at least one of the following positions of the first UI element: above, below, left and right of the first UI element.

[0030] The step of determining the UI element displayed in the display interface based on the eye movement data comprises the following steps.

[0031] If it is determined based on the current eye movement data that the current UI element to be controlled is the first UI element, a moving direction of the user's line of sight at the next time is obtained based on the eye movement data, and a control instruction is triggered according to the second UI element corresponding to the moving direction.

[0032] The first UI element is controlled according to the control instruction.

[0033] Optionally, the step of triggering the control instruction according to the second UI element corresponding to the moving direction comprises the following steps.

[0034] When it is determined that the distance of the user's line of sight in the moving direction is greater than a second preset distance, the control instruction corresponding to the second UI element is triggered.

[0035] Optionally, the method further comprises the following steps.

[0036] An eye movement angle of the user is obtained, and the first preset distance is determined according to the eye movement angle.

[0037] In a second aspect, the embodiments of the present application further provide a head-mounted display device, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection communication between the processor and the memory, wherein the computer program, when executed by the processor, realizes the control method of the UI element according to the first aspect.

[0038] In a third aspect, the embodiments of the present application further provide a storage medium for computer-readable storage, wherein the storage medium stores one or more programs executable by one or more processors to realize the control method of the UI element according to the first aspect.

[0039] The embodiments of the present application provide a control method of a UI element, a head-mounted display device and a storage medium. In the embodiments of the present application, when the head-mounted display device starts an eye movement control function, the UI element displayed on the display interface of the head-mounted display device is acquired, wherein the distance between each UI element is greater than a first preset distance after the eye movement control function is started; the eye movement data of the user wearing the head-mounted display device is acquired; and the UI element displayed on the display interface is controlled according to the eye movement data. By making the distance between each UI element on the display interface greater than the first preset distance, when the UI element is controlled by the eye movement data of the user, the head-mounted display device will not misinterpret the intention of the user and control the UI element incorrectly due to the too close distance between the two adjacent UI elements, thereby expanding the range of eye movement and reducing the false touch rate of the eye movement control operation. In the embodiments of the present application, the user can control the UI element only by eye movement, without the need for the user to confirm the UI element determined by the eye movement data again (for example, the user needs to confirm the operation to be performed on the UI element by voice, gesture, etc. after the user's line of sight is determined to fall into the UI element by the eye movement data) such as voice, gesture, etc. Not only does this free the user's hands, but it also makes it difficult for other users to discover the control intention of the user wearing the head-mounted display device, thereby improving the privacy of the control of the UI element displayed on the head-mounted display device. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0041] Figure 1 is a structural schematic block diagram of a head-mounted display device provided by the embodiments of the present application;

[0042] Figure 2 is a flowchart of a control method of a UI element provided by an embodiment of the present application;

[0043] Figure 3 is a first reference diagram of a UI element displayed in a display interface in an embodiment of the present application;

[0044] Figure 4 is a second reference diagram of a UI element displayed in a display interface in an embodiment of the present application;

[0045] Figure 5 is a third reference diagram of a UI element displayed in a display interface in an embodiment of the present application;

[0046] Figure 6 is a fourth reference diagram of a UI element displayed in a display interface in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0048] The flowcharts shown in the drawings are only exemplary and do not necessarily include all contents and operations / steps, nor do they have to be executed in the described order. For example, some operations / steps can be further decomposed, combined or partially merged, so the actual execution order can be changed according to actual situations.

[0049] It should be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0050] The embodiments of the present application provide a control method of a UI element, a head-mounted display device and a storage medium. The technical solutions provided by the embodiments of the present application.

[0051] In the embodiments of the present application, the head-mounted display device can be an AR(Augmented Reality, augmented reality) glasses, an AR helmet, a VR(Virtual Reality, virtual reality) glasses, etc.

[0052] Some embodiments of the present application will be described in detail with reference to the drawings, which are shown by way of illustration. The following embodiments and features described below can be combined with each other in the case of no conflict.

[0053] Figure 1 is a structural schematic block diagram of a head-mounted display device provided by an embodiment of the present application.

[0054] As shown in Figure 1 , the head-mounted display device 100 includes a processor 101 and a memory 102, and the processor 101 and the memory 102 are connected through a bus 103, such as an I2C (Inter-integrated Circuit) bus.

[0055] Specifically, the processor 101 is configured to provide computing and control capabilities to support the operation of the entire head-mounted display device 100. The processor 101 can be a central processing unit (CPU), and the processor 101 can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0056] Specifically, the memory 102 can be a Flash chip, a read-only memory (ROM) disk, an optical disk, a U disk or a mobile hard disk, etc.

[0057] Those skilled in the art can understand that Figure 1 the structure shown in the above is only a block diagram of part of the structure related to the embodiment of the present application, and does not constitute a limitation on the head-mounted display device to which the embodiment of the present application is applied. The specific head-mounted display device can include more or less components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0058] The processor 101 is configured to run a computer program stored in the memory 102, and the computer program can be set in the head-mounted display device, and the head-mounted display device implements any one of the UI element control methods provided by the embodiments of the present application when executing the computer program.

[0059] In an embodiment, the processor is configured to execute a computer program stored in the memory and implement the following steps when executing the computer program:

[0060] When the head-mounted display device starts the eye movement control function, a UI element displayed by the display interface of the head-mounted display device is acquired, wherein the distance between each UI element after starting the eye movement control function is greater than a first preset distance;

[0061] Eye movement data of a user wearing the head-mounted display device is acquired;

[0062] A UI element displayed in the display interface is determined according to the eye movement data.

[0063] In an embodiment, the processor is further configured to implement the following steps when executing the computer program:

[0064] A display mode of the UI element in the display interface is determined;

[0065] The step of determining the UI element displayed in the display interface according to the eye movement data includes:

[0066] If the display mode is a first display mode, the UI element displayed in the display interface is controlled according to at least one parameter in the eye movement data;

[0067] If the display mode is a second display mode, the UI element displayed in the display interface is controlled according to at least two parameters in the eye movement data.

[0068] In an embodiment, the step of determining the display mode of the UI element in the display interface includes:

[0069] A display area of a UI element to be switched is determined, wherein the UI element to be switched is at least partially displayed in the display interface;

[0070] If the display area is greater than a first preset value, the display mode is determined to be the first display mode;

[0071] If the display area is less than or equal to a second preset value, the display mode is determined to be the second display mode;

[0072] Or

[0073] A minimum value of display areas of all UI elements displayed by the display interface is determined;

[0074] If the minimum value is greater than a second preset value, the display mode is determined to be the first display mode;

[0075] If the minimum value is less than or equal to a second preset value, the display mode is determined as a second display mode.

[0076] In an embodiment, the UI elements displayed by the display interface include at least a first UI element and a second UI element corresponding to the first UI element, wherein the first UI element is used to display information, and the second UI element is a UI element corresponding to a function button.

[0077] The step of determining, according to the eye movement data, a control of a UI element displayed in the display interface includes:

[0078] If it is determined according to the eye movement data that the user's line of sight is within a first preset time length from the first UI element, and it is determined according to the eye movement data that the user's line of sight data matches the line of sight data corresponding to the second UI element, a control instruction corresponding to the second UI element is triggered, and the first UI element is controlled based on the control instruction.

[0079] Or

[0080] The step of determining, according to the eye movement data, a control of a UI element displayed in the display interface includes:

[0081] If it is determined according to the eye movement data that the user's line of sight is within a second preset time length from the first UI element and detects eye movement data meeting a preset condition, a control instruction is triggered, and the first UI element is controlled based on the control instruction.

[0082] In an embodiment, the display interface displays a UI element to be switched, and the UI element to be switched is at least partially displayed in the display interface. The step of determining, according to the eye movement data, a control of a UI element displayed in the display interface includes:

[0083] If it is determined according to the eye movement data that the user's line of sight moves from a target UI element displayed in the display interface to the UI element to be switched, the UI element to be switched is completely displayed in the display interface, and a display area of the UI element to be switched is a display area of the target UI element.

[0084] In an embodiment, the UI elements displayed by the display interface include at least a first UI element and a second UI element corresponding to the first UI element, wherein the first UI element is used to display information, and the second UI element is a UI element corresponding to a function button; the first UI element is displayed in the middle of the display interface, and the second UI element is displayed in at least one of the following positions of the first UI element: above, below, left of, and right of the first UI element.

[0085] The step of determining, according to the eye movement data, a control of a UI element displayed in the display interface comprises:

[0086] If it is determined, based on the current eye movement data, that the UI element to be currently controlled is a first UI element, a moving direction of a user's line of sight at a next time is obtained based on the eye movement data, and a control instruction is triggered according to a second UI element corresponding to the moving direction;

[0087] The first UI element is controlled according to the control instruction.

[0088] In an embodiment, the step of triggering the control instruction according to the second UI element corresponding to the moving direction comprises:

[0089] When it is determined that a distance of the user's line of sight in the moving direction is greater than a second preset distance, the control instruction corresponding to the second UI element is triggered.

[0090] In an embodiment, the processor, when executing the computer program, is further configured to implement the following steps:

[0091] An eye movement angle of the user is obtained, and the first preset distance is determined according to the eye movement angle.

[0092] It should be noted that, for the convenience and brevity of description, the specific working process of the head-mounted display device described above can refer to the corresponding process in the following control method of a UI element, and will not be described here.

[0093] In the following, the control method of a UI element provided by the embodiments of the present application will be described in detail in combination with the head-mounted display device in Figure 1 It should be noted that, Figure 1 The head-mounted display device in is only used to explain the control method of a UI element provided by the embodiments of the present application, but does not constitute a limitation on the application scenarios of the control method of a UI element provided by the embodiments of the present application.

[0094] Figure 2 Please refer to Figure 2 , which is a flowchart of the control method of a UI element provided by the embodiments of the present application.

[0095] As shown in Figure 2 , the control method of a UI element is applied to a head-mounted display device, and the control method of a UI element comprises steps S101, S102 and S103.

[0096] In step S101, when the head-mounted display device starts the eye movement control function, the head-mounted display device acquires the UI elements displayed on the display interface of the head-mounted display device. When the eye movement control function is started, the distance between the UI elements is greater than a first preset distance.

[0097] In this embodiment, the head-mounted display device can start the eye movement control function in the head-mounted display device after receiving a starting instruction for starting the eye movement control function. The starting instruction can be automatically triggered by the head-mounted display device in a certain specific scenario. For example, when the user wearing the head-mounted display device is in a motion scenario, the head-mounted display device can automatically trigger the starting instruction for starting the eye movement control function. When the head-mounted display device enters a specific display interface, the head-mounted display device can automatically trigger the starting instruction. For example, when the head-mounted display device starts to enter the main interface of an APP (application program), the head-mounted display device can automatically trigger the starting instruction. The starting instruction can also be triggered by the user through a touch strip in the head-mounted display device, voice, or gestures. When the head-mounted display device starts the eye movement control function, the head-mounted display device acquires the UI elements displayed on the display interface. When the eye movement control function is started, the distance between the UI elements is greater than a first preset distance. The first preset distance can be set according to specific needs, and this embodiment does not limit the first preset distance. When the distance between the UI elements is greater than the first preset distance, the distance between the UI elements displayed on the display interface can be the same or different.

[0098] In an embodiment, the first preset distance is greater than the distance between the UI elements when the eye movement control function is not started, that is, the distance between the UI elements when the eye movement control function is started is greater than the distance between the UI elements when the eye movement control function is not started. For example, if the first UI element and the second UI element are two adjacent elements in the display interface, if the distance between the first UI element and the second UI element is a first distance when the eye movement control function is started, and the distance between the first UI element and the second UI element is a second distance when the eye movement control function is not started (that is, before the eye movement control function is started), the first distance is greater than the second distance. The distance between the first UI element and the second UI element can be the distance between the two adjacent sides of the first UI element and the second UI element, that is, the shortest distance between the first UI element and the second UI element, or the distance between the middle point of the first UI element and the middle point of the second UI element.

[0099] It can be understood that, because the distance between the UI elements when the eye movement control function is started is greater than the first preset distance, the distance between different UI elements is expanded on the layout of the UI elements in the display interface when the eye movement control function is started, thereby expanding the range of eye movement and reducing the false touch rate of the eye movement control operation.

[0100] In an embodiment, the method further comprises: acquiring an eye movement angle of the user, and determining the first preset distance according to the eye movement angle.

[0101] In an embodiment, the head-mounted display device acquires an eye movement angle of the user, and determines the first preset distance according to the eye movement angle. In the head-mounted display device, a mapping relationship between the eye movement angle and the first preset distance can be pre-stored. When the head-mounted display device acquires the eye movement angle, the first preset distance can be determined according to the eye movement angle and the mapping relationship. For example, when the eye movement angle is 2°, the first preset distance can be set as 10 pixel units. In this embodiment, the distance between the UI elements after the eye movement control function is started is determined according to the eye movement angle of the user, so that the distance between the UI elements is not determined randomly. If the distance between the UI elements is too small, the error touch rate of the eye movement control operation will be high. If the distance between the UI elements is too large, the display interface resources will be wasted. The distance between the UI elements is determined according to the eye movement angle of the user, so that the above situations can be avoided.

[0102] In step S102, the eye movement data of the user wearing the head-mounted display device is acquired.

[0103] In step S103, the UI elements displayed in the display interface are controlled according to the eye movement data.

[0104] The head-mounted display device acquires the eye movement data of the user, wherein the user is the user wearing the head-mounted display device. The eye movement data at least includes one of the following parameters: eye movement angle and eye movement direction. The eye movement data can also include the following parameters: eye movement distance, distance between the fixation point of the user's line of sight and the eye, movement duration of the eye movement, and stop duration of the eye movement. The head-mounted display device can acquire the eye movement data of the user through an eye tracker, or through other related technologies. After the head-mounted display device acquires the eye movement data of the user, the head-mounted display device controls the UI elements displayed in the display interface according to the eye movement data. During the process of controlling the UI elements displayed in the display interface according to the eye movement data, the head-mounted display device performs different control operations based on different UI elements displayed in the display interface. For example, the head-mounted display device can switch the currently displayed UI element, delete the displayed UI element, or enter the sub UI element corresponding to the currently displayed UI element according to the eye movement data. For example, the currently displayed UI element is a web page, and there is a link in the web page. The head-mounted display device can switch from the current web page to the page corresponding to the link according to the eye movement data, and the page corresponding to the link is the sub UI element corresponding to the currently displayed UI element.

[0105] The embodiment of the present application acquires the UI elements displayed by the display interface of the head-mounted display device after the head-mounted display device starts the eye movement control function, wherein the distance between each UI element after the eye movement control function is started is greater than a first preset distance; acquires the eye movement data of the user wearing the head-mounted display device; and controls the UI elements displayed in the display interface according to the eye movement data. By making the distance between each UI element in the display interface greater than the first preset distance, when the UI elements are controlled by the user's eye movement data, the head-mounted display device will not misinterpret the user's intention and control the wrong UI element due to the too close distance between the two adjacent UI elements, thereby expanding the range of eye movement and reducing the false touch rate of eye movement control operation. In the embodiment of the present application, the user can only control the UI elements by eye movement, without the need for the user to further confirm the UI elements determined by the eye movement data through voice, gestures, etc. (such as confirming through voice, gestures, etc. that the user's line of sight falls into a certain UI element, and confirming that the user needs to perform what operation on the UI element), not only freeing the user's hands, but also preventing other users from discovering the control intention of the user wearing the head-mounted display device, thereby improving the privacy of the control of the UI elements displayed by the head-mounted display device.

[0106] In an embodiment, the method further comprises:

[0107] Step a, determining the display mode of the UI element in the display interface.

[0108] In an embodiment, the head-mounted display device determines the display mode of the UI element in the display interface, wherein the display mode of the UI element includes two types, which are the first display mode and the second display mode. The specific definition and distinction of the first display mode and the second display mode can be referred to the following description, which will not be repeated here.

[0109] In an embodiment, step a can be implemented in at least the following two ways:

[0110] Method one: determining the display area of the UI element to be switched, wherein the UI element to be switched is at least partially displayed in the display interface; if the display area is greater than a first preset value, the display mode is determined to be the first display mode; if the display area is less than or equal to a second preset value, the display mode is determined to be the second display mode.

[0111] In an embodiment, the UI element to be switched in the display interface of the head-mounted display device is included in an edge region of the display interface, i.e., the middle region of the display interface is not currently displayed, and the UI element to be switched is at least partially displayed in the display interface. Referring to the two UI elements of "Function 3" and "Function 4" in Figure 3 , "Function 3" and "Function 4" are not completely displayed in the display interface, in some embodiments, the UI element to be switched can also be completely displayed in the display interface, such as "Function 3" in Figure 6 . The UI element displayed in the middle region of the display interface is the UI element of the current display information, and the user can switch the UI element to be switched to the middle region of the display interface through the eye movement data, at which time the UI element to be switched is completely displayed in the display interface.

[0112] When the display interface of the head-mounted display device includes the UI element to be switched, the head-mounted display device determines the display area of the UI element to be switched. The display area of the UI element to be switched is the size of the area currently displayed in the display interface, and is not equal to the display area when the UI element to be switched is normally displayed in the middle region of the display interface. If the display area of the UI element to be switched is greater than a first preset value, the head-mounted display device determines that the display mode of the UI element in the display interface is a first display mode; if the display area of the UI element to be switched is less than or equal to the first preset value, the head-mounted display device determines that the display mode of the UI element in the display interface is a second display mode. The size of the first preset value can be set according to specific needs, and the size of the first preset value is not specifically limited in this embodiment.

[0113] It can be understood that when the display area of the UI element to be switched is greater than the first preset value, it indicates that the display area of the UI element to be switched is large enough, at which time in one direction, there can be one UI element to be switched, and therefore the control of the UI element can be completed through at least one parameter in the eye movement data. When the display area of the UI element to be switched is less than or equal to the first preset value, it indicates that the display area of the UI element to be switched is relatively small, at which time in one direction, there can be multiple UI elements to be switched, and therefore the control of the UI element needs to be completed through at least two parameters in the eye movement data.

[0114] In this embodiment, in addition to the UI element to be switched, the UI element displayed in the middle region of the display interface of the head-mounted display device also includes a first UI element and a second UI element corresponding to the first UI element, wherein the first UI element is used to display information, and the second UI element is a UI element corresponding to a function button. The description of the first UI element and the second UI element can be referred to the following embodiments, which will not be repeated here.

[0115] In a second mode, the head-mounted display device determines a minimum value of display areas of all UI elements displayed in the display interface. If the minimum value is greater than a second preset value, the head-mounted display device determines that the display mode is the first display mode. If the minimum value is less than or equal to the second preset value, the head-mounted display device determines that the display mode is the second display mode.

[0116] In an embodiment, the head-mounted display device obtains display areas of all UI elements displayed in the display interface. If the display interface includes the UI element to be switched, the obtained display areas of all UI elements also include the display area of the UI element to be switched. After the head-mounted display device obtains the display areas of all UI elements, the head-mounted display device determines a minimum value of the display areas of all UI elements, and determines whether the minimum value is greater than a second preset value. The second preset value can be equal to or different from the first preset value. When the head-mounted display device determines that the minimum value is greater than the second preset value, the head-mounted display device determines that the display mode of the UI element is the first display mode. When the head-mounted display device determines that the minimum value is less than or equal to the second preset value, the head-mounted display device determines that the display mode of the UI element is the second display mode.

[0117] In an embodiment, since the display interface only displays the first UI element, the second UI element corresponding to the first UI element, and the UI element to be switched, the head-mounted display device can also determine the number of the UI element to be switched in the same direction of the first UI element to determine the display mode of the UI element in the display interface. When there is only one UI element to be switched in the same direction (e.g., left or right) of the first UI element, the head-mounted display device determines that the display mode of the UI element is the first display mode. When there are two or more UI elements to be switched in the same direction of the first UI element, the head-mounted display device determines that the display mode of the UI element is the second display mode.

[0118] The step S103 includes:

[0119] In step b, if the display mode is the first display mode, the head-mounted display device controls the UI element displayed in the display interface according to at least one parameter in the eye movement data.

[0120] If the head-mounted display device determines that the display mode of the UI element is the first display mode, the head-mounted display device controls the UI element displayed in the display interface according to at least one parameter in the eye movement data. It can be understood that when the display mode of the UI element is the first display mode, it indicates that only the eye movement data is matched with the UI element in a large range when the UI element is controlled. For reference Figure 4 In Figure 4The UI elements contained in the middle include "back", "confirm", "function two" and "function three". If the user wants to switch from "function two" to "function three", the head-mounted display device can determine the control intention of the user to switch from "function two" to "function three" as long as the eye movement direction of the user is determined to be from left to right according to the eye movement data, and the eye movement direction and the position relationship between "function two" and "function three" match. Since "function three" is on the right of "function two", when the eye movement direction is from "function two" to the right, the head-mounted display device can determine that the user wants to switch "function two" to "function three". When "function two" is switched to "function three", the head-mounted display device displays "function three" in the middle area of the display interface, that is, "function three" is displayed in the position of "function two", and "function two" is not displayed in the central area of the display interface. At this time, the head-mounted display device can control the switching of the UI elements in the display interface according to the eye movement direction as one parameter.

[0121] In step c, if the display mode is the second display mode, the UI elements displayed in the display interface are controlled according to at least two parameters in the eye movement data.

[0122] If the head-mounted display device determines that the display mode of the UI element is the second display mode, the head-mounted display device controls the UI elements displayed in the display interface according to at least two parameters in the eye movement data. It can be understood that when the display mode of the UI element is the second display mode, it indicates that when the UI element is controlled, the eye movement data needs to be more accurately matched with the UI element than the first display mode, because the display area of the UI element is smaller at this time. If the eye movement data is not more accurately matched with the UI element, the situation of controlling the wrong UI element is likely to occur. For reference Figure 3 In Figure 3The UI elements contained in the middle include "back", "confirm", "function two", "function three" and "function four", but the display area of "function three" and "function four" is small, and if only the eye movement direction is from left to right, it cannot be determined whether "function two" is switched to "function three" or "function two" is switched to "function four", so the head-mounted display device needs to determine at least two parameters in the eye movement data to determine whether "function two" is switched to "function three" or "function two" is switched to "function four". At least two parameters of the eye movement data determined at this time are the eye movement direction and the eye movement angle. When "function three" and "function four" are both on the right side of "function two", if the eye movement direction at this time is from left to right, it can be determined that the user's intention is to switch "function two" to "function three" or to switch "function two" to "function four"; further, according to the eye movement angle, it can be determined that the user's intention is to switch "function two" to "function three" or to switch "function two" to "function four", so the relative position angle of "function three" and "function four" to "function two" is different, and the relative position angle is determined based on the eye position.

[0123] In an embodiment, in order to improve the accuracy of eye movement control operation, if the head-mounted display device determines that the display mode of the UI element is the first display mode, the display of the UI element in the display interface is controlled according to at least two parameters in the eye movement data; if it is determined that the display mode of the UI element is the second display mode, the display of the UI element in the display interface is controlled according to at least three parameters in the eye movement data.

[0124] The embodiment gives different control strategies for controlling UI elements (whether to control the display of the UI element in the display interface according to at least one parameter in the eye movement data or to control the display of the UI element in the display interface according to at least two parameters in the eye movement data) based on the different display modes of the UI elements in the display interface, thereby reducing the data processing amount of the head-mounted display device to a certain extent when accurately controlling the UI elements through eye movement.

[0125] In an embodiment, the UI elements displayed by the display interface include at least a first UI element and a second UI element corresponding to the first element, wherein the first UI element is used to display information, and the second UI element is a UI element corresponding to a function button.

[0126] In an embodiment, the head-mounted display device comprises a first UI element and a second UI element corresponding to the first element, wherein the first UI element is used to display information, and the second UI element is a UI element corresponding to a function button. The second UI elements corresponding to different first UI elements can be the same or different, and the second UI element corresponding to each first UI element is pre-set. When the first UI element in the middle region of the display interface changes, the corresponding second UI element will also change correspondingly. For example, in Figure 4 , if the first UI element in the middle region of the display interface is switched from "function two" to "function three", the second UI element corresponding to "function two" will also change to the second UI element corresponding to "function three". It can be understood that the number of second UI elements corresponding to different first UI elements can be equal or not equal. For example, as shown in Figure 4 , the second UI elements corresponding to "function two" are "confirm" and "return". As shown in Figure 5 , the second UI elements corresponding to "function three" are "confirm", "return" and "delete".

[0127] In an embodiment, the second UI element corresponding to the UI element to be switched is not displayed in the display interface at the beginning, and only after the UI element to be switched is switched to the middle region of the display interface, the second UI element corresponding to the UI element to be switched is displayed in the display interface.

[0128] The step S103 comprises:

[0129] Step d, if it is determined according to the eye movement data that the user's line of sight is out of the first UI element within the first preset time length, it is determined according to the eye movement data that the user's line of sight data matches the line of sight data corresponding to the second UI element, a control instruction corresponding to the second UI element is triggered, and the first UI element is controlled based on the control instruction.

[0130] When the head-mounted display device obtains the eye movement data, the head-mounted display device can determine the user's visual line based on the eye movement data. The process of determining the user's visual line based on the eye movement data can refer to related technologies, and this embodiment will not be described in detail. When the head-mounted display device determines the user's visual line, if the head-mounted display device determines that the user's visual line is within a first preset time period from the first UI element, the user's visual line data matches the visual line data corresponding to the second UI element, the head-mounted display device triggers the control instruction corresponding to the second UI element, and controls the first UI element based on the control instruction. The size of the first preset time period can be set as needed, such as 1.5 seconds, or 3 seconds, etc. It can be understood that the user's visual line data matches the visual line data corresponding to the second UI element, that is, the user's visual line data is within the visual line data range corresponding to the second UI element, or the user's visual line data is the same as the visual line data corresponding to the second UI element. Referring to Figure 3 If the user's visual line is within the first preset time period from the first UI element "Function 2", the user's visual line data is determined to be "upward movement X cm (centimeters)" based on the eye movement data, and the visual line data corresponding to the second UI element is at least Y cm upward movement based on the first UI element, and X is greater than Y, it can be determined that the user's visual line data matches the visual line data corresponding to the second UI element, and at this time the head-mounted display device can trigger the control instruction corresponding to the second UI element, that is, trigger the return instruction to make the first UI element return to the previous page or the previous operation.

[0131] Or include:

[0132] Step e, if the eye movement data that meets the preset condition is detected within the second preset time period according to the eye movement data that the user's visual line falls into the first UI element, the control instruction is triggered, and the first UI element is controlled based on the control instruction.

[0133] If the head-mounted display device determines that the eye movement data that meets the preset condition is detected within the second preset time period according to the eye movement data that the user's visual line falls into the first UI element, the head-mounted display device triggers the control instruction, and controls the first UI element based on the control instruction. The second preset time period can be equal to the first preset time period, or can be different from the first preset time period. The preset condition can be determined according to the information displayed in the first UI element. For example, when the information displayed in the first UI element includes a certain link, if the eye movement data that meets the preset condition is detected within the second preset time period according to the eye movement data that the user's visual line corresponds to the fixation point falls into the position corresponding to the link, and at this time the user's blinking action is detected, or the user's visual line corresponding to the fixation point is not removed from the position corresponding to the link within a certain time period, it is considered that the eye movement data that meets the preset condition is detected within the second preset time period, at this time the head-mounted display device can trigger the entering instruction of the page corresponding to the link, and display the page corresponding to the link based on the entering instruction.

[0134] The embodiment realizes the control of the display content of the head-mounted display device based on the eye movement data only, that is, the control of the UI element displayed in the display interface of the head-mounted display device by the line of sight, so that other users cannot determine the control intention of the user wearing the head-mounted display device, and the privacy of the control of the display content of the head-mounted display device is improved.

[0135] In an embodiment, the display interface displays a UI element to be switched, and the UI element to be switched is at least partially displayed in the display interface. The step S103 comprises:

[0136] If it is determined according to the eye movement data that the line of sight of the user moves from a target UI element displayed in the display interface to the UI element to be switched, the UI element to be switched is completely displayed in the display interface, and the display area of the UI element to be switched is the display area of the target UI element.

[0137] In an embodiment, the display interface of the head-mounted display device displays a UI element to be switched, and the UI element to be switched is at least partially displayed in the display interface. If the head-mounted display device determines according to the eye movement data that the line of sight of the user moves from a target UI element displayed in the display interface to the UI element to be switched, the head-mounted display device completely displays the UI element to be switched in the display interface, and displays the UI element to be switched in the middle region of the display interface, and the display area of the UI element to be switched is the display area of the target UI element, that is, the UI element to be switched is displayed at the position where the target UI element is displayed. The target UI element is one UI element in the display interface determined by the user to be switched with the UI element to be switched. It can be understood that, Figure 4 and Figure 6 The target UI element in the above embodiment is "function two", and the UI element to be switched is "function three". When the head-mounted display device completely displays the UI element to be switched in the display interface, the display state corresponding to "function three" in the above embodiment can be referred to. Figure 5

[0138] In an embodiment, before the UI element to be switched is switched, the UI element to be switched can also be completely displayed in the display interface. However, the display area of the UI element to be switched is smaller than that when the UI element to be switched is displayed in the middle region of the display interface before the UI element to be switched is switched, that is, before the UI element to be switched is displayed in the middle region of the display interface. The second UI element corresponding to the UI element to be switched is also hidden, as shown in the above embodiment. Only when the UI element to be switched is switched to the middle region of the display interface, the UI element to be switched and the second UI element corresponding to the UI element to be switched are normally displayed, as shown in the above embodiment. Figure 6 Figure 5 ​​The first UI element and the second UI element are illustrated in FIG. 1. It can be understood that the UI element to be switched is also the first UI element, and the description between the first UI element and the second UI element can refer to other embodiments, which will not be repeated herein.

[0139] In the embodiment, the switching of the UI element in the display interface can be realized by the movement of the user's line of sight, and the switching efficiency of the UI element is improved.

[0140] In an embodiment, in order to improve the accuracy of the switching of the UI element and avoid misoperation, when the head-mounted display device determines that the user's line of sight moves from the target UI element displayed in the display interface to the UI element to be switched according to the eye movement data, the movement distance of the user's line of sight is determined, and only when the movement distance is greater than a preset distance value, the head-mounted display device will switch the UI element to be switched to be displayed in the display interface, and the preset distance value is greater than or equal to the distance between the adjacent two edges of the target UI element and the UI element to be switched.

[0141] In an embodiment, the UI element displayed in the display interface includes at least a first UI element and a second UI element corresponding to the first element, wherein the first UI element is used to display information, and the second UI element is used for the UI element corresponding to the function button; the first UI element is displayed in the middle of the display interface, and the second UI element is displayed in at least one of the following positions of the first UI element: above, below, left and right of the first UI element.

[0142] The step S103 includes:

[0143] Step g, if the current UI element to be controlled is the first UI element based on the current eye movement data, the movement direction of the user's line of sight at the next moment is obtained based on the eye movement data, and the control instruction is triggered according to the second UI element corresponding to the movement direction.

[0144] Step f, the first UI element is controlled according to the control instruction.

[0145] In an embodiment, the UI element displayed in the display interface of the head-mounted display device includes at least a first UI element and a second UI element corresponding to the first element, wherein the first UI element is used to display information, and the second UI element is used for the UI element corresponding to the function button, the first UI element is displayed in the middle of the display interface, and the second UI element is displayed in at least one of the following positions of the first UI element: above, below, left and right of the first UI element. It can be understood that the second UI element is displayed around the first UI element, so that when the user's eye movement amplitude is small, the control instruction for controlling the first UI element can be triggered through the second UI element, thereby improving the control efficiency of the first UI element through the second UI element.

[0146] When the head-mounted display device determines that the UI element to be controlled currently is the first UI element based on the current eye movement data, the head-mounted display device obtains the moving direction of the user's line of sight at the next time based on the eye movement data. It can be understood that the head-mounted display device is continuously obtaining the eye movement data of the user, and the "current" and "next time" in the embodiment are only to distinguish the eye movement data at two different times. After the head-mounted display device obtains the moving direction of the user's line of sight, if the head-mounted display device determines that the moving direction of the user's line of sight is the direction of the second UI element, a control instruction is triggered according to the second UI element corresponding to the moving direction, and the first UI element is controlled according to the control instruction. If the first UI element is a photo, and the second UI element corresponding to the moving direction is a key corresponding to a "delete" function, the head-mounted display device triggers a delete instruction, so as to delete the photo displayed by the first UI element.

[0147] In an embodiment, the step of triggering a control instruction according to the second UI element corresponding to the moving direction in step g comprises:

[0148] In step g1, when it is determined that the distance of the user's line of sight in the moving direction is greater than a second preset distance, the control instruction corresponding to the second UI element is triggered.

[0149] In an embodiment, after the head-mounted display device obtains the moving direction of the user's line of sight at the next time based on the eye movement data, the head-mounted display device determines the distance of the user's line of sight in the moving direction, and obtains the starting point of the user's line of sight when moving from the first UI element to the second UI element, calculates the shortest distance from the starting point to the second UI element, and determines the second preset distance based on the shortest distance, that is, the second preset distance is greater than or equal to the shortest distance. It can be understood that the starting point is in the first UI element. When the head-mounted display device determines that the distance of the user's line of sight in the moving direction is greater than the second preset distance, the head-mounted display device triggers the control instruction corresponding to the second UI element; when the head-mounted display device determines that the distance of the user's line of sight in the moving direction is less than or equal to the second preset distance, the head-mounted display device does not trigger the control instruction.

[0150] In the embodiment, the first UI element is set on the display interface, and the second UI element that can trigger control of the first UI element is set, so that when the UI element that needs to be controlled is determined through the eye movement data, the control instruction for triggering control of the first UI element is not required to be triggered through other manners such as voice or gesture, and the control of the first UI element can be realized only through the eye movement data. In another embodiment, the control instruction corresponding to the second UI element is triggered only when the distance of the user's line of sight in the moving direction is greater than the second preset distance, so as to control the first UI element through the control instruction, thereby avoiding that when the display area of the first UI element is relatively large, the moving direction of the user's line of sight corresponds to the position of the second UI element, but the user's line of sight is still in the range corresponding to the first UI element, the control instruction corresponding to the second UI element is triggered, and the accuracy of controlling the first UI element through the eye movement data is improved.

[0151] The embodiment of the present application further provides a storage medium for computer readable storage, the storage medium storing one or more programs, the one or more programs being executable by one or more processors to implement any one of the control methods of the UI element provided in the specification of the embodiment of the present application.

[0152] The storage medium can be an internal storage unit of the head-mounted display device, for example, a hard disk or a memory of the head-mounted display device. The storage medium can also be an external storage device of the head-mounted display device, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0153] Those skilled in the art can understand that all or some of the steps in the methods disclosed above, the functional modules / units in the systems and devices can be implemented by software, firmware, hardware, or a combination thereof. In hardware embodiments, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component can have multiple functions, or one function or step can be performed by several physical components in cooperation. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on computer-readable media, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and can include any information delivery media.

[0154] It should be understood that the term "and / or" as used herein refers to any combination of associated listed items, and all possible combinations, and includes these combinations. It should be noted that the terms "comprising", "including", or any other variant thereof, are intended to cover non-exclusive inclusion, so that processes, methods, articles, or systems including a series of elements not only include those elements, but also include other elements not explicitly listed, or other elements inherent to such processes, methods, articles, or systems. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or system including the element.

[0155] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments. The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for controlling UI elements, characterized in that, The method is applied to a head-mounted display device, and the method includes: When the head-mounted display device activates the eye-tracking control function, the UI elements displayed on the display interface of the head-mounted display device are acquired, wherein the distance between each UI element is greater than a first preset distance after the eye-tracking control function is activated. Acquire eye movement data of the user wearing the head-mounted display device; The UI elements displayed in the display interface are determined based on the eye movement data.

2. The method for controlling UI elements according to claim 1, characterized in that, The method further includes: Determine how the UI element is displayed in the display interface; The step of determining the UI elements displayed in the display interface based on the eye movement data includes: If the display mode is the first display mode, then the UI elements displayed in the display interface are controlled according to at least one parameter in the eye movement data; If the display mode is the second display mode, then the UI elements displayed in the display interface are controlled according to at least two parameters in the eye movement data.

3. The method for controlling UI elements according to claim 2, characterized in that, The step of determining the display method of the UI element in the display interface includes: Determine the display area of ​​the UI element to be switched, wherein the UI element to be switched is at least partially displayed in the display interface; If the display area is greater than a first preset value, then the display mode is determined to be the first display mode; If the display area is less than or equal to the second preset value, then the display mode is determined to be the second display mode; or Determine the minimum display area among all UI elements displayed on the display interface; If the minimum value is greater than the second preset value, then the display mode is determined to be the first display mode; If the minimum value is less than or equal to the second preset value, then the display mode is determined to be the second display mode.

4. The method for controlling UI elements according to claim 1, characterized in that, The UI elements displayed on the display interface include at least a first UI element and a second UI element corresponding to the first element, wherein the first UI element is used to display information and the second UI element is a UI element corresponding to a function button; The step of determining the UI elements displayed in the display interface based on the eye movement data includes: If, based on eye movement data, it is determined that within a first preset time period after the user's gaze leaves the first UI element, the user's gaze data matches the gaze data corresponding to the second UI element, then a control command corresponding to the second UI element is triggered, and the first UI element is controlled based on the control command. or The step of determining the UI elements displayed in the display interface based on the eye movement data includes: If eye movement data that meets preset conditions is detected within a second preset time period when the user's gaze falls on the first UI element, a control command is triggered, and the first UI element is controlled based on the control command.

5. The method for controlling UI elements according to claim 1, characterized in that, The display interface displays UI elements to be switched, and the UI elements to be switched are at least partially displayed on the display interface. The step of determining the UI elements displayed on the display interface based on the eye movement data further includes: If, based on the eye movement data, it is determined that the user's gaze moves from the target UI element displayed on the display interface to the UI element to be switched, the UI element to be switched is fully displayed on the display interface, and the display area of ​​the UI element to be switched is the display area of ​​the target UI element.

6. The method for controlling UI elements according to claim 1, characterized in that, The UI elements displayed on the display interface include at least a first UI element and a second UI element corresponding to the first element. The first UI element is used to display information, and the second UI element is used as the UI element corresponding to the function button. The first UI element is displayed in the middle of the display interface, and the second UI element is displayed in at least one of the following positions of the first UI element: above, below, to the left, and to the right of the first UI element. The step of determining the UI elements displayed in the display interface based on the eye movement data includes: If the UI element to be controlled is determined to be the first UI element based on the current eye movement data, then the direction of the user's gaze movement at the next moment is obtained based on the eye movement data, and the control command is triggered according to the second UI element corresponding to the movement direction. The first UI element is controlled according to the control instructions.

7. The method for controlling UI elements according to claim 6, characterized in that, The step of triggering a control command based on the second UI element corresponding to the movement direction includes: When it is determined that the distance of the user's line of sight in the direction of movement is greater than a second preset distance, the control command corresponding to the second UI element is triggered.

8. The method for controlling UI elements according to any one of claims 1 to 7, characterized in that, The method further includes: The user's eye movement angle is obtained, and the first preset distance is determined based on the eye movement angle.

9. A head-mounted display device, characterized in that, The head-mounted display device includes a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for enabling communication between the processor and the memory, wherein when the computer program is executed by the processor, it implements a method for controlling UI elements as described in any one of claims 1 to 8.

10. A storage medium for computer-readable storage, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the UI element control method according to any one of claims 1 to 8.

Images