Control selection method and apparatus, electronic device, and computer-readable storage medium

Abstract

This application discloses a control selection method, apparatus, electronic device, and computer-readable storage medium. Embodiments of this application display a target screen, which includes multiple controls. In response to a user's first gaze operation on the target screen, a first control corresponding to the first gaze operation is determined. In response to a user's second gaze operation on the target screen, a second control corresponding to the second gaze operation is determined. A path from the first control to the second control is determined, wherein the path consists of two or more controls. In response to a confirmation operation on the target screen, the controls on the path are selected. Embodiments of this application can improve the efficiency of control selection.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to a control selection method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the development of science and technology, head-mounted displays based on Virtual Reality (VR) technology are finding increasingly wider applications. For example, they can be used to play games.

[0003] When playing games using a head-mounted display (HMD), the controls a player wants to trigger can be determined by the player's gaze focus on the HMD. However, when a player wants to trigger multiple controls simultaneously, their eyes need to move multiple times on the HMD, making control selection very inefficient. Summary of the Invention

[0004] This application provides a control selection method, apparatus, electronic device, and computer-readable storage medium, which can improve the efficiency of control selection.

[0005] In a first aspect, embodiments of this application provide a control selection method, the method comprising:

[0006] Display the target screen, which includes multiple controls;

[0007] In response to the user's first gaze operation on the target screen, determine the first control corresponding to the first gaze operation;

[0008] In response to the user's second gaze operation on the target screen, determine the second control corresponding to the second gaze operation;

[0009] Determine the path from the first control to the second control, wherein the path consists of two or more controls;

[0010] In response to the confirmation operation on the target screen, the controls on the path are selected.

[0011] Secondly, embodiments of this application also provide a control selection device, the device comprising:

[0012] The screen display module is used to display the target screen, which includes multiple controls;

[0013] The first response module is used to respond to the user's first gaze operation on the target screen and determine the first control corresponding to the first gaze operation.

[0014] The second response module is used to respond to the user's second gaze operation on the target screen and determine the second control corresponding to the second gaze operation.

[0015] The control determination module is used to determine the path from the first control to the second control, wherein the path consists of two or more controls;

[0016] The control selection module is used to select controls on the path in response to the confirmation operation of the target screen.

[0017] Thirdly, embodiments of this application also provide an electronic device, including a memory storing multiple instructions; the processor loads instructions from the memory to execute any of the control selection methods provided in embodiments of this application.

[0018] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute any of the control selection methods provided in embodiments of this application.

[0019] In this embodiment, a target screen is displayed, which includes multiple controls. In response to a user's first gaze operation on the target screen, a first control corresponding to the first gaze operation is determined. In response to a user's second gaze operation on the target screen, a second control corresponding to the second gaze operation is determined. A path from the first control to the second control is determined, wherein the path consists of two or more controls. In response to a confirmation operation on the target screen, the controls on the path are identified as selected controls. This solution performs simple tracking of the player's gaze, thereby determining the path through the player's gaze, and then selecting multiple controls at once through the path, eliminating the need for the user to gaze at all the controls to be selected, thus improving the efficiency of control selection. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the control selection system provided in an embodiment of this application;

[0022] Figure 2 This is a flowchart illustrating one embodiment of the control selection method provided in this application.

[0023] Figure 3 This is a schematic diagram of the target screen provided in the embodiments of this application;

[0024] Figure 4 This is a schematic diagram of another target screen provided in the embodiments of this application;

[0025] Figure 5 This is a schematic diagram of the line-of-sight trajectory and path provided in the embodiments of this application;

[0026] Figure 6 This is a schematic diagram of another line-of-sight trajectory and path provided in the embodiments of this application;

[0027] Figure 7 This is a schematic diagram of the finger movement trajectory provided in the embodiments of this application;

[0028] Figure 8 This is a schematic diagram of finger pinching provided in the embodiments of this application;

[0029] Figure 9 This is a schematic diagram of the third style provided in the embodiments of this application;

[0030] Figure 10 This is a schematic diagram of the fourth style provided in the embodiments of this application;

[0031] Figure 11 This is a schematic diagram of another embodiment of the control selection method provided in this application.

[0032] Figure 12 This is a schematic diagram of the structure of the head-mounted display device provided in the embodiments of this application;

[0033] Figure 13 This is a schematic diagram of the control selection device provided in the embodiments of this application;

[0034] Figure 14 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] Before providing a detailed explanation of the embodiments of this application, some terms involved in the embodiments of this application will be explained.

[0037] In the description of the embodiments of this application, the terms "first," "second," etc., may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0038] This application provides a control selection method, apparatus, electronic device, and computer-readable storage medium. Specifically, the control selection method of this application can be executed by an electronic device, which can be a terminal or server, etc.

[0039] The terminal can be a smartphone, tablet, laptop, touch screen, game console, personal computer (PC), personal digital assistant (PDA), or head-mounted display device, etc. The terminal can also include a client, which can be a game application client, a browser client with game programs, or an instant messaging client, etc.

[0040] A server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0041] For example, such as Figure 1 As shown, the electronic device is illustrated using terminal 10 as an example. This terminal can display a target screen, which includes multiple controls. In response to a user's first gaze operation on the target screen, a first control corresponding to the first gaze operation is determined. In response to a user's second gaze operation on the target screen, a second control corresponding to the second gaze operation is determined, and the first and second controls are sent to server 20. Server 20 determines the path from the first control to the second control, wherein the path consists of two or more controls, and returns the path to terminal 10. In response to a confirmation operation on the target screen, terminal 10 determines the controls on the path as the selected controls.

[0042] The following is a detailed description in conjunction with the accompanying drawings. It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments. Although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown in the drawings.

[0043] In this embodiment, a terminal is used as an example for illustration. This embodiment provides a control selection method, such as... Figure 2 As shown, the specific process of this control selection method can be as follows:

[0044] 201. Display the target screen, which includes multiple controls.

[0045] The target screen can be an interface provided by the client, which can be a UI interface for user interaction. The type of client can be selected according to the actual situation; for example, the client can be a game client or a camera client, and this embodiment of the application does not limit it.

[0046] The form of the client can be set according to the actual situation. For example, the client can exist in the form of a mini-program, an application, or a webpage. This application embodiment does not limit this.

[0047] The positions of the multiple controls on the target screen can be set according to the actual situation. For example, if the target screen includes 9 controls, their positions on the target screen can be set as follows: Figure 3 As shown, for example, the target screen includes 6 controls, and the positions of the 6 controls on the target screen can be as follows: Figure 4 As shown, the embodiments in this application are not limited here.

[0048] The object represented by the control can be set according to the actual situation. For example, the object represented by the control can be a game prop. In this case, the embodiment of this application can be applied to a game scene. Or, the object represented by the control can be a photo. In this case, the embodiment of this application can be applied to a photo selection scene. The embodiment of this application is not limited here.

[0049] The terminal can respond to the trigger operation of the client to display the target screen, or the terminal can also obtain the display instruction sent by other devices and display the target screen according to the display instruction. This application embodiment does not limit this.

[0050] 202. In response to the user's first gaze operation on the target screen, determine the first control corresponding to the first gaze operation.

[0051] The first gaze operation refers to the viewing operation of a user of an electronic device on a certain control in the target screen. When a user of an electronic device performs a first gaze operation on a certain control, that control can be identified as the first control corresponding to the first gaze operation. That is, the first control is the control at the position of the user's gaze point on the target screen.

[0052] Optionally, when the user's gaze on the first control is equal to a preset time length, the terminal responds to the user's first gaze operation on the target screen and determines the first control corresponding to the first gaze operation. The preset time length can be set according to the actual situation. For example, the preset time length can be equal to 2 seconds or 3 seconds. This application embodiment does not limit it here.

[0053] Optionally, when the terminal determines the first control corresponding to the first gaze operation, the first control can also be displayed in a first style. When the first control is displayed in the first style, it indicates that the first control is in a pre-selected state. The first style can be set according to the actual situation. For example, the first style can be a display with a preset fill color or a thick outline. This embodiment of the application does not limit this.

[0054] 203. In response to the user's second gaze operation on the target screen, determine the second control corresponding to the second gaze operation.

[0055] The second gaze operation refers to the viewing operation of a user of an electronic device on a certain control in the target screen. When the user of the electronic device performs a second gaze operation on a certain control, that control is the second control corresponding to the second gaze operation. In other words, the second control is the control at the position of the user's gaze point on the target screen.

[0056] It should be noted that, in the embodiments of this application, when the user's gaze lingers at any position in the target image for a certain duration, it is considered that the user's gaze operation has been detected at that position, that is, the first gaze operation or the second gaze operation in this application has been detected.

[0057] Optionally, when the user's gaze on the second control for the duration equal to the target preset time length, the terminal responds to the user's second gaze operation on the target screen and determines the second control corresponding to the second gaze operation. The target preset time length can be set according to the actual situation. For example, the preset time length can be equal to 2 seconds or 3 seconds. This application embodiment does not limit it here.

[0058] Optionally, when the terminal determines the second control corresponding to the second gaze operation, the second control can also be displayed in a second style. When the second control is displayed in the second style, it indicates that the second control is in a pre-selected state. The second style and the first style can be the same or different.

[0059] For example, when the first style and the second style are the same, and both the first style and the second style are filled with dark gray, the first control and the second control can be displayed as follows: Figure 5 As shown, the first control is control c2, and the second control is control c5.

[0060] In some embodiments, in response to a user's second gaze operation on a target screen, determining a second control corresponding to the second gaze operation includes:

[0061] In response to a second gaze operation by the user on the target screen within a preset time after the first gaze operation, the second control corresponding to the second gaze operation is determined.

[0062] The preset duration can be set according to the actual situation. For example, the preset duration can be 2 seconds or 3 seconds. This embodiment of the application does not limit it.

[0063] In this embodiment of the application, in response to a second gaze operation of the user on the target screen within a preset time after the first gaze operation, a second control corresponding to the second gaze operation is determined, so that the second control corresponding to the second gaze operation is determined only when the second gaze operation is executed within the preset time after the first gaze operation, so that within the preset time after the first gaze operation, the user's second gaze operation on the target screen can be responded to, thereby obtaining the start point and end point of the path from the first gaze operation and the second gaze operation respectively.

[0064] 204. Determine the path from the first control to the second control, wherein the path consists of two or more controls.

[0065] The path consists of two or more controls, which can be understood as the path including at least two controls.

[0066] In this embodiment, the user only needs to select the first control and the second control through gaze operation, and then the controls on the path from the first control to the second control are the controls that the user needs to select. The user does not need to gaze at all the controls that need to be selected, which improves the convenience of users of electronic devices to select multiple controls through gaze operation, thereby improving the efficiency of selecting multiple controls.

[0067] In some embodiments, determining the path from the first control to the second control includes:

[0068] Obtain the user's gaze trajectory as they move from the first control to the second control;

[0069] Determine the path from the first control to the second control based on the line of sight.

[0070] In this embodiment, the user's gaze trajectory from the first control to the second control is obtained; based on the gaze trajectory, the path from the first control to the second control is determined, thereby realizing the determination of the distance from the first control to the second control based on the gaze trajectory.

[0071] Optionally, the method for determining the path from the first control to the second control based on the gaze trajectory can be set according to the actual situation. For example, the gaze trajectory can be directly determined as the path from the first control to the second control, and / or the path from the first control to the second control can be determined according to the arrangement order of the controls in the target screen. This embodiment of the application does not limit this.

[0072] When the gaze path is directly defined as the path from the first control to the second control, and / or multiple controls are arranged sequentially in the target screen, and the path from the first control to the second control is determined based on the arrangement order of the controls in the target screen, determining the path from the first control to the second control based on the gaze path includes:

[0073] When the gaze path is a vertical path, the gaze path is determined as the path from the first control to the second control, and / or, the intermediate control between the first control and the second control is determined according to the arrangement order of multiple controls in the target screen, and the path between the first control, the intermediate control, and the second control is determined as the path from the first control to the second control;

[0074] When the line of sight is not perpendicular, the intermediate control between the first control and the second control is determined according to the arrangement order of multiple controls in the target screen, and the path between the first control, the intermediate control, and the second control is determined as the path from the first control to the second control.

[0075] Among them, determining the intermediate control between the first control and the second control based on the arrangement order of multiple controls in the target screen can refer to determining the control whose arrangement order is between the sorting order of the first control and the sorting order between the second control as the intermediate control.

[0076] For example, the first control, the second control, and the gaze path can be like this: Figure 5As shown in Figure 501, the first control is control c2, the second control is control c5, and the viewing trajectory is a vertical trajectory. When the viewing trajectory is determined as a path from the first control to the second control, the controls on the path include the first control and the second control. When determining the intermediate control between the first and second controls based on the arrangement order of multiple controls in the target screen, the arrangement order of the controls is control c1-control c2-control c3-control c4-control c5-control c6-control c7-control c8-control c9. The controls between the arrangement order of the first and second controls are control c3 and control c4. Therefore, the intermediate controls between the first and second controls can be determined as control c3 and control c4. At this time, the path from the first control to the second control can be as follows: Figure 5 As shown in Figure 502, the controls on the path include control c2, control c3, control c4 and control c5.

[0077] It should be noted that in this embodiment, the number of rows and columns to be displayed in the target screen can be preset, and multiple controls can be arranged sequentially according to the number of rows and columns in order to obtain... Figure 5 The display style in [the context].

[0078] For example, the first control, the second control, and the gaze trajectory can be like... Figure 6 As shown in Figure 601, the first control is control c1, the second control is control c6, the viewing trajectory is a non-perpendicular trajectory, and the arrangement order of the controls is control c1-control c2-control c3-control c4-control c5-control c6-control c7-control c8-control c9. Based on the arrangement order of multiple controls in the target screen, the intermediate controls between the first and second controls can be determined to be controls c2, c3, c4, and c5. Therefore, the path from the first control to the second control can be as follows: Figure 6 As shown in Figure 602, the controls on the path include control c1, control c2, control c3, control c4, control c5 and control c6.

[0079] In this embodiment, when the gaze trajectory is a vertical trajectory, the gaze trajectory is determined as a path from the first control to the second control, and / or, the intermediate control between the first control and the second control is determined according to the arrangement order of multiple controls in the target screen, and the path between the first control, the intermediate control, and the second control is determined as a path from the first control to the second control; when the gaze trajectory is a non-vertical trajectory, the intermediate control between the first control and the second control is determined according to the arrangement order of multiple controls in the target screen, and the path between the first control, the intermediate control, and the second control is determined as a path from the first control to the second control, which can reduce the player's memory burden, facilitate the player to quickly determine the currently selectable or determined path, and thus facilitate the player to quickly implement the next operation.

[0080] When multiple controls are arranged sequentially in the target screen, the path from the first control to the second control is determined, including:

[0081] Based on the arrangement order of multiple controls in the target screen, determine the intermediate control between the first control and the second control;

[0082] The path between the first control, the middle control, and the second control is defined as the path from the first control to the second control.

[0083] In this embodiment of the application, an intermediate control between the first control and the second control is determined according to the arrangement order of multiple controls in the target screen; the path between the first control, the intermediate control and the second control is determined as the path from the first control to the second control.

[0084] For example, the order of controls c1, c2, c3, c4, c5, c6, c7, c8, and c9 in the target screen is: control c1-control c2-control c3-control c4-control c5-control c6-control c7-control c8-control c9. When the view trajectory is as follows... Figure 5 As shown in Figure 501, the first control is control c2, and the second control is control c5. Based on the arrangement order of multiple controls in the target screen, it is determined that there is no intermediate control between the first and second controls. The path from the first control to the second control can be as follows: Figure 5 As shown in Figure 501, at this point, the path and the line of sight coincide. Alternatively, based on the arrangement order of multiple controls in the target screen, the intermediate controls between the first and second controls are determined to be controls c3 and c4. The path from the first control to the second control can be as follows: Figure 5 As shown in Figure 502, at this point, the path and the line of sight do not overlap.

[0085] In this embodiment, the path from the first control to the second control can be further determined based on the gaze trajectory and the arrangement order of multiple controls in the target screen. That is, determining the path from the first control to the second control includes:

[0086] Obtain the user's gaze trajectory as they move from the first control to the second control;

[0087] Based on the eye movement and the arrangement order of multiple controls in the target screen, determine the path from the first control to the second control.

[0088] In this approach, the terminal can first determine at least two paths from the first control to the second control based on the arrangement order of multiple controls on the target screen, and then filter the path from the first control to the second control from the at least two paths based on the eye trajectory. Alternatively, the terminal can also determine the controls located on the eye trajectory based on the arrangement order of multiple controls on the target screen, and determine the path formed by the controls located on the eye trajectory as the path from the first control to the second control.

[0089] For example, when determining at least two paths from the first control to the second control based on the arrangement order of multiple controls on the target screen, and then filtering the path from the first control to the second control from these at least two paths based on the eye trajectory, the order of controls c1, c2, c3, c4, c5, c6, c7, c8, and c9 on the target screen is: control c1-control c2-control c3-control c4-control c5-control c6-control c7-control c8-control c9. When the eye trajectory is as follows... Figure 5 As shown in Figure 501, the first control is control c2, and the second control is control c5. Based on the arrangement order of the multiple controls in the target screen, the path from the first control to the second control can be determined as follows: Figure 5 The path shown in 501 and as follows Figure 5 The path shown in 502, due to Figure 5 The path shown in section 501 coincides with the line-of-sight trajectory, therefore, Figure 5 The path shown in 501 is determined to be the path from the first control to the second control.

[0090] In other embodiments, determining the path from the first control to the second control includes:

[0091] Obtain the user's third gesture;

[0092] Based on the third gesture, determine the path from the first control to the second control.

[0093] The third gesture can be the user's arm and / or finger gestures.

[0094] In this embodiment of the application, a third operation gesture of the user can be obtained, and a path from the first control to the second control can be determined based on the third operation gesture, thereby realizing the determination of the path from the first control to the second control based on the gesture.

[0095] Wherein, when the third operation gesture is a finger gesture of a user using an electronic device, if the finger gesture is the movement trajectory of the user's target finger, the user's third operation gesture is acquired, including:

[0096] Obtain video of the user's hands;

[0097] The system identifies the user's target finger by performing hand video recognition.

[0098] Determine the movement trajectory of the target finger in the hand video.

[0099] At this point, after obtaining the movement trajectory of the target finger, the terminal first determines the shape of the route corresponding to the movement trajectory, and then uses the route shape as the path from the first control to the second control.

[0100] For example, such as Figure 7 As shown in Figure 701, the first control is control c2, the second control is control c5, and the target finger is the right index finger. The movement trajectory of the right index finger in the hand video is as follows... Figure 7 As shown in Figure 701, if the shape of the path corresponding to the movement trajectory is a straight line, then the path from the first control to the second control is as follows: Figure 7 The straight line shown.

[0101] For example, such as Figure 7 As shown in Figure 702, the first control is control c2, the second control is control c5, and the target finger is the right index finger. The movement trajectory of the right index finger in the hand video is as follows... Figure 7 As shown in Figure 702, the shape of the route corresponding to the movement trajectory is S-shaped (in this case, the S-shape is a positive S-shape). Therefore, the path from the first control to the second control is an S-shaped route. The path from the first control to the second control can be as follows: Figure 7 As shown in Figure 702.

[0102] For example, such as Figure 7 As shown in Figure 703, the first control is control c2, the second control is control c5, and the target finger is the right index finger. The movement trajectory of the right index finger in the hand video is as follows... Figure 7 As shown in Figure 703, the shape of the route corresponding to the movement trajectory is S-shaped (in this case, the S-shape is the reverse S-shape). Therefore, the path from the first control to the second control is an S-shaped route. The path from the first control to the second control can be as follows: Figure 7 As shown in Figure 703.

[0103] It should be understood that when the terminal is a head-mounted display device, the movement trajectory of the target finger can be the trajectory generated after the target finger slides in the air.

[0104] It should be understood that when taking the path shape corresponding to the movement trajectory of the target finger as the path from the first control to the second control, the first control can be taken as the starting point of the path shape and the second control as the ending point of the path shape. Alternatively, the ending point can also be the adjacent control of the first control as the starting point of the path shape and the adjacent control of the second control as the ending point of the path shape.

[0105] For example, for example Figure 7As shown in Figure 702, the first control is control c2, the second control is control c5, the adjacent control of control c2 can be control c3, and the adjacent control of control c5 can be control c4. Therefore, control c2 can be used as the starting point of the route shape, and control c4 can be used as the ending point of the route shape.

[0106] In other embodiments, determining the path from the first control to the second control includes:

[0107] Obtain user selection habits data for controls and determine the positional relationship between the first and second controls;

[0108] Based on selection habits and positional relationships, determine the path from the first control to the second control.

[0109] In this embodiment of the application, the path from the first control to the second control can be determined based on the user's selection habit data and the positional relationship between the first control and the second control.

[0110] The method for determining the path from the first control to the second control based on selection habit data and positional relationships can be set according to the actual situation. For example, a prediction model can be trained based on selection habit data and historical positional relationships (historical positional relationships are the positional relationships between the first and second controls selected in the past) to obtain a trained prediction model. Then, the positional relationships are input into the trained prediction model for prediction processing to obtain the path from the first control to the second control. Alternatively, the selection habit data and positional relationships can also be mapped to obtain the path from the first control to the second control.

[0111] In other embodiments, determining the path from the first control to the second control includes:

[0112] Get the positions of the first control and the second control in the target screen;

[0113] Determine the target positional relationship between the first control and the second control based on their positions.

[0114] Use the preset path corresponding to the target position relationship as the path from the first control to the second control.

[0115] In this embodiment of the application, various positional relationships and various preset paths can be pre-set. Then, after the target positional relationship is obtained, the preset path corresponding to the positional relationship that matches the target positional relationship is used as the preset path corresponding to the target positional relationship. Finally, the preset path corresponding to the target positional relationship is used as the path from the first control to the second control.

[0116] 205. In response to the confirmation operation on the target screen, the controls on the path are selected.

[0117] The response to the confirmation operation on the target screen can be a response to the user's nodding operation, gaze operation, or target operation gesture, or it can be a response to the user's confirmation operation or a response to the user's operation of using the electronic device not being detected within a preset time period.

[0118] For example, in response to a confirmation operation on the target screen, the process of identifying controls on the path as selected controls can be as follows:

[0119] In response to the user's target gesture, the controls on the path are identified as the selected controls.

[0120] The target gesture can refer to the user's arm posture and / or finger posture. The finger posture can be set according to the actual situation. For example, the finger posture can be a pinching posture between different fingers, or the number of fingers pinching, or the pinching between different fingers. This embodiment of the application does not limit this.

[0121] When the finger posture involves pinching together different fingers, for example, it can be like... Figure 8 As shown, the target gesture can be a pinching motion between the right thumb and right index finger. When the terminal detects the pinching motion between the user's right thumb and right index finger, it will identify the control on the path as the selected control.

[0122] Optionally, after confirming the target screen and identifying the controls on the path as selected controls, the controls on the path can be triggered. For example, when all the controls on the path represent game items, the terminal can directly pick up or use the game items represented by the controls on the path.

[0123] For example, when the first control, the second control, and the third control all represent photos, the terminal can delete the photos represented by the controls on the path.

[0124] Optionally, when a control on the path is selected, the selected control can be displayed in a third style. The third style can be the same as or different from the first style. For example, if the path from the first control to the second control is as follows... Figure 5 As shown in 502, the third style can be as follows: Figure 9 As shown.

[0125] In some embodiments, in response to a confirmation operation on the target screen, determining the control on the path as the selected control includes:

[0126] In response to receiving a second gesture from the user, the first control, the middle control, and the second control are identified as the selected controls.

[0127] The second gesture can be set according to the actual situation. For example, the second gesture can be a pinching posture between different fingers, the number of fingers pinching, or the pinching between different fingers. This application embodiment does not limit this.

[0128] When the second gesture is a pinching motion between different fingers, for example, it can be like... Figure 8 As shown, the second gesture can be a pinching motion between the right thumb and right index finger. When the terminal detects the pinching motion between the user's right thumb and right index finger, it determines the first control, the middle control, and the second control as the selected controls.

[0129] Optionally, when the first control, the second control, and the middle control are determined to be selected controls, the middle control can be displayed in a fourth style. When a control is displayed in the fourth style, it indicates that the control is selected.

[0130] The fourth style is a display method that is different from both the first and second styles.

[0131] For example, the first style and the second style are as follows Figure 5 As shown, the first control is control c2, the second control is control c5, and the middle controls are controls c3 and c4. Therefore, the fourth style can be as follows: Figure 10 As shown.

[0132] In some embodiments, when there are two or more paths from the first control to the second control, in response to a confirmation operation on the target screen, the controls on the paths are identified as the selected controls, including:

[0133] In response to receiving the user's first operation gesture, determine the target path corresponding to the first operation gesture from two or more paths;

[0134] Select the controls on the target path as the selected controls.

[0135] The first gesture can be an arm posture or a finger posture. The finger posture can be the movement trajectory of the fingers, the shape formed by different fingers, the pinching of different fingers, or the number of fingers pinching, etc., which are not limited in this embodiment.

[0136] In this embodiment of the application, in response to receiving a user's first operation gesture, a target path corresponding to the first operation gesture is determined from two or more paths; the control on the target path is determined as the selected control, thereby enabling the selection of multiple controls through gestures and gaze operations, and improving the efficiency of control selection.

[0137] For example, when the path from the first control to the second control can be like... Figure 5 The path shown by the line of sight of 501 and Figure 5 When following the path shown in 502, if the first operation gesture is as follows: Figure 7 As shown in 701, then... Figure 5 The path shown by the line of sight of 501 is determined to be the target path.

[0138] As can be seen from the above, in this embodiment, a target screen is displayed, which includes multiple controls; in response to a user's first gaze operation on the target screen, a first control corresponding to the first gaze operation is determined; in response to a user's second gaze operation on the target screen, a second control corresponding to the second gaze operation is determined; a path from the first control to the second control is determined, wherein the path consists of two or more controls; in response to a confirmation operation on the target screen, the controls on the path are determined as the selected controls; this solution performs simple tracking of the player's gaze, thereby determining the path through the player's gaze, and then selecting multiple controls at once through the path, without requiring the user to gaze at all the controls that need to be selected, thus improving the efficiency of control selection.

[0139] The following is based on Figure 11 This application further explains the control selection method provided in this application. In this embodiment, the head-mounted display device will be used as an example for explanation. In this case, the user using the electronic device can be the user wearing the head-mounted display device. Figure 11 As shown, the specific process of this control selection method can be as follows:

[0140] S1101, The head-mounted display device displays the target screen, which includes multiple controls.

[0141] S1102. In response to a user wearing the head-mounted display device's first gaze operation on a target screen, the head-mounted display device determines a first control corresponding to the first gaze operation and displays the first control corresponding to the first gaze operation in a first style.

[0142] S1103. The head-mounted display device responds to the user's second gaze operation on the target screen within a preset time after the first gaze operation, determines the second control corresponding to the second gaze operation, and displays the second control corresponding to the second gaze operation in a second style.

[0143] S1104. The head-mounted display device determines a path from the first control to the second control, wherein the path consists of two or more controls.

[0144] S1105. In response to the confirmation operation on the target screen, the head-mounted display device identifies the controls on the path as the selected controls.

[0145] The method for determining the path from the first control to the second control in this embodiment can be referred to the above embodiment of the control selection method, and will not be repeated here.

[0146] The structure of the head-mounted display device can Figure 12 As shown. The head-mounted display device 1200 may include a device body 1201, a processor 1202 with one or more processing cores disposed on the device body 1201, one or more memory 1203, one or more display generation units 1204, one or more eye tracking units 1205 and one or more hand tracking units 1206, and a fixing strap 1207 connected to the device body, etc.

[0147] like Figure 12 As shown, the processor 1202 can be the control center of the head-mounted display device, connecting various parts of the head-mounted display device 1200 through various interfaces and lines. It executes various functions and processes data of the head-mounted display device 1200 by running or loading software programs and / or modules stored in the memory 1203, and by calling data stored in the memory 1203, thereby providing overall monitoring of the head-mounted display device 1200. The processor can be a central processing unit (CPU), graphics processing unit (GPU), network processor (NP), etc., and can execute the methods, steps, and logic diagrams disclosed in the embodiments of this application.

[0148] The display generation unit 1204 may include one or more display screens. The display generation unit is used to display the images output by the head-mounted display device. In one embodiment, the display screen may be built into the device body. The display generation unit can receive video data and display virtual scenes on the display screen according to the video data. In another embodiment, the head-mounted display device 1200 may include a display screen port. The display screen may be provided by a mobile terminal (e.g., a smartphone and a tablet computer) that is plugged into the display screen port.

[0149] The eye-tracking unit 1205 may include one or more cameras, which are located inside the head-mounted display device to capture the user's eye images when the user wears the head-mounted display device, and obtain the user's gaze position on the display screen and the user's eye movements based on the eye images. The eye movements include, but are not limited to, blinking, opening the eyes, and rotating the eyeballs.

[0150] The hand tracking unit 1206 may include one or more cameras located on the outside of the head-mounted display device to capture images of the user's hands when the user wears the head-mounted display device and determine the user's gestures based on the hand images.

[0151] The securing strap 1207 can be used to secure the head-mounted display device to the user's head.

[0152] In one embodiment, the head-mounted display device 1200 may further include stereoscopic lenses for displaying the content shown on the screen in a stereoscopic form. These lenses may be selected from at least one of aspherical lenses, ultra-tough lenses, photochromic lenses, tinted lenses, progressive multifocal lenses, anti-radiation lenses, and resin lenses. For example, when aspherical lenses are used, the dual-screen display content is viewed through two aspherical lenses, and the different angles at which the objects are viewed by each eye create a sense of depth, thus producing a stereoscopic effect.

[0153] In one embodiment, the head-mounted display device 1200 may further include bone conduction headphones, which may be located on the device body 1201 near the ear, and can output audio to the user.

[0154] The specific implementation process and corresponding beneficial effects of this application embodiment can be referred to the above-described control selection method embodiment, which will not be repeated here.

[0155] To better implement the above methods, this application also provides a control selection device, which can be integrated into an electronic device, such as a terminal or server.

[0156] The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0157] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the control selection device specifically integrated into the terminal as an example. This embodiment provides a control selection device, such as... Figure 13 As shown, the control selection device may include:

[0158] The screen display module 1301 is used to display the target screen, which includes multiple controls.

[0159] The first response module 1302 is used to respond to the user's first gaze operation on the target screen and determine the first control corresponding to the first gaze operation.

[0160] The second response module 1303 is used to respond to the user's second gaze operation on the target screen and determine the second control corresponding to the second gaze operation.

[0161] The control determination module 1304 is used to determine the path from the first control to the second control, wherein the path consists of two or more controls.

[0162] The control selection module 1305 is used to select controls on the path in response to a confirmation operation on the target screen.

[0163] In some embodiments, the second response module 1303 is specifically used to perform:

[0164] In response to a second gaze operation by the user on the target screen within a preset time after the first gaze operation, the second control corresponding to the second gaze operation is determined.

[0165] In some embodiments, the control determination module 1304 is specifically used to perform:

[0166] Obtain the user's gaze trajectory as they move from the first control to the second control;

[0167] Determine the path from the first control to the second control based on the line of sight.

[0168] In some embodiments, multiple controls are arranged sequentially on the target screen, and the control determination module 1304 is specifically used to perform:

[0169] When the gaze path is a vertical path, the gaze path is determined as the path from the first control to the second control, and / or, the intermediate control between the first control and the second control is determined according to the arrangement order of multiple controls in the target screen, and the path between the first control, the intermediate control, and the second control is determined as the path from the first control to the second control;

[0170] When the line of sight is not perpendicular, the intermediate control between the first control and the second control is determined according to the arrangement order of multiple controls in the target screen, and the path between the first control, the intermediate control, and the second control is determined as the path from the first control to the second control.

[0171] In some embodiments, when there are two or more paths from the first control to the second control, the control selection module 1305 is specifically used to perform:

[0172] In response to receiving the user's first operation gesture, determine the target path corresponding to the first operation gesture from two or more paths;

[0173] Select the controls on the target path as the selected controls.

[0174] In some embodiments, multiple controls are arranged sequentially on the target screen, and the control determination module 1304 is specifically used to perform:

[0175] Based on the arrangement order of multiple controls in the target screen, determine the intermediate control between the first control and the second control;

[0176] The path between the first control, the middle control, and the second control is defined as the path from the first control to the second control.

[0177] In some embodiments, the control selection module 1305 is specifically used to perform:

[0178] In response to receiving a second gesture from the user, the first control, the middle control, and the second control are identified as the selected controls.

[0179] In some embodiments, the first response module 1302 is further configured to perform:

[0180] Display the first control in the first style.

[0181] In some embodiments, the second response module 1303 is further configured to perform:

[0182] The second control is displayed in the second style.

[0183] In some embodiments, the control selection module 1305 is also used to perform:

[0184] The selected control is displayed in the third style.

[0185] In some embodiments, the control selection module 1305 is specifically used to perform:

[0186] In response to the user's target gesture, the controls on the path are identified as the selected controls.

[0187] In some embodiments, the control determination module 1304 is specifically used to perform:

[0188] Obtain the user's third gesture;

[0189] Based on the third gesture, determine the path from the first control to the second control.

[0190] In some embodiments, the control determination module 1304 is specifically used to perform:

[0191] Obtain user selection habits data for controls and determine the positional relationship between the first and second controls;

[0192] Based on selection habits and positional relationships, determine the path from the first control to the second control.

[0193] In some embodiments, the control determination module 1304 is specifically used to perform:

[0194] Get the positions of the first control and the second control in the target screen;

[0195] Determine the target positional relationship between the first control and the second control based on their positions.

[0196] Use the preset path corresponding to the target position relationship as the path from the first control to the second control.

[0197] In practice, each of the above modules can be implemented as an independent entity or can be combined arbitrarily to be implemented as the same or several entities. For the specific implementation methods and corresponding beneficial effects of each of the above modules, please refer to the previous method embodiments, which will not be repeated here.

[0198] Accordingly, embodiments of this application also provide an electronic device, which can be a terminal, such as a smartphone, tablet computer, laptop computer, touch screen, game console, personal computer (PC), personal digital assistant (PDA), or other terminal device. Figure 14 As shown, Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 1400 includes a processor 1401 with one or more processing cores, a memory 1402 with one or more computer-readable storage media, and a computer program stored on the memory 1402 and executable on the processor. The processor 1401 and the memory 1402 are electrically connected. Those skilled in the art will understand that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0199] The processor 1401 is the control center of the electronic device 1400. It connects various parts of the electronic device 1400 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 1402, and calling data stored in the memory 1402, it performs various functions of the electronic device 1400 and processes data, thereby monitoring the electronic device 1400 as a whole.

[0200] In this embodiment, the processor 1401 in the electronic device 1400 loads the instructions corresponding to the processes of one or more applications into the memory 1402 according to the following steps, and the processor 1401 runs the applications stored in the memory 1402 to realize various functions, such as:

[0201] Display the target screen, which includes multiple controls;

[0202] In response to the user's first gaze action on the target screen, determine the first control corresponding to the first gaze action;

[0203] In response to the user's second gaze operation on the target screen, determine the second control corresponding to the second gaze operation;

[0204] Determine the path from the first control to the second control, where the path consists of two or more controls;

[0205] In response to the confirmation operation on the target screen, the controls on the path are selected.

[0206] For details on the specific implementation methods and corresponding beneficial effects of the above operations, please refer to the detailed description of the control selection method above, which will not be repeated here.

[0207] Optionally, such as Figure 14 As shown, the electronic device 1400 also includes: a touch display screen 1403, a radio frequency circuit 1404, an audio circuit 1405, an input unit 1406, and a power supply 1407. The processor 1401 is electrically connected to the touch display screen 1403, the radio frequency circuit 1404, the audio circuit 1405, the input unit 1406, and the power supply 1407. Those skilled in the art will understand that... Figure 14 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0208] The touch display screen 1403 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 1403 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar technologies. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program according to the operation commands. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 1401. It can also receive and execute commands from the processor 1401. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 1401 to determine the type of touch event. Subsequently, the processor 1401 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the touch display screen 1403 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 1403 can be implemented as two independent components to achieve input and output functions. That is, the touch display screen 1403 can also be used as part of the input unit 1406 to achieve input functions.

[0209] The radio frequency circuit 1404 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other electronic devices, and to transmit and receive signals with network devices or other electronic devices.

[0210] Audio circuit 1405 can be used to provide an audio interface between a user and an electronic device via a speaker and a microphone. Audio circuit 1405 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuit 1405, converted back into audio data, and then processed by processor 1401 before being transmitted via radio frequency circuit 1404 to, for example, another electronic device, or output to memory 1402 for further processing. Audio circuit 1405 may also include an earphone jack to provide communication between peripheral headphones and electronic devices.

[0211] The input unit 1406 can be used to receive input numbers, character information or user feature information (such as fingerprints, iris, facial information, etc.), and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0212] Power supply 1407 is used to supply power to various components of electronic device 1400. Optionally, power supply 1407 can be logically connected to processor 1401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 1407 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0213] although Figure 14 As not shown in the diagram, the electronic device 1400 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0214] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0215] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0216] Therefore, embodiments of this application provide a computer-readable storage medium storing multiple computer programs that can be loaded by a processor to execute any of the control selection methods provided in embodiments of this application. For example, the computer program can perform the following steps:

[0217] Display the target screen, which includes multiple controls;

[0218] In response to the user's first gaze action on the target screen, determine the first control corresponding to the first gaze action;

[0219] In response to the user's second gaze operation on the target screen, determine the second control corresponding to the second gaze operation;

[0220] Determine the path from the first control to the second control, where the path consists of two or more controls;

[0221] In response to the confirmation operation on the target screen, the controls on the path are selected.

[0222] For details on the specific implementation methods and corresponding beneficial effects of the above operations, please refer to the detailed description of the control selection method above, which will not be repeated here.

[0223] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0224] Since the computer program stored in the computer-readable storage medium can execute any of the control selection methods provided in the embodiments of this application, it can achieve the beneficial effects that any of the control selection methods provided in the embodiments of this application can achieve, as detailed in the preceding embodiments, and will not be repeated here.

[0225] The foregoing has provided a detailed description of a control selection method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for selecting a control, characterized in that, The method includes: Display the target screen, which includes multiple controls; In response to a user’s first gaze operation on the target screen, a first control corresponding to the first gaze operation is determined, and the first control is displayed in a first style; In response to a second gaze operation by the user on the target screen within a preset time period after the first gaze operation, a second control corresponding to the second gaze operation is determined, and the second control is displayed in a second style; Determine a path from the first control to the second control, wherein the path consists of two or more controls; In response to the user's target gesture, the control on the path is identified as the selected control, wherein the selected control is displayed in a third style.

2. The control selection method as described in claim 1, characterized in that, Determining the path from the first control to the second control includes: Obtain the user's gaze trajectory as they move from the first control to the second control; Based on the gaze trajectory, determine the path from the first control to the second control.

3. The method according to claim 2, characterized in that, The multiple controls are arranged in sequence on the target screen; Determining the path from the first control to the second control based on the gaze trajectory includes: When the gaze trajectory is a vertical trajectory, the gaze trajectory is determined as a path from the first control to the second control, and / or, the intermediate control between the first control and the second control is determined according to the arrangement order of the multiple controls in the target screen, and the path between the first control, the intermediate control, and the second control is determined as a path from the first control to the second control; When the gaze trajectory is a non-vertical trajectory, the intermediate control between the first control and the second control is determined according to the arrangement order of the multiple controls in the target screen, and the path between the first control, the intermediate control, and the second control is determined as the path from the first control to the second control.

4. The method according to claim 3, characterized in that, When there are two or more paths from the first control to the second control, the step of determining the controls on the paths as the selected controls in response to the user's target operation gesture includes: In response to receiving a user's first operation gesture, a target path corresponding to the first operation gesture is determined from the two or more paths; The controls on the target path are selected.

5. The control selection method according to claim 1, characterized in that, The multiple controls are arranged in sequence on the target screen; Determining the path from the first control to the second control includes: Based on the arrangement order of the multiple controls in the target screen, determine the intermediate control between the first control and the second control; The path between the first control, the intermediate control, and the second control is determined as the path from the first control to the second control.

6. The control selection method according to claim 5, characterized in that, The step of determining the control on the path as the selected control in response to the user's target operation gesture includes: In response to receiving the user's second gesture, the first control, the intermediate control, and the second control are identified as the selected controls.

7. The control selection method as described in claim 1, characterized in that, Determining the path from the first control to the second control includes: Obtain the user's third gesture; Based on the third gesture, the path from the first control to the second control is determined.

8. The control selection method as described in claim 1, characterized in that, Determining the path from the first control to the second control includes: Obtain the user's selection habits data for controls and determine the positional relationship between the first control and the second control; Based on the selection habit data and the positional relationship, determine the path from the first control to the second control.

9. The control selection method as described in claim 1, characterized in that, Determining the path from the first control to the second control includes: Obtain the position of the first control and the position of the second control in the target screen; Determine the target positional relationship between the first control and the second control based on their positions; The preset path corresponding to the target position relationship is used as the path from the first control to the second control.

10. A control selection device, characterized in that, The device includes: The screen display module is used to display the target screen, which includes multiple controls; The first response module is used to respond to the user's first gaze operation on the target screen, determine the first control corresponding to the first gaze operation, and display the first control in a first style; The second response module is used to respond to the user's second gaze operation on the target screen within a preset time after the first gaze operation, determine the second control corresponding to the second gaze operation, and display the second control in a second style. A control determination module is used to determine a path from the first control to the second control, wherein the path consists of two or more controls; The control selection module is used to determine the controls on the path as selected controls in response to the user's target operation gesture, wherein the selected controls are displayed in a third style.

11. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to execute the control selection method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the control selection method according to any one of claims 1 to 9.

Images