Menu triggering method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a menu triggering method and device, equipment, a storage medium and a program product, and the method comprises the steps: responding to a menu triggering gesture presented by the hand of a user, and detecting a movement track of the hand of the user when the pose of the head relative to the hand meets a preset triggering condition; and displaying the menu panel to the user in response to the condition that the moving track meets a preset condition. According to the embodiment of the invention, head-hand cooperation can be used as a triggering condition, false triggering is effectively reduced, the triggering accuracy is improved, the triggering process accords with instinctive intuition of a user, the user does not need to change gestures in the whole process, the burden of memorizing combined gestures by the user is reduced, and the user experience is effectively improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular, to a menu triggering method, apparatus, device, storage medium, and program product. Background Art

[0002] With the continuous development of computer technology, wearable display devices such as VR (Virtual Reality) devices have been more and more widely used. Wearable display devices are usually worn on the user's head and can provide the user with an immersive experience environment.

[0003] In order to improve human-computer interaction capabilities, wearable display devices provide ways to interact through peripherals (such as handles, etc.) or bare-handed interaction, allowing users to control the display screen, for example, triggering menu display, etc.

[0004] Currently, when triggering a menu through bare-hand interaction, there are problems of poor triggering accuracy and easy mis-triggering, resulting in a poor user experience. Summary of the invention

[0005] Embodiments of the present disclosure provide a menu triggering method, apparatus, device, storage medium, and program product to reduce the problem of menu mis-triggering during freehand interaction.

[0006] In a first aspect, an embodiment of the present disclosure provides a menu triggering method, including:

[0007] In response to the user's hand presenting a menu triggering gesture, and the posture of the head relative to the hand meeting a preset triggering condition, detecting the movement trajectory of the user's hand;

[0008] In response to the movement trajectory satisfying a preset condition, a menu panel is displayed to the user.

[0009] In a second aspect, an embodiment of the present disclosure provides a menu triggering device, including:

[0010] A detection module, configured to detect a movement trajectory of the user's hand in response to the user's hand presenting a menu triggering gesture and the position of the head relative to the hand meeting a preset triggering condition;

[0011] The display module is used to display the menu panel to the user in response to the movement trajectory meeting the preset condition.

[0012] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a memory and at least one processor;

[0013] The memory stores computer-executable instructions;

[0014] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the menu triggering method described in the first aspect above.

[0015] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the menu triggering method described in the first aspect above is implemented.

[0016] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the menu triggering method described in the first aspect above.

[0017] The menu triggering method, apparatus, device, storage medium and program product provided by the embodiments of the present disclosure can respond to the user's hand presenting a menu triggering gesture, and the position of the head relative to the hand satisfies the preset triggering conditions, detect the movement trajectory of the user's hand, and display the menu panel to the user in response to the movement trajectory satisfying the preset conditions, thereby being able to use the coordination of the head and hands as a triggering condition, effectively reducing false triggers and improving the accuracy of the triggering, and the triggering process is in line with the user's instinctive intuition. The user does not need to change gestures during the entire process, reducing the burden of the user remembering combined gestures, and effectively improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0020] Figure 2 A schematic diagram of the principle of a menu triggering method provided by an embodiment of the present disclosure;

[0021] Figure 3 A schematic diagram of a tap gesture provided in an embodiment of the present disclosure;

[0022] Figure 4A A schematic diagram of a user looking at his hands with his head provided in an embodiment of the present disclosure;

[0023] Figure 4B A schematic diagram of a user's head not looking at his hands provided in an embodiment of the present disclosure;

[0024] Figure 5A schematic diagram of a fingertip visual feedback provided by an embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram of a menu panel in an intermediate state provided by an embodiment of the present disclosure;

[0026] Figure 7 A schematic diagram of a menu panel in a final state provided by an embodiment of the present disclosure;

[0027] Figure 8 A schematic diagram of operating a menu panel provided by an embodiment of the present disclosure;

[0028] Fig. 9 A schematic diagram of a trajectory of a menu panel that fails to be normally awakened provided by an embodiment of the present disclosure;

[0029] Fig.10 A schematic diagram of closing a menu panel provided in an embodiment of the present disclosure;

[0030] Fig.11 A flowchart of a menu triggering method provided by an embodiment of the present disclosure;

[0031] Fig.12 A schematic diagram of a process for providing visual feedback provided in an embodiment of the present disclosure;

[0032] Fig.13 A flowchart of another menu triggering method provided by an embodiment of the present disclosure;

[0033] Fig.14 A structural block diagram of a menu triggering device provided in an embodiment of the present disclosure;

[0034] Fig.15 A structural block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0036] The embodiments of the present disclosure can be applied to any electronic device that can trigger menu display through bare-hand interaction, and in particular can be applied to wearable display devices, such as VR devices, AR (Augmented Reality) devices, MR (Mixed Reality) devices, XR (Extended Reality) devices, etc.

[0037] The wearable display device may be a helmet or glasses, etc., worn on the user's head. The wearable display device may be provided with one or more shooting devices, which can shoot the environment picture and display the corresponding virtual picture or virtual-real fusion picture to the user according to the environment picture.

[0038] On this basis, users can control the device through peripherals or bare-handed interaction.

[0039] In scenarios with peripherals, users can move the handle or press buttons when using the device. The device can detect the position of the handle or the pressed button, thereby triggering corresponding functions such as screenshots, screen recording, global navigation (Home), calibration, etc.

[0040] In the scenario of bare-hand interaction, there is no need for peripherals. The user can interact with one or both hands and realize the relevant functions of the peripherals.

[0041] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure. Figure 1 As shown, in a bare-hand interaction scenario, when a user wears the wearable display device 101, the wearable display device 101 can capture the user's hands and determine whether certain functions are currently triggered, such as whether to trigger the display of a menu panel, by analyzing the user's hands.

[0042] In the bare-hand interaction scenario, since the peripherals are eliminated, how to improve the trigger accuracy and prevent false triggering as much as possible becomes a problem that needs to be solved.

[0043] In some technologies, in order to improve the accuracy of triggering, complex gestures can be designed to implement the triggering operation of the menu, for example, by making certain gestures with both hands to trigger the menu, or by making multiple static gestures or dynamically changing gestures with one hand in sequence to trigger the menu. These methods require users to accurately remember various gestures, and the user experience is poor.

[0044] In other technologies, icons can be used to assist in triggering menus. For example, when a user makes certain gestures, an icon can be displayed to the user. When the user looks at the icon and makes another gesture, the menu can be displayed. This method still requires the user to remember multiple gestures and requires icons as an aid. It is complex, inefficient, and has a poor user experience.

[0045] In view of this, the embodiments of the present disclosure provide a menu triggering method that is in line with human factors, which can accurately judge the user's intention on the basis of reducing the user's burden, prevent false triggering as much as possible, and improve triggering accuracy.

[0046] Figure 2 The schematic diagram of the principle of a menu triggering method provided by the embodiment of the present disclosure is as follows. Figure 2 As shown, in the embodiment of the present disclosure, the menu triggering function can be implemented through the following steps.

[0047] Step a: monitor two types of input data: head data or eye movement data, and hand data, and determine the user's intention based on the two types of input data.

[0048] Optionally, when it is determined based on the head data or the eye movement data that the user is looking at the hand, and it is determined based on the hand data that the user's hand is presenting a tap gesture, it can be considered that the user has the intention to trigger the menu. Figure 3 A schematic diagram of a tap gesture provided in an embodiment of the present disclosure. Figure 4A A schematic diagram of a user looking at his hands from his head provided in an embodiment of the present disclosure. Figure 4B A schematic diagram of a user not looking at his hands provided in an embodiment of the present disclosure. The arrows in the figure represent the direction of the user's head or the direction of his eyes.

[0049] Step b: when it is determined that the user intends to trigger the menu, visual feedback is given to the fingertip, that is, the movement of the fingertip causes visual trailing.

[0050] Figure 5 A schematic diagram of a fingertip visual feedback provided by an embodiment of the present disclosure. Figure 5 As shown, visual feedback can be given to the user in the form of fingertip dragging, so that the user can understand his or her actions.

[0051] Step c: Calculate the distance between the current position of the fingertip and the initial position. When the distance gradually decreases, the menu panel grows from small to large, accompanied by the adjustment of opacity.

[0052] Step d: When the fingertips close the circle, the menu panel changes to normal size and the opacity is 100%.

[0053] Figure 6A schematic diagram of a menu panel in an intermediate state provided by an embodiment of the present disclosure. Figure 7 A schematic diagram of a menu panel in the final state provided by an embodiment of the present disclosure. Figure 6 and Figure 7 As shown, after determining that the user has the intention to trigger the menu, starting from the current initial position, a visual shadow of the fingertip is provided. During the movement of the user's finger, the fingertip will first gradually move away from the initial position, and after reaching the critical point, it will gradually approach the initial position. After the critical point, that is, when the distance between the fingertip and the initial position gradually decreases, the menu panel can be displayed. At this time, the menu panel size is small and the opacity is low. As the fingertip approaches the initial position, the size and opacity of the menu panel gradually increase. When the fingertip returns to the initial position, that is, when the movement trajectory of the fingertip presents a closed loop, the menu panel is adjusted to a normal size and normal opacity. Visual feedback can enable users to understand the significance of their behavior and enhance user experience.

[0054] Step e. Use the tap gesture to click on the menu panel.

[0055] Figure 8 A schematic diagram of operating a menu panel provided by an embodiment of the present disclosure. Figure 8 As shown, by clicking, one or more function options can be selected in the menu panel.

[0056] In addition, the disclosed embodiment also provides a method for canceling the display of a menu panel.

[0057] Step f: when it is detected that the fingertip deviates while gradually approaching the initial position, the visual feedback is canceled and the menu panel is stopped from being called up.

[0058] Optionally, this step can be performed after step c. Fig. 9 A schematic diagram of a trajectory of a menu panel that fails to be normally awakened is provided in an embodiment of the present disclosure. Fig. 9 As shown, the menu panel will be displayed normally only after the movement trajectory of the fingertip forms a closed loop. If deviation occurs during the movement, for example, after exceeding the critical point, it starts to move away from the initial position at a certain position, it is considered that the user does not really want to call up the menu panel. At this time, the visual feedback can be stopped, that is, the fingertip shadow disappears, and the display of the menu panel is cancelled.

[0059] Step g: when it is detected that the finger tip is moving away from the menu panel quickly, it is determined that the user's current intention is to close the menu panel.

[0060] Optionally, this step can be performed after step e. Fig.10 A schematic diagram of closing a menu panel provided in an embodiment of the present disclosure. Fig.10As shown, after the menu panel is displayed normally, if the finger tip gradually moves away from the menu panel at a fast speed, the menu panel can be closed.

[0061] In summary, the menu triggering method provided by the embodiment of the present disclosure can use the head looking at the hand and the hand in the tap gesture as the triggering condition, which effectively reduces false triggers and conforms to the human body's active triggering habits, thereby improving the accuracy of the triggering; there is no need to display the icon first and implement the menu triggering based on the interaction between the user and the icon, which effectively improves the efficiency and stability of the triggering; and, during the entire triggering process, the user only needs to use one hand shape, that is, the tap hand shape used when finally operating the menu panel, and the user does not need to remember complex combination gestures. The triggering process conforms to the user's instinctive intuition, effectively reducing the user's burden; in addition, visual feedback matching the triggering progress can also be provided at the appropriate time, which is convenient for the user to understand the current situation and make real-time adjustments according to their own intentions, thereby further improving the user experience.

[0062] In the following, some embodiments of the present disclosure are described in detail in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and features in the embodiments can be combined with each other.

[0063] Fig.11 The present invention provides a flowchart of a menu triggering method according to an embodiment of the present invention. The execution subject of the method may be an electronic device, in particular, a wearable display device. Fig.11 As shown, the menu triggering method may include:

[0064] Step 1101: In response to the user's hand presenting a menu trigger gesture, and the position of the head relative to the hand meeting a preset trigger condition, detecting the movement trajectory of the user's hand.

[0065] Optionally, the menu triggering gesture may be a preset gesture, such as a tap gesture: extending the index finger and holding the other four fingers. Alternatively, it may be other gestures, such as extending other fingers except the index finger, or extending multiple fingers.

[0066] In practical applications, the key points of the user's hand can be detected through the image captured by the device, and the user's current gesture can be determined based on the position of the key points. Alternatively, the user's hand image can be input into a gesture recognition model to directly determine the user's current gesture.

[0067] In addition to the user's gestures, this step can also detect the posture of the user's head, where the head can refer to the entire head of the user or a certain position of the user's head, such as the eyes. The posture can include position and / or posture, and the posture can include a tilt angle, such as a pitch angle, etc., which can also be understood as a direction.

[0068] The trigger condition can be set according to actual needs. Optionally, the trigger condition is used to indicate that the head is looking at the hand. Accordingly, the position of the head relative to the hand meets the preset requirements, which may include: the head is facing the hand, that is, the line between the head and the hand just matches the direction of the head.

[0069] Alternatively, the position of the head relative to the hand meets preset requirements, which may include: the eyes are looking at the hand, that is, the position of the eye (more specifically, the position of the pupil) relative to the position of the hand meets certain requirements.

[0070] Various types of sensor devices may be provided in the wearable display device, for example, a camera, an IMU (Inertial Measurement Unit), an eye movement detection device, etc. The specific form of the trigger condition may be set according to the information collected by the sensor device.

[0071] In one example, the wearable display device can determine the orientation of the user's head through the IMU, and determine whether the condition of the head facing the hand is met based on the orientation of the user's head.

[0072] In another example, the wearable display device can obtain the user's eye movement data through an eye movement detection device, such as an eye tracker, and the eye movement data can be used to provide information such as the user's pupil position or line of sight direction, and based on this information, it can be determined whether the user is looking at his hands.

[0073] In another example, the wearable display device can capture images or videos of the surrounding environment through one or more shooting devices such as cameras. Since the user's head and the wearable display device are usually in a tightly fixed relationship, the position of the shooting device relative to the hand can be used to reflect the position of the user's head relative to the hand, and the position of the shooting device relative to the hand will affect the captured picture. Therefore, the position of the head relative to the hand can also be deduced from the captured picture. For example, when the head is facing the hand, the hand is usually in the middle of the picture.

[0074] In a wearable display device, one or more sensor devices may be provided, and the information collected by the sensor devices may be used to directly or indirectly determine whether the position of the head relative to the hand meets the trigger condition. The specific determination method is not limited here.

[0075] Step 1102: In response to the movement trajectory satisfying a preset condition, display a menu panel to the user.

[0076] If it is detected that the user's hand is presenting a menu trigger gesture, and the head posture relative to the hand meets the preset trigger conditions, it can be considered that the user has the intention to trigger the menu. It should be noted that the user's intention to trigger the menu mentioned here can be considered as a start signal for the trigger, and does not mean that the menu panel must be displayed to the user immediately as long as the intention is detected. Specifically, after determining that the user has the intention to trigger the menu, it can be detected whether the movement trajectory of the hand meets the preset conditions. If so, the menu panel can be displayed to the user to reduce false triggering.

[0077] Optionally, in response to the movement trajectory satisfying a preset condition, displaying the menu panel to the user may include: in response to the user's hand moving under the menu trigger gesture and the movement trajectory satisfying the preset condition, displaying the menu panel to the user.

[0078] Exemplarily, the menu trigger gesture is a tap gesture. When the user looks at his hand with the hand in the tap gesture, the device determines that the user has the intention to trigger the menu. The user can further move the hand while maintaining the tap gesture. When the movement trajectory meets the preset conditions, the corresponding menu can be displayed to the user.

[0079] Optionally, the preset condition may include: the movement trajectory is a closed shape.

[0080] In one example, the movement trajectory may be any closed shape. Optionally, the closed shape may be a regular closed shape, such as a circle, a square, a triangle, etc., or an irregular shape, as long as the user's hand can return to the initial position to form a closed loop.

[0081] In another example, the movement track may be a specific closed shape. For example, the specific closed shape is a circle. When the user draws a circle, the menu panel may be displayed, and when the user draws a square, the menu panel may not be displayed. The specific closed shape may be preset or selected by the user.

[0082] In another example, the movement track can be a closed shape, and the size of the closed shape is within a preset size range. The size of the closed shape can be expressed in a variety of forms, such as diameter, side length, perimeter, area, etc. The preset size range can be pre-set, for example, the diameter is greater than a first threshold and less than a second threshold, so that the menu display is triggered only when the user moves within a reasonable range, preventing false triggering when the user moves too much or too little.

[0083] The above scheme detects whether the movement trajectory of the hand is in a closed shape, which is more in line with the user's instinctive action of actually wanting to call up the menu. In addition, by limiting the maintenance of the menu trigger gesture during the movement, the constraint of the trigger can be further increased. However, the number of gestures that the user needs to remember will not increase. It can reduce false triggers and improve trigger accuracy without adding too much burden to the user, thus taking into account both efficiency and accuracy.

[0084] In other optional implementation schemes, there is no restriction on maintaining the menu trigger gesture during the movement process, which can reduce the amount of calculation and improve efficiency. Alternatively, the movement trajectory can also be a shape other than a closed shape, for example, the movement trajectory is in the shape of a check mark, or the user can set the trigger shape to suit their own needs.

[0085] In summary, the menu triggering method provided in the embodiments of the present invention responds to the user's hand presenting a menu trigger gesture and the position of the head relative to the hand satisfying a preset trigger condition, detects the movement trajectory of the user's hand, and displays the menu panel to the user in response to the movement trajectory satisfying the preset condition, thereby being able to use the coordination of the head and the hand as a trigger condition, effectively reducing false triggers, and improving the accuracy of the trigger. The triggering process is in line with the user's instinctive intuition, and the user does not need to change gestures during the entire process, reducing the burden of the user remembering combined gestures, and effectively improving the user experience.

[0086] In one or more embodiments of the present disclosure, optionally, after determining that the user has the intention to trigger the menu, visual feedback may be provided to the user, wherein the visual feedback may include: visual feedback corresponding to the input, and / or visual feedback corresponding to the output.

[0087] Specifically, the visual feedback corresponding to the input may be the user's input signal, that is, the visual feedback of the hand movement. The visual feedback corresponding to the output may be the visual feedback corresponding to the signal to be expressed to the user, for example, it can be used to indicate the extent of the operation of calling up the menu panel. The visual feedback corresponding to the input and the visual feedback corresponding to the output may be used separately or in combination. The following is a detailed description using the combination of the two as an example.

[0088] Optionally, in response to the movement trajectory satisfying a preset condition, displaying a menu panel to the user may include: during the movement of the user's hand, providing corresponding visual feedback according to the progress of the movement, until the preset condition is met, and then displaying the menu panel to the user.

[0089] The movement completion progress can be determined by referring to the preset condition. During the hand movement, the movement completion progress gradually increases. When the preset condition is reached, the movement completion progress is 100%. The visual feedback corresponding to the input and the visual feedback corresponding to the output can match the current movement completion progress.

[0090] In one or more embodiments of the present disclosure, it is optional to provide corresponding visual feedback according to the progress of the movement completion during the movement of the user's hand, including: during the movement of the user's hand, displaying a shadow or movement trajectory corresponding to a preset part of the user's hand, and displaying a gradient menu panel according to the progress of the movement completion, until the preset conditions are met, and then displaying the menu panel in the final state.

[0091] Optionally, the preset part can be determined by a menu trigger gesture, and may be one or more parts that can be clearly detected in the menu trigger gesture, and the part displays a trail or moving trajectory that conforms to the user's subjective habits. For example, when the menu trigger gesture is a tap gesture, the preset part may be the tip of the index finger, and the displayed visual feedback effect may allow the user to feel the effect of fingertip writing.

[0092] Among them, both displaying the movement trajectory and displaying the afterimage can let users understand their own behavior, but the two are slightly different. The movement trajectory includes the overall movement trajectory of the hand after determining that the user has the intention to trigger the menu, and the afterimage can be the movement trajectory within the current period of time. For example, as the hand moves, the previous trajectory can gradually disappear, and only the movement trajectory near the current position of the fingertip is retained.

[0093] In addition, a gradient menu panel can be displayed, which is equivalent to outputting corresponding visual feedback. The change of the menu panel can be related to the current progress of the movement. When the movement trajectory meets the preset conditions, the menu panel of the final state is displayed.

[0094] Through the above method, visual feedback that matches the current movement progress can be provided to the user, which is convenient for the user to understand the current situation and further improve the user experience. In addition, in the visual feedback provided, the trailing shadow of the fingertip is used to show the user the changes in the user's own actions, which is the visual feedback corresponding to the user input; the menu panel is the output object that is ultimately provided to the user, and the gradient menu panel reflects the changes in the output object, which is the visual feedback corresponding to the output. Therefore, the above method provides two types of visual feedback corresponding to input and output respectively. The trailing shadow allows the user to understand the movement trajectory of his own finger, and the gradient menu allows the user to see that the menu is currently changing towards the final state he expects. The combination of the two allows the user to have a dynamic and intuitive understanding of the input and output conditions at the same time, thereby improving the user experience.

[0095] Fig.12 A flow chart of providing visual feedback is provided in an embodiment of the present disclosure. Fig.12 As shown, according to the progress of the movement, a gradient menu panel is displayed, which may include:

[0096] Step 1201: In response to the movement completion progress satisfying the progress condition, display the menu panel in the initial state.

[0097] The attribute information of the menu panel in the initial state is different from the attribute information of the final state. The attribute information includes: size, and / or opacity. For example, the size of the menu panel in the final state is 10, and the opacity is 100%, and the size of the menu panel in the initial state is 5, and the opacity is 50%.

[0098] Optionally, in response to the progress of the movement completion satisfying the progress condition, displaying the menu panel in the initial state may include: during the movement of the preset part, calculating the distance between the current position and the initial position of the preset part, and displaying the menu panel in the initial state when it is detected that the distance is reduced.

[0099] The initial position is the position of the preset part when it is determined that the user's hand presents a menu trigger gesture and the posture of the head relative to the hand meets the preset trigger condition.

[0100] For example, when it is determined that the user intends to trigger the panel, the position of the user's fingertip is recorded as the initial position, and during the movement of the user's hand, the current position of the fingertip is monitored in real time, and the distance between the current position and the initial position is calculated. When the user's fingertip draws a closed shape in the air, the distance generally increases gradually, and then gradually decreases after reaching a critical point, and the menu panel in the initial state can be displayed at the critical point.

[0101] Step 1202: During the movement process, display a menu panel that gradually changes from an initial state to a final state according to the progress of the movement.

[0102] The closer the moving completion progress is to the preset conditions, the closer the displayed menu panel is to the final state.

[0103] Optionally, displaying a menu panel that gradually changes from an initial state to a final state based on the progress of movement completion may include: determining attribute information that matches the distance between the current position and the initial position of the preset part, and displaying a corresponding menu panel based on the attribute information, until the distance is less than or equal to a distance threshold, and then displaying the menu panel in the final state.

[0104] The attribute information is negatively correlated with the distance. That is, when other conditions remain unchanged, the greater the distance, the smaller the attribute information, and the smaller the distance, the greater the attribute information. Optionally, the current attribute information can be calculated according to the distance through a function, for example, the two can be linearly related, or inversely proportional, etc., or the attribute information corresponding to the current distance can be determined by looking up a table, etc., which is not limited here.

[0105] In this way, according to the distance between the current position and the initial position of the preset part, the menu panel of the corresponding state is displayed, which effectively simplifies the calculation and improves the processing efficiency.

[0106] In actual applications, after reaching the critical point and displaying the menu panel in the initial state, the user's fingertip movement continues to be detected. The distance between the current position of the fingertip and the initial position becomes smaller and smaller, which means that it is getting closer to the preset condition, that is, it is getting closer to the normal display of the menu panel. At this time, the visual feedback provided to the user can be the enlargement of the menu panel and the adjustment of the opacity.

[0107] In other optional implementations, the menu panel may also be displayed before the critical point. In addition, during the hand movement process, in addition to the distance between the fingertip and the initial position, other information, such as the length of the movement trajectory, the direction of the current position relative to the initial position, etc., may also be used to determine the current attribute information of the menu panel, as long as it can reflect the progress of the movement completion, and this embodiment does not limit this.

[0108] Optionally, the size of the menu panel in the final state can also be determined based on the size of the closed shape formed by the moving trajectory; wherein the size of the closed shape is positively correlated with the size of the menu panel in the final state, and the position of the menu panel in the final state is located inside the closed shape formed by the moving trajectory.

[0109] For example, if the closed shape is a circle, then when the critical point is reached, the diameter of the circle can be determined, and the size of the menu panel in the final state can be determined based on the diameter of the circle. The larger the diameter, the larger the size of the menu panel in the final state; the smaller the diameter, the smaller the size of the menu panel in the final state. In addition, the size of the menu panel in the final state (such as the diameter or the diagonal length) can be smaller than the diameter of the closed shape, and the menu panel finally displayed can be located inside the circle.

[0110] By determining the size of the menu panel according to the shape of the closed loop drawn by the user, the user can determine the size of the menu panel to be called out according to his or her current needs. Users with different vision can call out menu panels of different sizes. In different lighting scenes, the size of the called out menu panel can also be adjusted to meet the usage needs of different users, or the same user in different scenes.

[0111] In summary, this embodiment can display a menu panel with gradually changing size and opacity, reflecting the current movement progress from multiple dimensions, so that the user can understand the current situation more accurately and improve the reliability of visual feedback.

[0112] In one or more embodiments of the present disclosure, optionally, a function of canceling the display of the menu panel may be provided to the user. The menu panel may be canceled at various times: during the movement of the hand, and / or after the menu panel is normally displayed.

[0113] Optionally, after determining that the user intends to trigger the menu, the user usually moves the hand as expected and keeps looking at the hand during the process. If during the movement of the user's hand, it is detected that the movement trajectory deviates from the expected trajectory, or the head no longer looks at the hand, the triggering of the menu panel can be canceled. If there is visual feedback during the movement, the visual feedback can also be stopped.

[0114] Exemplarily, after displaying the menu panel in the initial state, in response to detecting at least one of the following, the display of the ghost or movement track stops, and the display of the gradient menu panel stops: the distance between the current position and the initial position of the preset part increases; the position of the user's head relative to the hand does not meet the trigger condition.

[0115] Take the tap gesture as an example. After the critical point, the finger tip will get closer and closer to the initial position. If the finger tip is detected to be gradually moving away from the initial position during the movement, it means that the movement trajectory deviates from expectations and the user may not intend to call up the menu panel. In this case, the visual feedback can be canceled. In addition, during the movement, if the user no longer looks at the hand but shifts the head or sight to other places, the visual feedback can also be canceled.

[0116] Optionally, after the menu panel is normally displayed, the user usually operates the menu panel with his hands and keeps his head looking at his hands during the process. Therefore, if it is detected that the hand is away from the menu panel or the head is no longer looking at the hand, the menu panel can be closed.

[0117] Exemplarily, after displaying the menu panel in the final state, the menu panel is closed in response to detecting at least one of the following: the distance between the preset position and the menu panel is greater than a distance threshold and the speed away from the menu panel is greater than a speed threshold; the position of the user's head relative to the hand does not meet the trigger condition.

[0118] Specifically, when the user is operating the menu panel, the hand may temporarily leave the menu panel but there is no intention to close it at this time. In this embodiment, when it is detected that the user's fingertips are getting farther and farther away from the panel and are moving away quickly, it is judged that the user's current intention is to close the panel, which can improve accuracy and reduce the occurrence of accidental closing.

[0119] Optionally, the detection of the fingertips and the detection of the head can be used in combination or one of them can be used. For example, when both the fingertips and the head meet the corresponding requirements, the display of the menu panel is canceled. Alternatively, as long as the fingertips meet the corresponding requirements, the display of the menu panel can be canceled.

[0120] Through the above solution, the user can be provided with a matching operation of canceling the menu panel, which is in line with the user's instinctive intuition, does not require the user to remember special closing gestures, has high accuracy, and is not prone to misoperation.

[0121] Fig.13 FIG. 1 is a flow chart of another menu triggering method provided by an embodiment of the present disclosure. Fig.13 As shown, the method includes:

[0122] Step 1301: Detect the head orientation of the user and / or the sight direction of the user.

[0123] Optionally, this step can be applied to a wearable display device, wherein the wearable display device is provided with a sensor device for collecting user head information. The head information can be the aforementioned head data and / or eye movement data, and the user's head orientation and / or the user's line of sight direction can be determined based on the head information.

[0124] In an optional implementation, the head information collected by the sensor device may include the user's head orientation and / or the user's line of sight direction, so the head orientation and / or line of sight direction can be directly obtained based on the information collected by the sensor device.

[0125] In another optional implementation, the head information collected by the sensing device may include other information, for example, the position of the eyes and / or pupils, and the line of sight direction is determined by the position of the eyes and / or pupils; or, the head information may include the angular velocity of the wearable display device, and the current head orientation may be determined based on the angular velocity and the initial posture of the head.

[0126] Step 1302: determine the intersection position of the head orientation and the hand area of ​​the user, and / or determine the intersection position of the sight line direction and the hand area, and calculate the confidence that the user has the intention to trigger the menu based on the determined intersection position.

[0127] The hand area is an area formed with a preset part of the hand as the center; the distance between the intersection position and the preset part is negatively correlated with the corresponding confidence.

[0128] Optionally, a hand image can be captured by a shooting device provided in the wearable display device. Based on the hand image, the hand area can be determined. The hand area can be centered on a key position (preset position) of the hand. For example, when the hand is in a tap gesture, the preset position can be the fingertip. The hand area is formed on a plane perpendicular to the head direction / line of sight, with the fingertip as the center and a preset length as the radius.

[0129] If the head direction / line of sight direction does not intersect with the hand area, the confidence is 0. If the intersection position of the head direction / line of sight direction and the hand area is exactly at the position of the preset part, the confidence is 1. If the intersection position is not at the position of the preset part, the confidence can be determined based on the distance between the intersection position and the preset part. The closer the distance, the higher the confidence, and the farther the distance, the lower the confidence.

[0130] Step 1303: In response to the confidence level satisfying a preset requirement and the user's hand presenting a menu triggering gesture, detecting a movement trajectory of the user's hand.

[0131] Optionally, the preset requirement may include: the confidence level is greater than a preset confidence level threshold, such as 0.8. If the confidence level meets the preset requirement, it can be considered that the position of the head relative to the hand meets the preset trigger condition.

[0132] In practical applications, one of the head orientation or the gaze direction can be selected to calculate the confidence, and the confidence can be compared with a threshold. Alternatively, the confidence can be calculated separately by the head orientation and the gaze direction, and the corresponding confidences can be weighted and summed to obtain a fused confidence, which can be compared with the threshold.

[0133] Through the above steps 1301 to 1303, it is possible to achieve: in response to the user's hand presenting a menu triggering gesture, and the head posture relative to the hand meeting the preset triggering condition, the movement trajectory of the user's hand is detected. In addition, the confidence level is determined according to the intersection of the head direction / line of sight and the hand, which can improve the accuracy of the detection and has strong applicability in practical applications.

[0134] Step 1304: In response to the movement trajectory satisfying a preset condition, display a menu panel to the user.

[0135] The specific implementation principle and process of this step can be found in the above-mentioned embodiment and will not be repeated here.

[0136] Step 1305: After the menu panel is displayed to the user, the function option selected by the user from the menu panel through the menu selection gesture is obtained, and a corresponding operation is performed according to the function option selected by the user.

[0137] The menu panel is used to provide at least one function option. Exemplarily, the function option may include: screenshot, calibration, display settings, etc., and the user may be allowed to add or hide function options.

[0138] The menu triggering gesture is the same as the menu selecting gesture. After the menu panel is displayed to the user, the user can select a function option from the menu panel while maintaining the gesture unchanged, and the device can perform a corresponding operation according to the function option selected by the user.

[0139] Optionally, during the movement of the user's hand, the device may or may not provide visual feedback. If visual feedback is to be provided, the "after displaying the menu panel to the user" referred to in this step may refer to after displaying the menu panel in the final state to the user.

[0140] In this embodiment, the gesture for triggering the menu and the gesture for finally operating the menu are consistent, for example, both are tap gestures, so that the user only needs to use one gesture from the beginning to the final operation of the menu, without the need for the user to remember additional gestures, further reducing the user's burden.

[0141] Corresponding to the menu triggering method provided in the above embodiments, the embodiments of the present disclosure also provide a menu triggering device. Fig.14 This is a structural block diagram of a menu triggering device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Fig.14 , the device may include:

[0142] The detection module 1401 is used to detect the movement trajectory of the user's hand in response to the user's hand presenting a menu trigger gesture and the position of the head relative to the hand meeting a preset trigger condition;

[0143] The display module 1402 is used to display a menu panel to the user in response to the movement trajectory meeting a preset condition.

[0144] In one or more embodiments of the present disclosure, optionally, the display module 1402 is specifically used to:

[0145] In response to the user's hand moving under the menu trigger gesture and the movement trajectory meeting a preset condition, a menu panel is displayed to the user.

[0146] In one or more embodiments of the present disclosure, optionally, the preset condition includes any one of the following:

[0147] The moving trajectory is an arbitrary closed shape;

[0148] The moving trajectory is a specific closed shape;

[0149] The movement track is a closed shape, and the size of the closed shape is within a preset size range.

[0150] In one or more embodiments of the present disclosure, optionally, the display module 1402 is specifically used to:

[0151] During the process of the user's hand moving, corresponding visual feedback is provided according to the progress of the movement, until the preset condition is met, the menu panel is displayed to the user.

[0152] In one or more embodiments of the present disclosure, optionally, when the display module 1402 provides corresponding visual feedback according to the progress of the movement completion during the user's hand movement, it is specifically used to:

[0153] During the movement of the user's hand, the shadow or movement trajectory corresponding to the preset part of the user's hand is displayed, and according to the progress of the movement, a gradient menu panel is displayed until the preset conditions are met, and the menu panel in the final state is displayed.

[0154] In one or more embodiments of the present disclosure, optionally, when displaying the gradient menu panel according to the completion progress of the movement, the display module 1402 is specifically configured to:

[0155] In response to the completion progress of the movement satisfying the progress condition, displaying the menu panel in the initial state, wherein the attribute information of the menu panel in the initial state is different from the attribute information of the final state; wherein the attribute information includes: size, and / or opacity;

[0156] During the movement process, the menu panel is displayed, gradually changing from the initial state to the final state according to the progress of the movement.

[0157] In one or more embodiments of the present disclosure, optionally, when displaying the menu panel in the initial state, the display module 1402 is specifically used to:

[0158] During the movement of the preset part, calculating the distance between the current position and the initial position of the preset part, and displaying the menu panel in the initial state when it is detected that the distance decreases;

[0159] The menu panel that gradually changes from the initial state to the final state according to the progress of the movement includes:

[0160] According to the distance between the current position of the preset part and the initial position, attribute information matching the distance is determined, and a corresponding menu panel is displayed according to the attribute information, until the distance is less than or equal to a distance threshold, a menu panel in a final state is displayed;

[0161] Among them, the initial position is: when it is determined that the user's hand presents a menu trigger gesture and the posture of the head relative to the hand meets the preset trigger condition, the position of the preset part is located; the attribute information is negatively correlated with the distance.

[0162] In one or more embodiments of the present disclosure, optionally, the display module 1402 is further used to:

[0163] After displaying the menu panel in the initial state, in response to detecting at least one of the following, stopping displaying the smear or movement track, and stopping displaying the gradient menu panel: the distance between the current position and the initial position of the preset part increases; the position of the user's head relative to the hand does not meet the trigger condition;

[0164] and / or,

[0165] After displaying the menu panel in the final state, the menu panel is closed in response to detecting at least one of the following: the distance between the preset position and the menu panel is greater than a distance threshold and the speed away from the menu panel is greater than a speed threshold; the position of the user's head relative to the hand does not meet the trigger condition.

[0166] In one or more embodiments of the present disclosure, optionally, the display module 1402 is further used to:

[0167] Determining the size of the menu panel in a final state according to the size of the closed shape formed by the movement track;

[0168] The size of the closed shape is positively correlated with the size of the menu panel in the final state, and the position of the menu panel in the final state is located inside the closed shape formed by the moving track.

[0169] In one or more embodiments of the present disclosure, optionally,

[0170] The detection module 1401 is specifically used for:

[0171] Detecting the head orientation of the user and / or the sight direction of the user;

[0172] Determine the intersection position of the head orientation and the hand area of ​​the user, and / or determine the intersection position of the sight line direction and the hand area, and calculate the confidence that the user has the intention to trigger the menu according to the determined intersection position;

[0173] In response to the confidence level meeting a preset requirement and the user's hand presenting a menu triggering gesture, detecting a movement trajectory of the user's hand;

[0174] The hand area is an area formed with a preset part of the hand as the center; the distance between the intersection position and the preset part is negatively correlated with the corresponding confidence.

[0175] In one or more embodiments of the present disclosure, optionally, the menu panel is used to provide at least one functional option; the display module 1402 is also used to:

[0176] After displaying the menu panel to the user, obtaining the function option selected by the user from the menu panel through the menu selection gesture, and performing a corresponding operation according to the function option selected by the user;

[0177] The menu triggering gesture is the same as the menu selecting gesture.

[0178] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0179] Fig.15 This is a structural block diagram of an electronic device provided by an embodiment of the present disclosure. Fig.15 The electronic device 1500 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.15 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0180] like Fig.15As shown, the electronic device 1500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1502 or a program loaded from a storage device 1508 to a random access memory (RAM) 1503. Various programs and data required for the operation of the electronic device 1500 are also stored in the RAM 1503. The processing device 1501, the ROM 1502, and the RAM 1503 are connected to each other via a bus 1504. An input / output (I / O) interface 1505 is also connected to the bus 1504.

[0181] Typically, the following devices may be connected to the I / O interface 1505: input devices 1506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 1507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1509. The communication devices 1509 may allow the electronic device 1500 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.15 The electronic device 1500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0182] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 1509, or installed from a storage device 1508, or installed from a ROM 1502. When the computer program is executed by the processing device 1501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0183] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0184] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0185] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0186] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0187] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0188] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0189] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0190] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0191] In a first aspect, according to one or more embodiments of the present disclosure, a menu triggering method is provided, the method comprising:

[0192] In response to the user's hand presenting a menu triggering gesture, and the posture of the head relative to the hand meeting a preset triggering condition, detecting the movement trajectory of the user's hand;

[0193] In response to the movement trajectory satisfying a preset condition, a menu panel is displayed to the user.

[0194] According to one or more embodiments of the present disclosure, optionally, in response to the movement trajectory satisfying a preset condition, displaying a menu panel to the user includes:

[0195] In response to the user's hand moving under the menu trigger gesture and the movement trajectory meeting a preset condition, a menu panel is displayed to the user.

[0196] According to one or more embodiments of the present disclosure, optionally, the preset condition includes any one of the following:

[0197] The moving trajectory is an arbitrary closed shape;

[0198] The moving trajectory is a specific closed shape;

[0199] The movement track is a closed shape, and the size of the closed shape is within a preset size range.

[0200] According to one or more embodiments of the present disclosure, optionally, in response to the movement trajectory satisfying a preset condition, displaying a menu panel to the user includes:

[0201] During the process of the user's hand moving, corresponding visual feedback is provided according to the progress of the movement, until the preset condition is met, the menu panel is displayed to the user.

[0202] According to one or more embodiments of the present disclosure, optionally, during the process of the user's hand moving, corresponding visual feedback is provided according to the progress of the movement, including:

[0203] During the movement of the user's hand, the shadow or movement trajectory corresponding to the preset part of the user's hand is displayed, and according to the progress of the movement, a gradient menu panel is displayed until the preset conditions are met, and the menu panel in the final state is displayed.

[0204] According to one or more embodiments of the present disclosure, optionally, displaying a gradient menu panel according to the completion progress of the movement includes:

[0205] In response to the completion progress of the movement satisfying the progress condition, displaying the menu panel in the initial state, wherein the attribute information of the menu panel in the initial state is different from the attribute information of the final state; wherein the attribute information includes: size, and / or opacity;

[0206] During the movement process, the menu panel is displayed, gradually changing from the initial state to the final state according to the progress of the movement.

[0207] According to one or more embodiments of the present disclosure, optionally, in response to the movement completion progress satisfying the progress condition, displaying the menu panel in the initial state includes:

[0208] During the movement of the preset part, calculating the distance between the current position and the initial position of the preset part, and displaying the menu panel in the initial state when it is detected that the distance decreases;

[0209] The menu panel that gradually changes from the initial state to the final state according to the progress of the movement includes:

[0210] According to the distance between the current position of the preset part and the initial position, attribute information matching the distance is determined, and a corresponding menu panel is displayed according to the attribute information, until the distance is less than or equal to a distance threshold, a menu panel in a final state is displayed;

[0211] Among them, the initial position is: when it is determined that the user's hand presents a menu trigger gesture and the posture of the head relative to the hand meets the preset trigger condition, the position of the preset part is located; the attribute information is negatively correlated with the distance.

[0212] According to one or more embodiments of the present disclosure, optionally, the method further includes:

[0213] After displaying the menu panel in the initial state, in response to detecting at least one of the following, stopping displaying the smear or movement track, and stopping displaying the gradient menu panel: the distance between the current position and the initial position of the preset part increases; the position of the user's head relative to the hand does not meet the trigger condition;

[0214] and / or,

[0215] After displaying the menu panel in the final state, the menu panel is closed in response to detecting at least one of the following: the distance between the preset position and the menu panel is greater than a distance threshold and the speed away from the menu panel is greater than a speed threshold; the position of the user's head relative to the hand does not meet the trigger condition.

[0216] According to one or more embodiments of the present disclosure, optionally, the method further includes:

[0217] Determining the size of the menu panel in a final state according to the size of the closed shape formed by the movement track;

[0218] The size of the closed shape is positively correlated with the size of the menu panel in the final state, and the position of the menu panel in the final state is located inside the closed shape formed by the moving track.

[0219] According to one or more embodiments of the present disclosure, optionally, in response to a user's hand presenting a menu triggering gesture, and the posture of the head relative to the hand meeting a preset triggering condition, detecting a movement trajectory of the user's hand includes:

[0220] Detecting the head orientation of the user and / or the sight direction of the user;

[0221] Determine the intersection position of the head orientation and the hand area of ​​the user, and / or determine the intersection position of the sight line direction and the hand area, and calculate the confidence that the user has the intention to trigger the menu according to the determined intersection position;

[0222] In response to the confidence level meeting a preset requirement and the user's hand presenting a menu triggering gesture, detecting a movement trajectory of the user's hand;

[0223] The hand area is an area formed with a preset part of the hand as the center; the distance between the intersection position and the preset part is negatively correlated with the corresponding confidence.

[0224] According to one or more embodiments of the present disclosure, optionally, the menu panel is used to provide at least one functional option; the method further includes:

[0225] After displaying the menu panel to the user, obtaining the function option selected by the user from the menu panel through the menu selection gesture, and performing a corresponding operation according to the function option selected by the user;

[0226] The menu triggering gesture is the same as the menu selecting gesture.

[0227] In a second aspect, according to one or more embodiments of the present disclosure, a menu triggering device is provided, comprising:

[0228] A detection module, configured to detect a movement trajectory of the user's hand in response to the user's hand presenting a menu triggering gesture and the position of the head relative to the hand meeting a preset triggering condition;

[0229] The display module is used to display the menu panel to the user in response to the movement trajectory meeting the preset condition.

[0230] According to one or more embodiments of the present disclosure, optionally, the display module is specifically used for:

[0231] In response to the user's hand moving under the menu trigger gesture and the movement trajectory meeting a preset condition, a menu panel is displayed to the user.

[0232] According to one or more embodiments of the present disclosure, optionally, the preset condition includes any one of the following:

[0233] The moving trajectory is an arbitrary closed shape;

[0234] The moving trajectory is a specific closed shape;

[0235] The movement track is a closed shape, and the size of the closed shape is within a preset size range.

[0236] According to one or more embodiments of the present disclosure, optionally, the display module is specifically used for:

[0237] During the process of the user's hand moving, corresponding visual feedback is provided according to the progress of the movement, until the preset condition is met, the menu panel is displayed to the user.

[0238] According to one or more embodiments of the present disclosure, optionally, when the display module provides corresponding visual feedback according to the progress of the movement completion during the movement of the user's hand, it is specifically used to:

[0239] During the movement of the user's hand, the shadow or movement trajectory corresponding to the preset part of the user's hand is displayed, and according to the progress of the movement, a gradient menu panel is displayed until the preset conditions are met, and the menu panel in the final state is displayed.

[0240] According to one or more embodiments of the present disclosure, optionally, when the display module displays the gradient menu panel according to the completion progress of the movement, it is specifically used to:

[0241] In response to the completion progress of the movement satisfying the progress condition, displaying the menu panel in the initial state, wherein the attribute information of the menu panel in the initial state is different from the attribute information of the final state; wherein the attribute information includes: size, and / or opacity;

[0242] During the movement process, the menu panel is displayed, gradually changing from the initial state to the final state according to the progress of the movement.

[0243] According to one or more embodiments of the present disclosure, optionally, when displaying the menu panel in the initial state, the display module is specifically used to:

[0244] During the movement of the preset part, calculating the distance between the current position and the initial position of the preset part, and displaying the menu panel in the initial state when it is detected that the distance decreases;

[0245] The menu panel that gradually changes from the initial state to the final state according to the progress of the movement includes:

[0246] According to the distance between the current position of the preset part and the initial position, attribute information matching the distance is determined, and a corresponding menu panel is displayed according to the attribute information, until the distance is less than or equal to a distance threshold, a menu panel in a final state is displayed;

[0247] Among them, the initial position is: when it is determined that the user's hand presents a menu trigger gesture and the posture of the head relative to the hand meets the preset trigger condition, the position of the preset part is located; the attribute information is negatively correlated with the distance.

[0248] According to one or more embodiments of the present disclosure, optionally, the display module is further used for:

[0249] After displaying the menu panel in the initial state, in response to detecting at least one of the following, stopping displaying the smear or movement track, and stopping displaying the gradient menu panel: the distance between the current position and the initial position of the preset part increases; the position of the user's head relative to the hand does not meet the trigger condition;

[0250] and / or,

[0251] After displaying the menu panel in the final state, the menu panel is closed in response to detecting at least one of the following: the distance between the preset position and the menu panel is greater than a distance threshold and the speed away from the menu panel is greater than a speed threshold; the position of the user's head relative to the hand does not meet the trigger condition.

[0252] According to one or more embodiments of the present disclosure, optionally, the display module is further used for:

[0253] Determining the size of the menu panel in a final state according to the size of the closed shape formed by the movement track;

[0254] The size of the closed shape is positively correlated with the size of the menu panel in the final state, and the position of the menu panel in the final state is located inside the closed shape formed by the moving track.

[0255] According to one or more embodiments of the present disclosure, optionally, the detection module is specifically used to:

[0256] Detecting the head orientation of the user and / or the sight direction of the user;

[0257] Determine the intersection position of the head orientation and the hand area of ​​the user, and / or determine the intersection position of the sight line direction and the hand area, and calculate the confidence that the user has the intention to trigger the menu according to the determined intersection position;

[0258] In response to the confidence level meeting a preset requirement and the user's hand presenting a menu triggering gesture, detecting a movement trajectory of the user's hand;

[0259] The hand area is an area formed with a preset part of the hand as the center; the distance between the intersection position and the preset part is negatively correlated with the corresponding confidence.

[0260] According to one or more embodiments of the present disclosure, optionally, the menu panel is used to provide at least one functional option; and the display module is further used to:

[0261] After displaying the menu panel to the user, obtaining the function option selected by the user from the menu panel through the menu selection gesture, and performing a corresponding operation according to the function option selected by the user;

[0262] The menu triggering gesture is the same as the menu selecting gesture.

[0263] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device comprising: a memory and at least one processor; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the menu triggering method as described in any one of the first aspects.

[0264] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions, and when a processor executes the computer execution instructions, the menu triggering method as described in any one of the first aspects is implemented.

[0265] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program, which implements any method described in the first aspect when executed by a processor.

[0266] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0267] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0268] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A menu triggering method, characterized in that: The method comprises: In response to the user's hand presenting a menu triggering gesture, and the posture of the head relative to the hand meeting a preset triggering condition, detecting the movement trajectory of the user's hand; In response to the movement trajectory satisfying a preset condition, a menu panel is displayed to the user.

2. The method according to claim 1, characterized in that In response to the movement trajectory meeting a preset condition, a menu panel is displayed to the user, including: In response to the user's hand moving under the menu trigger gesture and the movement trajectory meeting a preset condition, a menu panel is displayed to the user.

3. The method according to claim 1 or 2, characterized in that: The preset condition includes any one of the following: The moving trajectory is an arbitrary closed shape; The moving trajectory is a specific closed shape; The movement track is a closed shape, and the size of the closed shape is within a preset size range.

4. The method according to claim 1, characterized in that: In response to the movement trajectory meeting a preset condition, a menu panel is displayed to the user, including: During the process of the user's hand moving, corresponding visual feedback is provided according to the progress of the movement, until the preset condition is met, the menu panel is displayed to the user.

5. The method according to claim 4, characterized in that As the user's hand moves, corresponding visual feedback is provided based on the progress of the movement, including: During the movement of the user's hand, the shadow or movement trajectory corresponding to the preset part of the user's hand is displayed, and according to the progress of the movement, a gradient menu panel is displayed until the preset conditions are met, and the menu panel in the final state is displayed.

6. The method according to claim 5, characterized in that According to the progress of the movement, a gradient menu panel is displayed, including: In response to the completion progress of the movement satisfying the progress condition, displaying the menu panel in the initial state, wherein the attribute information of the menu panel in the initial state is different from the attribute information of the final state; wherein the attribute information includes: size, and / or opacity; During the movement process, the menu panel is displayed, gradually changing from the initial state to the final state according to the progress of the movement.

7. The method according to claim 6, characterized in that In response to the moving completion progress satisfying the progress condition, displaying the menu panel in the initial state, including: During the movement of the preset part, calculating the distance between the current position and the initial position of the preset part, and displaying the menu panel in the initial state when it is detected that the distance decreases; The menu panel that gradually changes from the initial state to the final state according to the progress of the movement includes: According to the distance between the current position of the preset part and the initial position, attribute information matching the distance is determined, and a corresponding menu panel is displayed according to the attribute information, until the distance is less than or equal to a distance threshold, a menu panel in a final state is displayed; Among them, the initial position is: when it is determined that the user's hand presents a menu trigger gesture and the posture of the head relative to the hand meets the preset trigger condition, the position of the preset part is located; the attribute information is negatively correlated with the distance.

8. The method according to claim 6 or 7, characterized in that: Also includes: After displaying the menu panel in the initial state, in response to detecting at least one of the following, stopping displaying the smear or movement track, and stopping displaying the gradient menu panel: the distance between the current position and the initial position of the preset part increases; the position of the user's head relative to the hand does not meet the trigger condition; and / or, After displaying the menu panel in the final state, the menu panel is closed in response to detecting at least one of the following: the distance between the preset position and the menu panel is greater than a distance threshold and the speed away from the menu panel is greater than a speed threshold; the position of the user's head relative to the hand does not meet the trigger condition.

9. The method according to claim 6 or 7, characterized in that: The method further comprises: Determining the size of the menu panel in a final state according to the size of the closed shape formed by the movement track; The size of the closed shape is positively correlated with the size of the menu panel in the final state, and the position of the menu panel in the final state is located inside the closed shape formed by the moving track.

10. The method according to claim 1 or 2, characterized in that: In response to determining that the user's hand presents a menu triggering gesture, and the posture of the head relative to the hand satisfies a preset triggering condition, detecting a movement trajectory of the user's hand includes: Detecting the head orientation of the user and / or the sight direction of the user; Determine the intersection position of the head orientation and the hand area of ​​the user, and / or determine the intersection position of the sight line direction and the hand area, and calculate the confidence that the user has the intention to trigger the menu according to the determined intersection position; In response to the confidence level meeting a preset requirement and the user's hand presenting a menu triggering gesture, detecting a movement trajectory of the user's hand; The hand area is an area formed with a preset part of the hand as the center; the distance between the intersection position and the preset part is negatively correlated with the corresponding confidence.

11. The method according to claim 1 or 2, characterized in that: The menu panel is used to provide at least one function option; the method further includes: After displaying the menu panel to the user, obtaining the function option selected by the user from the menu panel through the menu selection gesture, and performing a corresponding operation according to the function option selected by the user; The menu triggering gesture is the same as the menu selecting gesture.

12. A menu triggering device, characterized in that: The device comprises: A detection module, configured to detect a movement trajectory of the user's hand in response to the user's hand presenting a menu triggering gesture and the position of the head relative to the hand meeting a preset triggering condition; The display module is used to display the menu panel to the user in response to the movement trajectory meeting the preset condition.

13. An electronic device, characterized in that: include: memory and at least one processor; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the menu triggering method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the menu triggering method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the menu triggering method described in any one of claims 1 to 11 is implemented.

Images