Augmented reality menu selection system purely based on head rotation
Through an extended realistic menu selection system based solely on head rotation, the IMU sensor and processing unit detect and process head rotation actions are used to detect and process head rotation actions, the problem of misoperation in complex environments in the prior art is solved, the operation stability and accuracy are achieved, and the user interface is enhanced.
Patent Information
- Application Number
- CN202411970735.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-15
AI Technical Summary
Existing head posture-based menu selection techniques are susceptible to natural head swings or external factors in complex environments, resulting in misidentification and involuntary selection confirmation.
The extended realistic menu selection system based on head rotation is adopted. The head rotation action is detected through the IMU sensor module. The processing unit controls the display content. The user performs one-dimensional pointing and selection confirmation through the head yaw and pitch actions. The system tracks and calculates the head rotation vector in real time to ensure the accuracy and smoothness of the operation.
It effectively avoids misoperation caused by head jitter or external interference in complex environments, improves the stability and accuracy of operations, and enhances the intuitiveness and interactivity of the user interface.
Smart Images
Figure CN119828894A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of virtual interaction, and in particular to an extended reality menu selection system based solely on head rotation. Background Art
[0002] The common problem of "phone addicts" refers to individuals who are constantly addicted to their mobile phones, which raises concerns about safety and musculoskeletal symptoms. Postural computing, as an emerging interaction paradigm, provides a promising solution to this problem using optical see-through head-mounted displays (OHMDs), which focuses on meeting users' mobile information needs in a collaborative way, i.e., digital assistance minimizes its interference with users' daily activities and allows them to maintain a natural posture.
[0003] In existing head gesture-based menu selection techniques, users usually rely on a composite head gesture consisting of yaw and pitch to control the pointer on the menu to move in 2D. However, confirming the selection of a target icon usually requires the user to move the pointer to the target icon boundary and stay for a certain time or cross its boundary multiple times. However, when users are engaged in daily activities such as walking in a supermarket or standing on a moving bus, the natural swing of their heads or external factors (e.g., other people, obstacles, etc.) can easily interfere with staying or cause misrecognition of multiple crossings. Staying is a frequently used selection confirmation technique for menu selection using only the head. However, expressing the intention of selection confirmation by staying may cause some problems, namely, a short stay time may still lead to involuntary selection confirmation, while a long stay will degrade performance. Summary of the invention
[0004] In view of the shortcomings of the prior art, the present invention provides an extended reality menu selection system based solely on head rotation, which solves the problem that head shaking during walking or external factors (for example, other people, obstacles, etc.) can easily interfere with staying or cause misrecognition of multiple crossings.
[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: an extended reality menu selection system based solely on head rotation, comprising: Optical see-through head mounted display; IMU sensor module, used to detect the user's head rotation movement; A processing unit, used to receive signals from the IMU sensor module and control the display content of the optical see-through head-mounted display, wherein the processing unit performs one-dimensional pointing through the user's head yaw movement and performs selection confirmation through the user's head pitch movement; The display interface is used to display menu options and visual feedback. Users can complete one-dimensional pointing with a simple head gesture - yaw or pitch. The specific operation process is that the user selects an icon by rotating the head along the x-axis or y-axis, and when the posture of the head on the other axis exceeds a specific selection confirmation threshold, this action becomes the key to triggering selection confirmation. In order to activate the menu, the system uses a head wake-up action, which is judged by monitoring whether the head pitch speed exceeds 20 degrees per second. In order to select a target icon, the user first pre-selects the target icon or the column containing it by yaw of the head, and then extends the arrow in the target icon by pitching the head until the arrow is complete. Full coverage of the target icon, that is, the pitch amplitude exceeds the selection confirmation threshold. Once the pitch speed is detected to meet the conditions, the system will record the horizontal gaze direction of the user's head as a fixed reference direction to ensure that the menu plane is always perpendicular to this reference direction. With the menu as the center, the user's head position is at the center point. When the user turns his head to the upper right, the rotation vector of the user's line of sight in front of him forms a projection on the menu plane, and then the projection is decomposed into two vertical components in the horizontal and vertical directions of the menu, respectively serving the selection and confirmation of the target. The entire system tracks and calculates the user's head rotation vector in real time to ensure the accuracy and smoothness of the selection and confirmation operations.
[0006] Preferably, the menu options are arranged in multiple rows or columns. The user pre-selects the target icon in the one-dimensional pointing stage by yaw head action. After the user activates the menu, the user can rotate the head left and right to project the ray to the menu plane and move the cursor to the menu area. When the head turns left and right, the cursor will move with the line of sight and cross to different menu columns. When the line of sight falls on a specific column, two triangular marks will be displayed at the top and bottom of the column respectively, indicating the location of the user's current line of sight, thereby realizing the selection of the column. The user confirms the selection by the pitch amplitude exceeding the selection confirmation threshold. The pitch direction can be upward or downward, and the selection confirmation threshold is statistically significantly greater than the shaking angle of the user's head when walking, so as to prevent the user from accidentally touching the screen due to head shaking when walking. The display interface includes an AA visualization system for displaying the amplitude and direction of the pitch posture required by the user. The amplitude arrow expands or contracts in the pre-selected icon to indicate the selection confirmation process. The amplitude arrow serves as a one-dimensional pointing stage pointer, and the length change of the arrow body is proportional to the pitch angle of the user's head. The angle between the head and the horizontal plane changes continuously, and the length of the amplitude arrow is also dynamically adjusted accordingly. When the angle between the head and the horizontal plane rotates from 0° to 6.46°, the amplitude arrow grows dynamically upward. When the angle between the head and the horizontal plane rotates from 0° to 10.72°, the amplitude arrow grows dynamically downward. The IMU sensor module includes a head wake-up unit for activating the menu display. When it is detected that the user's head pitch speed exceeds 20 degrees per second and the head elevation angle reaches a threshold of 20 degrees, the horizontal component of the head gaze direction is recorded as the reference direction, and the menu plane is always perpendicular to the reference direction. The menu icon adopts a black and white contrast strategy, and the icon boundary is a double-layer structure, the outer layer is black, the inner layer is white, and the icon logo is black and white.
[0007] Preferably, the viewing angle of the menu icons is set to 3 degrees, the vertical viewing angle gap between two icons is 0.23 degrees, the horizontal viewing angle of the menu area is 43 degrees, and the icon columns are evenly distributed horizontally in the menu area.
[0008] Preferably, the system also includes a real-time visual feedback mechanism for directly mapping the color increase of the target icon to the rotation amplitude of the user's head to intuitively display the selection process. The real-time visual feedback mechanism includes a dynamic arrow consisting of a triangular tip and a square body, and its length and direction change in real time according to the pitch posture of the user's head, and is used to instruct the user to confirm the selection. By introducing the arrow pointer as a one-dimensional pointing indicator, the intuitiveness and interactivity of the user interface are effectively improved. Since the arrow pointer only appears in the pre-selected icons, its unique design allows the arrow (in a single-line menu) or a combination of two arrows (in a double-line menu) to act as a pointer in the one-dimensional pointing stage, thereby eliminating the need for a traditional circle or cross cursor to indicate the exact point where the user's head gaze line intersects the menu plane in the one-dimensional pointing stage.
[0009] Beneficial Effects The present invention provides an extended reality menu selection system based solely on head rotation. Compared with the prior art, it has the following beneficial effects: 1. In the present invention, it is proved that head movement can be used as an effective input means, which greatly improves the input efficiency and the fluency of operation. In terms of reducing misoperation, the present invention cleverly uses the division of labor between yaw and pitch movements to avoid accidental selection confirmation caused by head yaw in complex environments, ensuring the operation stability of users when bypassing obstacles or crowds. In addition, the design takes into account the natural swing of the head during walking, and through the one-dimensional pointing mechanism that mainly affects pitch, it ensures the accuracy and stability of the user's operation in a moving state, which will not be affected by the head movement caused by walking. It not only brings significant improvements to the current head-mounted display device, but also provides a new interaction method for future virtual reality, augmented reality and mixed reality devices, and has broad application prospects and commercial potential; 2. In the present invention, the user's head rotation amplitude is mapped by color changes, providing the user with instant operation feedback. This design not only enhances the interactivity of the user interface, but also forms a complete selection confirmation closed loop through the visual feedback of AA real-time color growth, so that the user can intuitively confirm the selection through the expansion and contraction of the arrow, greatly improving the accuracy of the operation and the user's satisfaction. The flexible menu layout design, whether single row or double column, can adapt to different usage scenarios and user needs, improve space utilization, and enable users to operate freely in different environments; 3. By introducing the arrow pointer as an indicator for one-dimensional pointing, the intuitiveness and interactivity of the user interface are effectively improved. Since the arrow pointer only appears in the pre-selected icons, its unique design allows the arrow (in a single-line menu) or a combination of two arrows (in a double-line menu) to act as a pointer in the one-dimensional pointing stage, so that in the one-dimensional pointing stage, the traditional circle or cross cursor is no longer needed to indicate the exact point where the user's head gaze line intersects with the menu plane. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 This is a system framework diagram of an extended reality menu selection system based solely on head rotation proposed by the present invention; Figure 2 This is a projection display diagram of the rotation vector of the extended reality menu selection system based solely on head rotation on the menu plane proposed by the present invention; Figure 3 A process diagram of a user of an extended reality menu selection system based solely on head rotation proposed by the present invention using amplitude arrow technology to summon an application menu and select music while walking; Figure 4 This is a real-life diagram of the AA technology in multi-column menu selection in an extended reality menu selection system based solely on head rotation proposed by the present invention. DETAILED DESCRIPTION
[0011] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0012] See also Figure 1-Figure 4 The present invention provides two technical solutions, which specifically include the following embodiments: Embodiment 1: An extended reality menu selection system based solely on head rotation, comprising: Optical see-through head mounted display; An IMU sensor module is used to detect the yaw and pitch movements of the user's head. The IMU sensor module includes a head wake-up unit for activating the menu display. When it is detected that the pitch speed of the user's head exceeds a threshold of 20 degrees per second, the horizontal component of the head gaze direction is recorded as a reference direction, and the menu plane is always perpendicular to the reference direction. The processing unit is used to receive the signal of the IMU sensor module and control the display content of the optical perspective head-mounted display. The processing unit performs one-dimensional pointing through the user's head yaw movement and selects and confirms through the user's head pitch movement. The amplitude arrow is used as a pointer in the one-dimensional pointing stage. The length of the arrow body changes in direct proportion to the user's head pitch angle. The angle between the head and the horizontal plane changes continuously, and the length of the amplitude arrow is also dynamically adjusted accordingly. During the process of the head rotating upward from 0° to 6.46° with the horizontal plane, the amplitude arrow grows dynamically upward, and during the process of the head rotating downward from 0° to 10.72° with the horizontal plane, the amplitude arrow grows dynamically downward. The user can control the selection process with simple head movements: as the color of the target icon gradually increases, the user can intuitively know that the selection process is progressing, until the color reaches the maximum value, indicating that the selection has been completed, the user first deflects his head to pre-select the target icon, and then pitches his head with an amplitude exceeding the selection confirmation threshold to confirm the selection. The pitch direction can be upward or downward. The above design can be further expanded to a double-column menu. In this case, two icons in one column are pre-selected simultaneously through the head tilt gesture in the one-dimensional pointing stage; see Figure 3 (d), if the user wants to select the lower (or upper) part of the pre-selected icon, he needs to tilt his head downward (or upward). When the menu icons are arranged in two rows, the user pre-selects icons in two columns at the same time in the one-dimensional pointing stage through the yaw head gesture; Figure 3 As shown in (d), if users want to select a lower (or higher) pre-selected icon, they need to tilt their heads downward (or upward); Figure 3 As shown in (e), in a mobile environment, head oscillations caused by walking or other external interferences, such as crowds or obstacles, may trigger cross-recognition and lead to undesirable selection confirmation. In contrast, the amplitude arrow technology releases the potential of the pitch head gesture by simplifying the pointing operation from two dimensions to one dimension, making it a direct and reliable means of selection confirmation. By setting appropriate pitch-up and pitch-down thresholds, the AA technology can effectively resist the influence of walking and external factors on head movement, achieving high accuracy, fast response and excellent user experience. The display interface is used. The user pre-selects the target icon in the one-dimensional pointing stage through the yaw head movement. The user confirms the selection by the pitch amplitude exceeding the selection confirmation threshold. The pitch direction can be upward or downward. The display interface includes an AA visualization system for displaying the amplitude and direction of the user's desired pitch posture. The amplitude arrow expands or contracts in the pre-selected icon to indicate the selection confirmation process, as shown in the attached figure. Figure 2As shown in the figure, the user only needs to use a simple head posture - yaw or pitch - to complete one-dimensional pointing. The specific operation process is that the user selects an icon by rotating the head along the x-axis or y-axis, and when the posture of the head on the other axis exceeds a specific selection confirmation threshold, this action becomes the key to triggering selection confirmation. In order to activate the menu, the system uses a head wake-up action, which monitors whether the head pitch speed exceeds 20 degrees per second and the head elevation angle reaches a threshold of 20 degrees as a judgment. In order to select a target icon, the user first pre-selects the target icon or the column containing it by yaw of the head, and then expands the arrow in the target icon by pitching the head until the arrow completely covers the target icon, that is, the pitch amplitude exceeds the selection confirmation threshold. Once it is detected that the pitch speed meets the conditions, the system records the horizontal gaze direction of the user's head as a fixed reference direction to ensure that the menu plane is always perpendicular to this reference direction, as shown in FIG. Figure 2 As shown, with the menu as the center, the user's head is at the center point. When the user turns his head to the upper right, this rotation vector is projected on the menu plane, thereby generating a head rotation direction vector. The vector is then decomposed into two vertical components, which serve the selection and confirmation of the target respectively. The entire system tracks and calculates the user's head rotation vector in real time, ensuring the accuracy and smoothness of the selection and confirmation operations.
[0013] The layout of the two-dimensional menu icons is converted into two one-dimensional horizontal and vertical directions for pointing and confirmation respectively, and the yaw and pitch movements of the user's head are used to realize pointing operations. The user can complete the pointing operation only through head gestures, which not only reduces the user's cognitive burden and improves the intuitiveness of the operation, but also enhances the friendliness of the user interface, so that even non-technical users can quickly get started. At the same time, this efficient input method proves that head movement can be used as an effective input means through continuous simple head gestures, greatly improving the input efficiency and the smoothness of operation. In terms of reducing misoperation, the present invention cleverly uses the division of labor of yaw and pitch movements to avoid accidental selection confirmation caused by head yaw in complex environments, thereby ensuring the user's operational stability when bypassing obstacles or crowds.
[0014] Embodiment 2: On the basis of the first embodiment, the menu icon adopts a black and white contrast strategy, the icon boundary is a double-layer structure, the outer layer is black, the inner layer is white, the icon logo is black and white, the menu icon's viewing angle is set to 3 degrees, the vertical viewing angle gap between the two icons is 0.23 degrees, and the horizontal viewing angle of the menu area is 43°. It should be noted that the horizontal viewing angle adopts the horizontal viewing angle of the HoloLens device. In practice, the horizontal viewing angle will be different according to different devices. The icons are evenly distributed horizontally in the menu area. The system also includes a real-time visual feedback mechanism for directly mapping the color change of the target icon to the user's head rotation amplitude to intuitively display the selection process. The real-time visual feedback mechanism includes a dynamic arrow, which is composed of a triangular tip and a square body. Its length and direction change in real time according to the pitch posture of the user's head, which is used to instruct the user to confirm the selection. The feedback is realized by an arrow composed of a triangular tip and a square body. The arrow will expand or shrink accordingly according to the needs of the pre-selected icon. The direction of the arrow clearly indicates what kind of pitch posture the user needs to perform to confirm the pre-selected icon. The length of the arrow body dynamically displays the real-time amplitude of the head pitch. The amplitude thresholds of 6.46° and 10.72° are set for the upward and downward pitch respectively. The field of view angle of each menu icon is fixed at 3°, and the change in the length of the arrow body intuitively maps the head pitch angle. Specifically, the upward growth of the arrow indicates that the user needs to rotate the head upward by 6.46°, and the downward growth corresponds to the head rotating downward by 10.72°. The color increase is used to map the user's head rotation amplitude, providing users with instant operation feedback. This design not only enhances the interactivity of the user interface, but also forms a complete selection confirmation closed loop through the visual feedback of AA real-time color growth, allowing users to intuitively confirm the selection through the expansion and contraction of the arrow, greatly improving the accuracy of the operation and user satisfaction. The flexible menu layout design, whether single-row or double-column, can adapt to different usage scenarios and user needs, improve space utilization, and enable users to operate freely in different environments. Preferably, with Figure 4The AA technique in multi-column menu selection is demonstrated: 1. The subject quickly raises his head upward until a head-driven cursor and application menu appear; 2. When the head-driven cursor enters the menu area, the cursor disappears and two arrows appear on the two icons in the column where the head is fixated; 3. The subject swings his head to the left to pre-select the second column; 4. In order to select the target icon (the lower icon in the second column), the subject swings his head downward to expand the arrow in the target icon until the arrow completely covers the target icon, that is, the swing amplitude exceeds the selection confirmation threshold (10.72°). Since the arrow pointer only appears in the pre-selected icons, its unique design allows the arrow (in a single-row menu) or a combination of two arrows (in a double-row menu) to act as a pointer in the one-dimensional pointing stage, so that the traditional circle or cross cursor is no longer needed in the one-dimensional pointing stage to indicate the exact point where the user's head gaze line intersects the menu plane.
[0015] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the application should be included in the protection scope of the present application.
Claims
1. An extended reality menu selection system based solely on head rotation, characterized in that: include: Optical see-through head mounted display; IMU sensor module, used to detect the user's head rotation movement; A processing unit, used to receive signals from the IMU sensor module and control the display content of the optical see-through head-mounted display, wherein the processing unit performs one-dimensional pointing through the user's head yaw movement and performs selection confirmation through the user's head pitch movement; Display interface, used to present menu options and visual feedback.
2. The augmented reality menu selection system based solely on head rotation according to claim 1, characterized in that: The menu options are arranged in multiple rows or columns. The user pre-selects the target icon in the one-dimensional pointing stage through a yaw head movement. The user confirms the selection by a pitch amplitude exceeding a selection confirmation threshold. The pitch direction can be upward or downward, and the selection confirmation threshold is statistically significantly greater than the shaking angle of the user's head when walking, thereby preventing the user from accidentally touching the icon due to head shaking when walking.
3. The augmented reality menu selection system based solely on head rotation according to claim 1, characterized in that: The display interface includes an AA visualization system for displaying the amplitude and direction of the pitch posture required by the user, and the amplitude arrow expands or contracts in the pre-selected icon to indicate the selection confirmation process.
4. The augmented reality menu selection system based solely on head rotation according to claim 3, characterized in that: The amplitude arrow serves as a one-dimensional pointing stage pointer. The length change of the arrow body is proportional to the pitch angle of the user's head. The angle between the head and the horizontal plane changes continuously, and the length of the amplitude arrow is dynamically adjusted accordingly. When the head rotates upward to 6.46°, the amplitude arrow dynamically grows upward. When the head rotates downward 10.72°, the amplitude arrow dynamically grows downward.
5. The augmented reality menu selection system based solely on head rotation according to claim 1, characterized in that: The IMU sensor module includes a head wake-up unit for activating the menu display; when it is detected that the user's head pitch speed exceeds 20 degrees per second and the head elevation angle reaches a threshold of 20 degrees, the horizontal component of the head gaze direction is recorded as the reference direction, and the menu plane is always perpendicular to the reference direction.
6. The augmented reality menu selection system based solely on head rotation according to claim 5, characterized in that: The menu icon adopts a black and white contrast strategy, the icon border is a double-layer structure, the outer layer is black, the inner layer is white, and the icon logo is black and white.
7. The augmented reality menu selection system based solely on head rotation according to claim 6, characterized in that: The viewing angle of the menu icons is set to 3 degrees, the vertical viewing angle gap between two icons is 0.23 degrees, the horizontal viewing angle of the menu area is 43°, and the icon columns are evenly distributed horizontally in the menu area.
8. The augmented reality menu selection system based solely on head rotation according to claim 1, characterized in that: The system also includes a real-time visual feedback mechanism for directly mapping the color increase of the target icon to the user's head rotation amplitude to intuitively display the selection process.
9. The augmented reality menu selection system based solely on head rotation according to claim 8, characterized in that: The real-time visual feedback mechanism includes a dynamic arrow consisting of a triangular tip and a square body, the length and direction of which change in real time according to the pitch posture of the user's head, and is used to instruct the user to confirm the selection.
10. The augmented reality menu selection system based solely on head rotation according to claim 1, characterized in that: The system includes a data storage unit for storing user usage habits, preference settings and system configuration information.