A method for adaptive display and control of a cursor in virtual reality

Abstract

The application discloses a kind of virtual reality in cursor adaptive display and control method, it is related to interactive interface cursor adaptive display technical field, solve the technical problem that cursor control accuracy is not high in three-dimensional man-machine interface, visual feedback information is insufficient, its technical scheme main point is based on interactive target prediction technology, the final target control is predicted, and based on this, cursor form control, control pre-naked hand interaction of cursor adaptive display rendering and zoom cursor C / D ratio gain are carried out.Not only the pointing accuracy and efficiency are enhanced, but also the operation of user provides immediate visual feedback, significantly improves the user experience.Meanwhile, the method has high universality, can be well migrated to other three-dimensional man-machine interaction pointing technology.

Description

Technical Field

[0001] This application relates to the field of adaptive cursor display technology for interactive interfaces, and more particularly to a method for adaptive cursor display and control in virtual reality. Background Technology

[0002] In recent years, the emergence and development of immersive display technology—virtual reality—has provided new opportunities for interface interaction design and promoted the development of user experience. Virtual reality is a computer system that can create and experience virtual worlds. It is computer-generated and acts on the user through sight, hearing, touch, and smell, creating an interactive visual simulation that gives the user an immersive feeling. Immersion, interactivity, and imaginativeness are the three basic characteristics of virtual reality systems. The interactive relationships in three-dimensional scenes are more complex, and the input and output methods are more diverse, involving multiple sensory channels such as vision, hearing, and touch.

[0003] Adaptive cursor control includes controlling the display control ratio and controlling cursor shape changes. The former refers to adjusting the ratio of the amount of motion of the real hand to the change in the cursor's shape, thus determining how the displacement and rotation of the input device are transmitted to the cursor in the virtual environment. The latter controls the cursor shape changes based on the different characteristic attributes of the interactive object. Currently, research on adaptive cursor display is relatively mature in the field of 2D human-computer interface interaction. However, with the development of virtual reality technology, the significance of human-computer interaction research is no longer limited to traditional 2D human-computer interface environments based on keyboard and mouse input. It also has broad research value in 3D scene interaction processes based on ray input from controllers or gestures in virtual human-computer interaction systems. In 3D human-computer interaction scenarios, the reliability and smoothness of adaptive cursor control still need further improvement to enable the computer to correctly understand and respond to human behavioral intentions in real time.

[0004] Currently, many studies have proposed different methods based on adaptive cursor control. However, 3D human-computer interfaces are more complex than 2D human-computer interfaces and have entirely new characteristics. Most existing cursor adaptive control technologies rely on fixed algorithms for passive adjustment by the machine and have not been adapted to the characteristics of 3D human-computer interaction interfaces. At present, no mature cursor adaptive display design scheme for 3D human-computer interfaces has been proposed. Summary of the Invention

[0005] This application provides a method for adaptive cursor display and control in virtual reality. Its technical objective is to offer a cursor adaptive display design scheme for 3D human-computer interfaces, enhancing cursor pointing accuracy. Based on interactive target prediction technology, the final target control is predicted, and cursor shape control, cursor adaptive display rendering for pre-hand interaction of controls, and cursor C / D ratio gain scaling are performed accordingly. This not only enhances pointing accuracy and efficiency but also provides immediate visual feedback to user operations, significantly improving the user experience. Furthermore, this method has high versatility and can be easily transferred to other 3D human-computer interaction pointing technologies, improving accuracy and efficiency and enhancing the user experience.

[0006] The above-mentioned technical objective of this application is achieved through the following technical solution:

[0007] A method for adaptive cursor display and control in virtual reality, comprising:

[0008] S1: Based on the interactive target point prediction method, obtain the azimuth coordinates of the predicted interactive target point;

[0009] S2: Traverse the controls within the spatial angle θ+2σ near the predicted interactive target point to obtain n controls, and encode the azimuth coordinates of each of the n controls; where θ and σ represent the average error and standard deviation of the interactive target point prediction method in spatial angle, respectively.

[0010] S3: Based on the azimuth coordinate encoding of each control, calculate the distance between each control and the predicted interactive target point, obtain the total azimuth difference between each control and the predicted interactive target point in space, obtain the total distance from all controls to the predicted interactive target point based on the total azimuth difference, obtain the probability value of each control becoming the final target control based on the total distance, and determine the maximum probability value and its corresponding control.

[0011] S4: Determine whether the maximum probability value has reached the preset threshold. If so, the control corresponding to the maximum probability value is confirmed as the final target control, and the cursor shape changes accordingly to a shape that conforms to its task semantics and is displayed. Otherwise, the cursor shape remains unchanged.

[0012] S5: Obtain the cursor C / D ratio gain based on the position of the final target control;

[0013] S6: Obtain the final display position of the cursor based on the C / D ratio gain, and perform cursor scaling control based on the final display position.

[0014] The beneficial effects of this application are as follows: The cursor adaptive display and control method in virtual reality described in this application is applicable to cursor adaptive display in bare-handed interaction. Based on interactive target prediction technology, it predicts the final target control and performs cursor shape control, cursor adaptive display rendering for bare-handed interaction, and scaling the cursor C / D ratio gain. This not only enhances pointing accuracy and efficiency but also provides real-time visual feedback to user operations, significantly improving the user experience. Furthermore, this method has high versatility and can be easily transferred to other pointing technologies for 3D human-computer interaction. Attached Figure Description

[0015] Figure 1 This is a flowchart of a method for adaptive cursor display and control in virtual reality, as described in an embodiment of this application.

[0016] Figure 2 A schematic diagram illustrating the division of the scaling space range;

[0017] Figure 3 This is a schematic diagram of the distance-based C / D ratio scaling curve;

[0018] Figure 4 This is a schematic diagram of the C / D ratio scaling curve corrected by offset. Detailed Implementation

[0019] The technical solution of this application will be described in detail below with reference to the accompanying drawings.

[0020] The development scenario of the method described in this application is a bare-hand interaction scenario in a virtual environment. Control is achieved by emitting shoulder and hand rays through a gesture tracking device. This method can also be well transferred to other pointing technologies such as hand-eye rays and controller rays.

[0021] like Figure 1 As shown, the present application describes a method for adaptive cursor display and control in virtual reality, comprising:

[0022] S1: Based on the interactive target point prediction method, the azimuth coordinates of the predicted interactive target point are obtained.

[0023] Specifically, based on interactive target point prediction technology, the directional attributes of interactive targets in the task can be obtained in advance, enabling the machine to respond better to user behavior. Since the predicted interactive target point is a point on the interactive interface and does not always fall on a specific control, it is necessary to predict the final target control in real time so that the machine can better understand the user's behavior and provide feedback.

[0024] In this embodiment, the predicted interactive target point and cursor position are both represented by azimuth angles. If the positive direction of the z-axis extension is configured as directly forward, the orientation of the hand ray or eye is represented by the horizontal and vertical azimuth angles formed by them in the x-z and y-z planes, respectively. The horizontal and vertical azimuth angles are expressed as follows:

[0025]

[0026]

[0027] in, A three-dimensional unit vector representing the spatial angular orientation of an object, expressed in radians; express x-coordinate, express y-coordinate, express The z-coordinate.

[0028] According to the horizontal azimuth angle ω horizontal and vertical azimuth ω vertical The calculation method can obtain the azimuth coordinates (ω) of the predicted interactive target point A. hA ,ω vA ), (ω hA ,ω vA Update based on each frame.

[0029] S2: Traverse the controls within the spatial angle θ+2σ near the predicted interactive target point to obtain n controls, and encode the azimuth coordinates of each of the n controls; where θ and σ represent the average error and standard deviation of the interactive target point prediction method in spatial angle, respectively.

[0030] Specifically, assuming that the average error of the interactive target point prediction method in terms of spatial angle is θ and the standard deviation is σ, and that it conforms to a normal distribution, then approximately 97.8% of the data falls within the interval of the mean plus or minus two standard deviations, i.e., within [θ-2σ, θ+2σ]. Therefore, we only need to consider the controls within the spatial angle θ+2σ near the predicted interactive target point and calculate their probability of becoming the final target control.

[0031] Encode the azimuth coordinates of each of the n controls as follows: B1(ω h1 ,ω v1 B2(ω) h2 ,ω v2 ), ..., B n (ω hn ,ω vn ).

[0032] S3: Based on the azimuth coordinate encoding of each control, calculate the distance between each control and the predicted interactive target point, obtain the total azimuth difference between each control and the predicted interactive target point in space, obtain the total distance from all controls to the predicted interactive target point based on the total azimuth difference, obtain the probability value of each control becoming the final target control based on the total distance, and determine the maximum probability value and its corresponding control.

[0033] The total azimuth difference d between each control and the predicted interactive target point in space. i Represented as:

[0034]

[0035] d hi =|ω hA -ω hi |;

[0036] d vi =|ω vA -ω vi |;

[0037] Where, d hi and d vi These represent the horizontal and vertical azimuth differences between the i-th control and the predicted interaction target point A, respectively.

[0038] All controls (B1 to B) n The total distance to the predicted interaction target point is expressed as:

[0039]

[0040] The probability value of each control becoming the final target control is expressed as follows:

[0041]

[0042] Among them, K i This represents the probability that the i-th control will become the final target control.

[0043] In this embodiment, the position of the control that obtains the highest probability value will be marked as the final target control position T, which serves as the benchmark for C / D ratio gain scaling. The interactive target point prediction technology updates the azimuth coordinates of the predicted interactive target point every frame. Therefore, the above algorithm will also be called every frame to update the probability value of each control becoming the final target control.

[0044] S4: Determine whether the maximum probability value has reached the preset threshold. If so, the control corresponding to the maximum probability value is confirmed as the final target control, and the cursor shape changes accordingly to a shape that conforms to its task semantics and is displayed. Otherwise, the cursor shape remains unchanged.

[0045] In this embodiment, the control can be pre-rendered based on the obtained probability value. When the probability value reaches a certain value, the edge of the control will glow, and its glow intensity will gradually increase as the probability value increases. There may be multiple glowing controls at the same time, but when the probability value of a certain control reaches a preset threshold, the target control will be identified as the final target control, and its edge light will reach its brightest point. Other controls will stop glowing, and the cursor shape will change accordingly to a cursor shape that conforms to its task semantics. For example, if the target control is a text box, the cursor shape will change to an I-shaped pointer. Refer to Table 1 for details on how the cursor shape changes.

[0046] Table 1

[0047]

[0048]

[0049] As shown in Table 1, cursor shapes include standard cursor, copy cursor, loading cursor, crosshair cursor, hand cursor, zoom-in cursor, zoom-out cursor, stretch cursor, and T-shaped pointer. The standard cursor indicates the selection of content and interface elements and interaction with them; the copy cursor indicates the copying of content and interface elements; the loading cursor indicates that the interactive element is in a responsive state; and the crosshair cursor indicates that the interactive element can be precisely selected.

[0050] The hand cursor includes a clickable hand cursor, a draggable hand cursor, and a draggable hand cursor. The clickable hand cursor indicates that the interactive element is clickable, the draggable hand cursor indicates that the interactive element is draggable, and the draggable hand cursor indicates that the interactive element is being dragged.

[0051] The zoom-in cursor indicates zooming in on the current interactive view, and the zoom-out cursor indicates zooming out on the current interactive view.

[0052] The stretch cursor includes a vertical stretch cursor, a horizontal stretch cursor, and a diagonal stretch cursor. The vertical stretch cursor indicates adjustment of the vertical dimension of the current interactive element, the horizontal stretch cursor indicates adjustment of the horizontal dimension of the current interactive element, and the diagonal stretch cursor indicates adjustment of the diagonal dimension of the current interactive element. The diagonal stretch cursor includes the two diagonal stretch forms shown in Table 1.

[0053] The "I" pointer indicates the selection and insertion of text in the interface.

[0054] S5: Obtain the cursor C / D ratio gain based on the position of the final target control.

[0055] C / D gain is defined as the ratio of the distance the displayed content moves to the distance the controller moves, and it is an important parameter affecting the quality of human-computer interaction. Based on the predicted final target control, this part aims to adaptively adjust the C / D gain in real time, so that the cursor slows down when it approaches the final target control, thereby improving pointing accuracy and efficiency.

[0056] Specifically, step S5 includes:

[0057] S51: Get the current cursor position P disp The distance between the final target control and its position T is expressed as:

[0058] D = TP disp ;

[0059] S52: Normalize D to the 0-1 range to obtain the adaptive scaling function of D, which is expressed as:

[0060]

[0061] Among them, D min D represents the minimum scaling threshold. max This represents the maximum scaling threshold; when |D|≤D min When, it indicates that it is within a fixed scaling area; when D min <|D| <D max When |D|≥D, it indicates that it is within the adaptive scaling region; max When the value is 0, it indicates that the area is within the no-scaling region.

[0062] The division into no-scaling, adaptive-scaling, and fixed-scaling regions is based on the position of the final target control and the cursor, with independent spatial divisions made horizontally and vertically. Both the horizontal and vertical spaces are divided into three areas: no-scaling, adaptive-scaling, and fixed-scaling. The distances of these three areas from the final target control decrease sequentially. Figure 2 As shown.

[0063] A region without scaling indicates that the cursor is far from the final target control within that region, and the cursor should be in a fast-moving phase, where the C / D ratio gain is always equal to 1.

[0064] Within the adaptive scaling region, the cursor position gradually approaches the final target control. Therefore, this stage provides adaptive non-linear scaling based on the distance between the controller and the target to prevent the user from perceiving a sudden change in the controller's C / D ratio gain, thus completing the transition from the no-acceleration region to the fixed scaling region and reducing the user's perception of changes in the C / D ratio gain.

[0065] Within the fixed scaling area, the cursor position is now very close to the final target control. To assist the user in pointing, the C / D ratio gain is set to a fixed value during this stage to improve pointing accuracy and ensure that the C / D ratio gain does not change abruptly, affecting the user experience.

[0066] S53: The cursor C / D ratio gain is calculated using an adaptive scaling function, expressed as:

[0067]

[0068] Where g represents the cursor C / D ratio gain, g min G represents the minimum value of the C / D ratio gain. max This represents the maximum value of the C / D ratio gain.

[0069] The adaptive scaling function described above is a sine function that has been scaled and translated, and its minimum value is g. min The maximum value is g max The entire function is continuous and its curvature is continuous, ensuring that the continuous transformation of the C / D ratio gain will not be noticeably perceived by the user. Figure 3 As shown.

[0070] In this embodiment, since the space is divided independently in the horizontal and vertical azimuth angles, the C / D ratio gain is also obtained through the single-dimensional motion state of the azimuth angle, and the subsequent scaling is also completed independently in both directions. The above-mentioned gain algorithm is based on the principle of scaling in a single direction, and the method and parameters are applicable to both the vertical and horizontal azimuth angles.

[0071] S6: Obtain the final display position of the cursor based on the C / D ratio gain, and complete the cursor movement control based on the final display position.

[0072] After obtaining the C / D ratio gain g, the cursor's display position is then assigned a value, and the final display position of the cursor is represented as:

[0073] P disp (j)=P disp (j-1)+g·ΔP(j);

[0074] ΔP(j)=P(j)-P(j-1);

[0075] Where P(j) represents the cursor position in frame j, and P(j-1) represents the cursor position in frame j-1; P disp (j-1) represents the final display position of the cursor in frame j-1. disp (j) indicates the final display position of the cursor in frame j.

[0076] In this embodiment of the application, the threshold parameters used during the scaling process are shown in Table 2:

[0077] Table 2 Thresholds used in scaling functions

[0078] <![CDATA[g min ]]> <![CDATA[g max ]]> <![CDATA[D min ]]> <![CDATA[D max ]]> 0.5 1 1 / π 8 / π

[0079] Preferably, to eliminate the loss of control sensation caused by the deviation between the actual hand position and the display position, this application's method incorporates an offset correction mechanism. When the user is in unintentional movement (i.e., the interaction target point prediction technology does not predict the interaction target point output) and an offset exists, the offset correction mechanism will be triggered. The offset correction also employs the method of changing the C / D ratio gain to reduce the offset in real time without affecting performance or being perceptible to the user.

[0080] Offset O is also at ω horizontal ω vertical The cursor is used to independently represent the position P in both dimensions. disp The difference between the actual position P of the controller and the actual position P is calculated and expressed as:

[0081] O = PP disp ;

[0082] Normalizing the offset 0 to the 0-1 range is represented as:

[0083]

[0084] Among them, O min Indicates the minimum offset threshold; O max This represents the maximum offset threshold.

[0085] The C / D ratio gain is corrected by offsetting, resulting in the corrected C / D ratio gain as follows: Figure 4 As shown, the corrected C / D ratio gain is expressed as:

[0086]

[0087] By determining whether the motion ΔP(j) of the current frame is in the same direction as the offset, the direction coefficient v is obtained, which is used to adjust the direction of the C / D gain. The direction coefficient v is expressed as:

[0088]

[0089] The direction of the corrected C / D ratio gain g is adjusted using the direction coefficient v, and the adjusted C / D ratio gain g' is calculated. When the frame motion is conducive to offset elimination, g' is made greater than 0 to accelerate and reduce offset; when the frame motion is not conducive to offset elimination, g' is made less than 0 to decelerate and reduce offset. Therefore, the adjusted C / D ratio gain g' is expressed as:

[0090] g' = 1 + v·(1-g)

[0091] After obtaining the C / D ratio gain g' after adjusting the direction, the final display position of the cursor is obtained according to step S6.

[0092] The threshold parameters used in the offset correction process are shown in Table 3:

[0093] Table 3. Threshold parameters used for offset correction

[0094] <![CDATA[g min ]]> <![CDATA[g max ]]> <![CDATA[O min ]]> <![CDATA[O max ]]> 0.5 1 0.5 / π 4 / π

[0095] In this embodiment of the application, in order to reduce unnecessary scaling and reduce the impact of erroneous prediction, the method of this application incorporates a scaling state exit mechanism. When any of the following conditions are met, it is considered that the user has interrupted or completed the current task, and the current scaling process is exited, and the C / D ratio gain is adjusted back to 1.

[0096] (a) The distance between the current cursor position and the predicted interactive target point is greater than 12° on any axis: An error greater than 12° may indicate that the target prediction model has failed to predict, or it may be the result of the user adjusting the motion state.

[0097] (b) Movement duration exceeds 5s: If the movement duration exceeds 5s, it is generally not considered to be a movement with a clear interactive intention, in order to avoid system misjudgment.

[0098] (c) The user has completed an interactive task (such as a click or swipe): The user's action has ended, and the interaction target point should be predicted again.

[0099] The above are exemplary embodiments of this application, and the scope of protection of this application is defined by the claims and their equivalents.

Claims

1. A method for adaptive display and control of a cursor in virtual reality, characterized by, Comprise: S1: based on the interaction target point prediction method, the azimuth coordinate of the predicted interaction target point is obtained; S2: Predicting spatial angles near the target interaction point Iterate through the controls within to get n Each control, for n Each control is encoded with its azimuth coordinates; among them... and These represent the average error and standard deviation of the interactive target point prediction method in terms of spatial angle, respectively. S3: according to the azimuth coordinate coding of each control, the distance between each control and the predicted interaction target point is calculated, the total azimuth difference value of each control and the predicted interaction target point in space angle is obtained, the total distance of all controls to the predicted interaction target point is obtained according to the total azimuth difference value, the probability value of each control becoming the final target control is obtained according to the total distance, and the maximum probability value and its corresponding control are determined; S4: whether the maximum probability value reaches the preset threshold is judged, if yes, the control corresponding to the maximum probability value is confirmed as the final target control, the cursor form is changed to the form corresponding to its task semantics and displayed, if not, the cursor form remains unchanged; S5: the C / D ratio gain of the cursor is obtained according to the position of the final target control; S6: the final display position of the cursor is obtained according to the C / D ratio gain, and the zoom control of the cursor is completed according to the final display position.

2. The method of claim 1, wherein, The method further comprises: when there is no output of the predicted interaction target point, and there is an offset, the offset correction mechanism is triggered, the C / D ratio gain obtained by step S5 will be corrected by the offset to obtain the corrected C / D ratio gain; whether the motion of the current frame is in the same direction as the offset direction is judged, and the direction coefficient is obtained; the direction of the corrected C / D ratio gain is adjusted according to the direction coefficient, and the adjusted direction C / D ratio gain is obtained, and the final display position of the cursor is obtained according to the adjusted direction C / D ratio gain in step S6; wherein the offset represents the difference between the final display position of the cursor and the actual position of the controller in the azimuth angle.

3. The method of claim 1, wherein, In step S6, the zoom control of the cursor includes a zoom exit mechanism, that is, the current zoom process is exited, and the C / D ratio gain is adjusted back to 1, and the triggering conditions of the zoom exit mechanism include: (a) the distance between the current position of the cursor and the predicted interaction target point is greater than 12 degrees in any axis; (b) the duration of cursor motion is more than 5s; (c) the user has completed an interaction task.

4. The method according to any one of claims 1 to 3, characterized in that, In the step S1, the interaction target point is predicted the azimuth coordinate is expressed as: ； ； wherein, The positive direction of the axis extension is configured as the positive front direction; represents a predicted interaction target point in a horizontal azimuth angle formed by the plane, represents a predicted interaction target point in a vertical azimuth angle formed by the plane; represents a three-dimensional unit vector of the object spatial angle orientation, expressed in radian value; represents coordinates of represents coordinates of represents coordinates of .​​ 5. The method of claim 4, wherein, In the step S3, the total azimuth difference in the spatial angle between each control and the predicted interaction target point is represented as: ； ； ； wherein, and respectively represent the horizontal azimuth angle difference value and the vertical azimuth angle difference value of the first control and the predicted interaction target point; represent the horizontal azimuth angle of the first control, represent the vertical azimuth angle of the first control;​ The total distance of all controls to the predicted interaction target point is represented as: ； The probability value of each control becoming the final target control is represented as: ； wherein, represents the probability value that the i-th control becomes the final target control. represents the probability value that the i-th control becomes the final target control.

6. The method of claim 5, wherein, The cursor form in the step S4 includes a standard cursor, a copy cursor, a load cursor, a cross cursor, a hand cursor, an enlarge cursor, a reduce cursor, a stretch cursor and an I-beam pointer, the standard cursor represents selecting and interacting with the content and interface elements, the copy cursor represents copying the content and interface elements, the load cursor represents that the interactive element is in a responding state, the cross cursor represents that the interactive element can be precisely framed, the hand cursor includes a click hand cursor, a dragable hand cursor and a dragging hand cursor, the click hand cursor represents that the interactive element is in a clickable state, the dragable hand cursor represents that the interactive element is in a dragable state, the dragging hand cursor represents that the interactive element is in a being dragged state, the enlarge cursor represents zooming in the current interactive view, the reduce cursor represents zooming out the current interactive view, the stretch cursor includes a vertical stretch cursor, a horizontal stretch cursor and a diagonal stretch cursor, the vertical stretch cursor represents adjusting the vertical direction size of the current interactive element, the horizontal stretch cursor represents adjusting the horizontal direction size of the current interactive element, the diagonal stretch cursor represents adjusting the diagonal direction size of the current interactive element, and the I-beam pointer represents selecting and inserting text in the interface.

7. The method of claim 6, wherein, The step S5 includes: S51: Obtain the current display position of the cursor the distance between the position of the final target control and the position of the cursor, denoted as: ； S52: scaling to the 0-1 interval to obtain an adaptive scaling function, denoted as: ； wherein, represents a minimum scaling threshold, represents a maximum scaling threshold; when represents being in a fixed scaling region; when represents being in an adaptive scaling region; when represents being in a no scaling region; S53: calculating the cursor C / D ratio gain by an adaptive scaling function, expressed as: ； wherein, represents the cursor C / D ratio gain, represents the minimum value of the C / D ratio gain, represents the maximum value of the C / D ratio gain.

8. The method of claim 7, wherein, The final display position of the cursor in the step S6 is expressed as: ； ； in, Indicates the cursor is at the 1st position. Frame position, Indicates the cursor is at the 1st position. Frame position; Indicates the cursor is at the 1st position. The final display position of the frame. Indicates the cursor is at the 1st position. The final display position of the frame.

9. The method of claim 8, wherein, The corrected C / D ratio gain is expressed as: ； wherein, denotes the offset normalized to the 0-1 interval, denoted as: ； represents a minimum offset threshold value; represents a maximum offset threshold value; represents an offset, represents a final display position of the cursor; represents an actual position of the controller; Directional coefficients is expressed as: ； C / D ratio gain after adjustment of direction is represented as: .

10. The method of claim 9, wherein, When the C / D ratio gain is calculated by an adaptive scaling function, , , , When the C / D ratio gain is corrected by an offset, , , , .

Images