A method and apparatus for dynamically adjusting an AR user interface

Abstract

This application discloses a method and apparatus for dynamically adjusting an AR user interface, relating to the field of augmented reality technology. The method includes: extracting features from real-time detected user head motion data to obtain a head motion state; the head motion state includes head motion rate and head motion direction; switching the interface mode of the AR user interface based on the head motion rate, with different interface modes corresponding to different sizes of the AR user interface; and determining the motion of the AR user interface based on the head motion direction and head motion rate. This solves the problem that existing AR user interfaces struggle to simultaneously accommodate external environment observation and AR information reading in highly dynamic task environments. It can adjust the interface mode and motion of the AR user interface according to the intensity and direction of the user's head motion, reducing interface occlusion. It provides a wider field of view during user head movement while simultaneously allowing the retrieval of key information from the AR user interface.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and in particular to a method and apparatus for dynamically adjusting an AR user interface. Background Technology

[0002] With the development of augmented reality (AR) technology, deploying drone ground station software onto AR glasses has become an emerging trend. This deployment method can overlay computer-generated virtual information onto the real-world scene, bringing users a more intuitive and richer interactive experience.

[0003] In highly dynamic mission environments (such as emergency drone control and fire rescue), users need to frequently and rapidly turn their heads to continuously observe the surrounding environment and the drone's status. However, in such situations, a single AR user interface display method may obstruct the user's field of vision or result in AR information overlay, making it difficult for the user to simultaneously observe the external environment and read key information. Summary of the Invention

[0004] This application provides a method and apparatus for dynamically adjusting an AR user interface, which solves the problem that existing AR user interfaces struggle to simultaneously observe the external environment and read AR information in highly dynamic task environments.

[0005] In a first aspect, embodiments of this application provide a method for dynamically adjusting an AR user interface, comprising: extracting features from real-time detected user head motion data to obtain a head motion state; the head motion state includes a head motion rate and a head motion direction; switching the interface mode of the AR user interface based on the head motion rate, wherein different interface modes correspond to different sizes of the AR user interface; and determining the motion of the AR user interface based on the head motion direction and the head motion rate.

[0006] In conjunction with the first aspect, in one possible implementation, the switching of the AR user interface mode based on the head movement rate includes: when the head movement rate does not exceed a lower threshold, the interface mode remains in standard mode; in standard mode, the AR user interface is not scaled and displays all interface elements; when the head movement rate is between the lower and upper thresholds, the interface mode is switched to compact mode; in compact mode, the AR user interface is scaled at a moderate ratio and displays important interface elements; when the head movement rate reaches the upper threshold, the interface mode is switched to minimalist mode; in minimalist mode, the AR user interface is scaled at the minimum ratio and only displays key interface elements.

[0007] In conjunction with the first aspect, in one possible implementation, determining the movement of the AR user interface based on the head movement direction and the head movement rate includes: when the head movement rate does not exceed a lower threshold, the relative position of the AR user interface in the ambient field of view remains unchanged; when the head movement rate exceeds the lower threshold, the AR user interface moves in the opposite direction of the head movement direction to a preset position, and then reverses back to the initial relative position in the ambient field of view.

[0008] In conjunction with the first aspect, in one possible implementation, the step of moving the AR user interface to a preset position in the opposite direction of the head movement direction when the head movement rate exceeds a lower threshold includes: the movement rate of the AR user interface is positively correlated with the head movement rate.

[0009] In conjunction with the first aspect, in one possible implementation, determining the motion of the AR user interface based on the head movement direction and the head movement rate includes: abstracting the AR user interface as a point mass in a spring-damped-mass model, and determining the actual position of the current AR user interface by solving the dynamic equations of the spring-damped-mass model; wherein the damping coefficient of the spring-damped-mass model is dynamically determined by the head movement state; determining the current interface elements according to the current interface mode, and rendering the AR user interface in conjunction with the actual position; if the current head movement data triggers a recovery condition, then restoring the AR user interface to the target position by reducing the damping coefficient.

[0010] In conjunction with the first aspect, in one possible implementation, the abstraction of the AR user interface as a point mass in a spring-damped-mass model includes: abstracting the AR user interface as a point mass, connecting the point mass to the target position through a spring and a damper respectively, to obtain a spring-damped-mass model; wherein the target position is determined in real time based on the user's head movement data.

[0011] In conjunction with the first aspect, in one possible implementation, after determining the actual position of the current AR user interface by solving the dynamic equations of the spring-damping-mass model, the method further includes: if the deviation between the actual position of the current AR user interface and the target position exceeds a preset offset, then the actual position of the current AR user interface is forced to stop moving until the user's head movement stops, and then the actual position of the current AR user interface is controlled to move towards the target position; if the deviation between the actual position of the current AR user interface and the target position does not exceed the preset offset, then the actual position of the current AR user interface is updated through the dynamic equations.

[0012] In conjunction with the first aspect, in one possible implementation, the reverse recovery to the initial relative position in the environmental field of view is triggered based on recovery conditions; the recovery conditions include: the instantaneous motion rate does not exceed a lower threshold, the motion acceleration is less than a preset acceleration threshold, and the motion angle is less than a preset angle threshold, and the duration exceeds a first time.

[0013] In conjunction with the first aspect, in one possible implementation, restoring the AR user interface to the target position by reducing the damping coefficient includes: gradually reducing the damping coefficient to the level of the base damping coefficient, so that the AR user interface moves toward the target position under the tension of the spring; and locking the position of the AR user interface when the real-time position of the AR user interface coincides with the target position.

[0014] Secondly, embodiments of this application provide a dynamic adjustment device for an AR user interface, comprising: a feature extraction module for extracting features from real-time detected user head motion data to obtain a head motion state; the head motion state includes a head motion rate and a head motion direction; a switching module for switching the interface mode of the AR user interface based on the head motion rate, wherein different interface modes correspond to different sizes of the AR user interface; and a determination module for determining the motion of the AR user interface based on the head motion direction and the head motion rate.

[0015] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0016] This application's embodiments, by extracting head motion states, can quantify the intensity of user head movement and characterize motion patterns, making interface mode switching and latency control clearer and more efficient. By switching interface modes, it can dynamically balance interface information and the user's field of vision based on the user's motion intensity. This effectively solves the problem that existing AR user interfaces struggle to simultaneously consider external environment observation and AR information reading in highly dynamic task environments. Furthermore, it can adjust the AR user interface mode and motion based on the intensity and direction of the user's head movement, reducing interface occlusion, providing a wider environmental field of vision during user head movement, and simultaneously extracting key information from the AR user interface. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1A flowchart illustrating a method for dynamically adjusting an AR user interface provided in an embodiment of this application;

[0019] Figure 2 An example diagram illustrating the changes in the field of view and the ambient field of view of the ground station software in the AR user interface during slow head rotation, as provided in this embodiment of the application.

[0020] Figure 3 An example diagram illustrating the changes in the field of view and the environmental field of view of the ground station software in the AR user interface during medium-speed head rotation, as provided in this embodiment of the application.

[0021] Figure 4 An example diagram illustrating the changes in the field of view and the environmental field of view of the ground station software in the AR user interface during high-speed head rotation, as provided in this application embodiment.

[0022] Figure 5 Example diagram of the spring-damping-mass model provided in the embodiments of this application;

[0023] Figure 6 The dynamic damping coefficient provided for the embodiments of this application A graph showing the change in instantaneous velocity V.

[0024] Figure 7 An example diagram illustrating the process of the AR user interface following the head movement when the user's head turns to the right, as provided in this embodiment of the application.

[0025] Figure 8 This is a schematic diagram of the structure of a dynamic adjustment device for an AR user interface provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] The following description of some technologies involved in the embodiments of this application is provided to aid understanding and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, some descriptions of well-known functions and structures are omitted in the following description.

[0028] Traditional AR user interfaces generally lack dynamic layout optimization mechanisms based on user movement states. When the user's head moves at high speed, existing technologies cannot adaptively adjust the arrangement, density, and following delay of interface elements according to the rate and direction of change of the user's posture, which can easily lead to information overlap, difficulty in recognition, and affect the accuracy and real-time performance of command issuance.

[0029] For example, a patent application filed by Google LLC (publication number CN118318219A) projects an AR user interface onto the surface of a head-mounted display device and moves the projected AR user interface on the surface in response to the movement of the head-mounted display device. While this technology involves the relationship between head movement and the AR user interface display, its core principle is to compensate for device movement by physically shifting the projected interface at the lower level to achieve image stabilization. It does not involve the adaptive dynamic layout of the upper-level AR user interface. The AR user interface is moved as a rigid unit, and its internal layout logic is static and fixed. Even when the user quickly turns their head, the stable AR user interface will still obstruct the current field of view as a whole.

[0030] Figure 1 This is a flowchart of a method for dynamically adjusting an AR user interface provided in an embodiment of this application, including steps 101 to 103. Figure 1 This is merely one execution order shown in the embodiments of this application, and does not represent the only execution order of a method for dynamically adjusting an AR user interface. Where the final result can be achieved, Figure 1 The steps shown can be performed in parallel or in reverse order.

[0031] Step 101: Extract features from the real-time detected user head motion data to obtain the head motion state. The head motion state includes head motion rate and head motion direction.

[0032] Specifically, head motion data is detected in real time by an inertial measurement unit (IMU) to measure the user's head movement rate (i.e., angular velocity) and direction (i.e., angle). This head motion data is captured by a set of miniature sensors (i.e., the inertial measurement unit) integrated into the AR headset (AR glasses). The inertial measurement unit typically includes a three-axis gyroscope and a three-axis accelerometer. The gyroscope measures the angular velocity of the AR glasses' rotation around the X, Y, and Z axes, while the accelerometer measures the linear acceleration of the AR glasses along the X, Y, and Z axes. The inertial measurement unit samples at a high frequency of 1000Hz, with 1000 data points read per second, enabling the capture of extremely subtle and rapid head movements.

[0033] In this embodiment, the head motion data can also be defined as a state vector. The state vector is then denoised and smoothed using a Kalman filter algorithm. Feature extraction is performed on the head motion data to obtain the head motion state. The head motion state includes instantaneous motion rate, average motion rate, motion direction vector, and motion acceleration.

[0034] Specifically, step 101 is the perceptual foundation of the entire method of this application, and its core task is to accurately and in real-time acquire the user's head motion data. Subsequent steps all depend on the accuracy and reliability of the head motion data or head motion state output by step 101. The raw head motion data acquired by the inertial measurement unit contains high-frequency electronic noise, temperature drift, or integration errors. This application uses the Kalman filter algorithm to denoise and smooth it. The Kalman filter algorithm effectively eliminates noise interference and obtains smooth and accurate head motion data by fusing the erroneous predicted and measured values ​​and recursively updating the state estimate.

[0035] Furthermore, the head motion data is defined as a 9-dimensional state vector X, which contains the user's head motion angle, angular velocity, and angular acceleration. The motion angle is obtained by integrating the angular velocity, and the angular acceleration is the time derivative of the angular velocity, or the second time derivative of the motion angle.

[0036] For example, the state vector is as follows:

[0037] ,in, , , .

[0038] Where X represents the state vector. Indicates the angle of motion. , , These represent the rotation angles (in degrees) of the user's head around the X, Y, and Z coordinate axes, respectively. Indicates angular velocity. , , These represent the angular velocities (unit: ° / s, i.e., degrees per second) of the user's head rotation around the X, Y, and Z coordinate axes, respectively. Represents angular acceleration. , , These represent the angular accelerations of the user's head rotating around the X, Y, and Z coordinate axes (unit: ° / s). 2 , which is degrees per second squared), T represents the transpose matrix.

[0039] The state equations and observation equations are constructed as follows:

[0040] ,

[0041] .

[0042] In the formula, Let A represent the state vector at time point k, and let B represent the state transition matrix, which maps the state vector at time point k-1 to the state vector at time point k, reflecting the physical relationship between motion angle, angular velocity, and angular acceleration. k This represents external control, which is usually absent in scenarios where the head moves freely, hence B. u k w is 0 k The process noise at time point k follows a normal distribution N(0,Q), where Q is 9. A 9-diagonal covariance matrix, where the diagonal elements are determined through tuning, dictates whether to place more trust in the predicted or measured values. Z k H represents the observation vector at time point k, and H represents the observation matrix, which is a 6-dimensional vector. A 9-dimensional matrix is ​​used to select six dimensions from the 9-dimensional state vector that can directly match the readings (angular velocity and linear acceleration) of the gyroscope and accelerometer. k The observation noise represents the measurement error of the sensor (i.e., the inertial measurement unit), and follows a multivariate normal distribution with a mean of 0 and a covariance matrix of R. R is used to describe the accuracy of the sensor and is obtained from the sensor manual.

[0043] Based on the aforementioned state vector, the constructed state equation, and the observation equation, the Kalman filter algorithm is executed to obtain a smooth and reliable state estimate. Feature extraction is then performed on this estimate to obtain the head motion state.

[0044] Specifically, instantaneous velocity of motion: ,in, , , These represent the angular velocities of the user's head rotating around the X, Y, and Z coordinate axes, respectively. V represents the instantaneous motion rate, which is the magnitude of the angular velocity vector. It quantifies the total speed of the user's head rotation without regard to direction, and the unit is ° / s.

[0045] Average speed of movement: ,in, Let N represent the instantaneous velocity at time point i, and let N represent the number of instantaneous velocities. The average motion rate is calculated by averaging the instantaneous motion rates over the past N time points. It is used to smooth out short-term fluctuations in instantaneous motion rates, prevent interface mode switching from being triggered by a noise spike or a slight jitter, and improve the stability of decision-making.

[0046] Motion direction vector: ,in, , , These represent the angular velocities of the user's head rotating around the X, Y, and Z coordinate axes, respectively, while V represents the instantaneous velocity. , , These represent the direction vectors of the angular velocity along the X, Y, and Z coordinate axes, respectively, while d represents the motion direction vector. It normalizes the angular velocity vector, retaining only the direction information and eliminating the influence of magnitude. This is used to indicate which coordinate axis or combination of directions the user's head rotates around. For example, (1,0,0) indicates that the user's head rotates purely around the X coordinate axis (pitch), while (0,0.707,0.707) indicates that the user's head rotates around a combination of the Y and Z coordinate axes.

[0047] Motion acceleration: ,in, This represents the instantaneous velocity at time point k. t represents angular velocity, and t represents time. The time interval is consistent with the sampling frequency of the inertial measurement unit, which is 1 ms. 'a' represents the motion acceleration, which is the rate of change of instantaneous motion speed, quantifying whether the user's head rotation is accelerating or decelerating. A high motion acceleration value indicates a rapid change in motion state and can be used to predict motion trends.

[0048] Step 102: Switch the AR user interface mode based on head movement rate. Different interface modes correspond to different sizes of the AR user interface. In this embodiment, when the head movement rate does not exceed the lower threshold, the interface mode remains in standard mode. In standard mode, the AR user interface is not scaled and displays all interface elements. When the head movement rate is between the lower and upper thresholds, the interface mode is switched to compact mode. In compact mode, the AR user interface is scaled at a moderate ratio to display important interface elements. When the head movement rate reaches the upper threshold, the interface mode is switched to minimalist mode. In minimalist mode, the AR user interface is scaled at the minimum ratio, displaying only key interface elements.

[0049] For example, the lower threshold is set to 30° / s and the upper threshold is set to 90° / s.

[0050] Specifically, when the instantaneous motion speed is less than or equal to 30° / s, the interface mode switches to standard mode. For example... Figure 2 As shown, in standard mode, the AR user interface has no scaling and displays all interface elements. In standard mode, the user is relatively stationary or in low-speed motion, and has a high demand for comprehensive information from all interface elements; displaying all elements satisfies the user's normal information browsing and interactive needs. When the instantaneous movement speed is between 30-90° / s, the interface mode switches to compact mode. For example... Figure 3 As shown, in compact mode, the AR user interface is scaled down by a moderate ratio, displaying only important interface elements, such as ground commands for drone takeoff, landing, and return, as well as task planning buttons like "Create New Route," "Start Task," and "Pause Task." When the user's head moves at a certain speed, shrinking the AR user interface and highlighting important interface elements ensures that the user can quickly obtain important information while reducing the obstruction of the field of view by interface elements. The moderate ratio is exemplarily set to 0.5, meaning the area of ​​the moderately scaled AR user interface is half of the AR field of view. When the instantaneous movement speed is greater than or equal to 90° / s, the interface mode switches to minimalist mode. Figure 4 As shown, in minimalist mode, the AR user interface is scaled down to a minimum, displaying only key interface elements, such as the "land" and "return" buttons, providing a maximized field of view. The minimum scale is exemplarily set to 0.2, meaning that in minimalist mode, the scaled AR user interface area is 1 / 5 of the AR field of view. When the user's head is moving rapidly, minimalist mode minimizes the interference of interface elements on the user's field of vision, allowing the user to focus on their surroundings and improving environmental awareness.

[0051] In addition, to prevent high-frequency oscillations in the interface mode, a switching delay can be set. The interface mode will only switch after the instantaneous motion rate state lasts for 1 second. For example, if the instantaneous motion rate of the user's head is less than 30° / s for 1 second, the interface will switch to the standard mode; otherwise, the existing interface mode will be maintained.

[0052] Step 103: Determine the movement of the AR user interface based on the head movement direction and head movement rate. In this embodiment, when the head movement rate does not exceed the lower threshold, the relative position of the AR user interface in the ambient field of view remains unchanged. When the head movement rate exceeds the lower threshold, the AR user interface moves in the opposite direction of the head movement direction to a preset position, and then reverses back to the initial relative position in the ambient field of view.

[0053] Among them, the movement rate of the AR user interface is positively correlated with the head movement rate.

[0054] In this embodiment, the AR user interface is abstracted as a point mass in a spring-damped-mass model, and the actual position of the current AR user interface is determined by solving the dynamic equations of the spring-damped-mass model. The damping coefficient of the spring-damped-mass model is dynamically determined by the head movement state. The current interface elements are determined based on the current interface mode, and the AR user interface is rendered in conjunction with the actual position. If the current head movement data triggers a recovery condition, the AR user interface is restored to the target position by reducing the damping coefficient.

[0055] Specifically, the AR user interface is abstracted as a point mass, and a spring and a damper are used to connect the point mass to the target position, respectively, resulting in a spring-damped-mass model. The target position is determined in real time based on the user's head movement data.

[0056] Furthermore, when constructing the spring-damping-mass model, the interface elements are determined based on the current interface mode. The AR user interface is abstracted as a point mass, and the point mass is connected to the desired head position (i.e., the target position) through springs and dampers. The target position is the ideal position where the interface element should be, and this target position is adjusted in real time according to changes in the user's head movement data. Figure 5 As shown in the figure, m represents the mass of the particle, C represents the damping coefficient of the damper, and K represents the spring constant.

[0057] Furthermore, the spring generates a restoring force (tension) to attempt to pull the particle towards the target position. In other words, the spring can simulate the tendency of an AR user interface to return to the center. The damper, on the other hand, is used to simulate resistance factors such as friction in the real world. Its function is to impede the motion of the particle, making its motion smoother and more stable, and avoiding excessive oscillations. That is, the damper can simulate the resistance of the AR user interface lagging behind.

[0058] By solving the dynamic equations of the spring-damped-mass model, the actual position of the particle at the current moment can be obtained. The dynamic equations comprehensively consider factors such as the spring constant, the damping coefficient of the damper, and the mass of the particle. For example, the dynamic equations are as follows:

[0059] ,

[0060] In the formula, The mass of a point mass is a scalar parameter used to control the inertial effect of the AR user interface. The larger the value of m, the slower the AR user interface starts and stops, and the more sluggish the motion feels. The unit is any relative unit. This represents the actual position of the corresponding particle, a vector indicating the current actual position of the AR user interface, with units consistent with Pt, where t represents time in seconds. C represents the damping coefficient of the damper, a scalar parameter that determines the speed at which the AR user interface resists the user's head movement; a larger C value results in a slower AR user interface response, measured in mass / time. K represents the spring constant, a scalar parameter that controls the force pulling the AR user interface towards the target position (i.e., the value of K); a larger K value results in a faster and tighter rebound of the AR user interface to the target position, measured in mass / time. 2 , This represents the target position of the corresponding particle, a vector calculated based on the user's head motion data. Theoretically, it should be located at the center of the AR user interface, following head movements without any delay. The unit can be pixel coordinates or angles. d / dt represents the differential operator, indicating the derivative with respect to time. dPr / dt represents the interface motion velocity vector. 2 Pr / dt 2 This represents the interface motion acceleration vector.

[0061] In this embodiment, to address the issue of interface occlusion, a dynamic damping coefficient-based interface delay following method is proposed. The core of this method is to dynamically adjust the damping coefficient of the interface following by real-time detection of the instantaneous movement rate of the user's head. When the user's head moves rapidly, the damping coefficient is increased, causing the AR user interface to produce a controllable hysteresis displacement opposite to the direction of head movement. Simultaneously, based on the instantaneous movement rate of the user's head, the interface switches to a standard mode, a compact mode, or a minimalist mode to give the user's view of the environment. When the user's head movement slows down or stops, the damping coefficient is decreased, allowing the AR user interface to smoothly return to the target position.

[0062] Furthermore, the damping coefficient in this application is not a constant, but a dynamic variable based on the instantaneous motion rate of the user's head, denoted as the dynamic damping model, as follows:

[0063] ,

[0064] In the formula, C0 represents the dynamic damping coefficient, i.e., the final damping value; C0 represents the base damping coefficient, i.e., the damping coefficient when the user's head moves at low speed, which ensures that the AR user interface has good tracking performance under normal operation; C1 represents the damping adjustment range, used to control the maximum range to which the damping coefficient can be increased. This indicates the adjustment of the curve steepness parameter, used to control the function. The steepness of the transition near V0, The larger the value, the faster the transition. This represents the instantaneous head movement rate of the user, in ° / s. V0 represents the motion rate adjustment reference point, which is a preset angular velocity threshold. In this application, it is exemplarily set to 30-60° / s. The motion rate adjustment reference point acts as a critical point, filtering out minor daily jitters while intervening promptly during head turns at medium to high speeds. The damping coefficient begins to increase significantly near the motion rate adjustment reference point V0. tanh() represents the hyperbolic tangent function, a smooth, S-shaped saturation function that ensures a smooth, non-abrupt transition of the damping coefficient with the instantaneous motion rate, preventing interface jumps and guaranteeing a smooth user experience. Figure 6 As shown, this represents the dynamic damping coefficient. A graph showing the change in instantaneous velocity V.

[0065] The current dynamic damping coefficient is substituted into the dynamic equation of the spring-damped-mass model to obtain the actual position of the current AR user interface.

[0066] When the user's head rotates rapidly, the target position changes quickly. Due to inertia, the mass in the spring-damped-mass model does not immediately reach the current target position. Instead, under the combined action of the spring's tension and the damper's resistance, it gradually moves closer to the target position. This dynamic adjustment process makes the movement of the AR user interface more natural and smooth, without any abrupt jumps, thus ensuring the stability and usability of the AR user interface under different head movement states.

[0067] In this embodiment, after determining the actual position of the current interface element, the following can be further performed: If the deviation between the actual position of the current AR user interface and the target position exceeds a preset offset, the actual position of the current AR user interface is forced to stop moving until the user's head movement stops, and then the actual position of the current AR user interface is controlled to move towards the target position. If the deviation between the actual position of the current AR user interface and the target position does not exceed the preset offset, the actual position of the current AR user interface is updated through a dynamic equation. The interface offset is determined based on the distance between the AR user interface and the screen edge; when the AR user interface coincides with the screen edge, it is considered to exceed the preset offset, and the movement stops.

[0068] Specifically, the AR user interface's ability to follow the user's head movement can be divided into two phases, both sharing a spring-damped-mass physical model. This application refers to these two phases as the hysteresis phase and the smooth recovery phase. In the hysteresis phase, the user's head movement stretches the spring, generating a significant pulling force. At this point, the dynamic damping model generates a dynamic damping coefficient based on the instantaneous head movement rate. The AR user interface slowly lags behind, attempting to keep up with the user's head movement. However, from the user's relative perspective, the AR user interface is far behind the user's head rotation, resulting in the visual effect of the AR user interface drifting in the opposite direction. To prevent the AR user interface from moving too far in the opposite direction, it stops moving when it is a preset offset from the center position. In the smooth recovery phase, the AR user interface moves towards its center position with an acceleration pointing towards the center. However, when the AR user interface reaches the midpoint, due to inertia, it will move to the right of the midpoint, creating an oscillation effect, and eventually stop at the center position. In this application, when the AR user interface moves to the center position (i.e., the target position), the interface position is immediately locked and the movement stops, thus preventing the vibration effect and improving the user experience. Figure 7 The image shown is an example of how the AR user interface follows the user's head movement when the user turns their head to the right.

[0069] In this embodiment, the current interface element is determined based on the current interface mode, and the AR user interface is rendered in conjunction with the actual position. If the current head motion data triggers the recovery condition, the AR user interface is restored to the target position by reducing the damping coefficient.

[0070] Specifically, based on the current interface mode and the actual position of the mass point calculated from the dynamic equations of the spring-damped-mass model, the interface elements in the current AR user interface and the actual position of the AR user interface are determined, rendered, and the rendered AR interface frame data is transmitted to the display screen of the AR glasses to generate the AR user interface visible to the user, so as to ensure real-time rendering and visual smoothness.

[0071] It should be noted that the refresh rate of the AR user interface in this application is consistent with the sampling rate of the inertial measurement unit, both being 1000Hz, meaning that rendering is completed once every 1ms, ensuring that the AR user interface is lag-free.

[0072] In this embodiment, the reverse recovery to the initial relative position in the environmental field of view is triggered by recovery conditions. The recovery conditions include: the instantaneous motion rate does not exceed a lower threshold, the motion acceleration is less than a preset acceleration threshold, and the motion angle is less than a preset angle threshold, and the duration exceeds a first time.

[0073] Specifically, the recovery conditions include the following four conditions: Condition 1: The instantaneous head movement rate of the current user is less than or equal to 30° / s, indicating that the user's head has no obvious movement or is moving slowly. Condition 2: The acceleration of the user's head movement is less than a preset acceleration threshold (exemplarily set to 5° / s). 2 Condition 1: The user's head does not move drastically. Condition 2: The user's head movement angle is less than a preset angle threshold (exemplarily set to 2°), indicating that the user's head does not tremble slightly. Condition 3: All three conditions above are met, and the duration of each condition exceeds the first time (exemplarily set to 1 second).

[0074] If the current head movement triggers the recovery condition, indicating that the user's head movement has ended, the AR user interface is restored to the target position by reducing the damping coefficient, and then the AR user interface is restored to the standard mode. Otherwise, it indicates that the user's head is still moving, and steps 101 to 103 are repeated.

[0075] In this embodiment, the damping coefficient is gradually reduced to the base damping coefficient, causing the AR user interface to move towards the target position under the tension of the spring. When the real-time position of the AR user interface coincides with the target position, the position of the AR user interface is locked.

[0076] Specifically, after the recovery condition is triggered, the damping coefficient is gradually reduced from its current value to the base damping coefficient C0 at a preset rate. Since the angular velocity decay of a human head when it comes to a natural stop is around 5-20° / s, the preset rate of damping coefficient reduction is set in the range of 10-30° / s. This ensures a smooth transition without excessive delay.

[0077] During this process, the spring tension becomes dominant, propelling the AR user interface towards the target location. Due to the gradual decrease in damping coefficient, the movement of the AR user interface becomes faster and smoother, avoiding the sluggish movement caused by excessive damping coefficient.

[0078] Simultaneously, the system monitors the real-time position of the AR user interface and compares it with the target position. When the real-time position of the AR user interface coincides with the target position, the system immediately locks the interface position to prevent overshoot or oscillation of interface elements due to inertia or other factors. At this point, the AR user interface has fully returned to standard mode, with no scaling, displaying all interface elements and providing users with a clear and comprehensive information display.

[0079] Furthermore, to ensure the smoothness and stability of the recovery process, this application continuously monitors the user's head movement data during recovery. If the user generates significant head movement again during recovery, and this head movement data triggers the interface mode switching condition, the recovery process will be paused, and the interface mode will be switched back based on the new head movement data, adjusting the position of the AR user interface. This dynamic adjustment mechanism ensures the flexibility and adaptability of the AR user interface in different usage scenarios.

[0080] In this embodiment, during the restoration of the interface position and interface mode, in addition to continuously monitoring the user's head movement data to dynamically adjust the interface mode, this application also introduces a restoration speed control mechanism. Specifically, the restoration speed is not fixed, but dynamically adjusted according to the distance between the current AR user interface and the target position. When the AR user interface is far from the target position, the restoration speed can be relatively fast to quickly close the gap between the AR user interface and the target position. When the AR user interface approaches the target position, the restoration speed gradually slows down to avoid overshooting or instability of the AR user interface caused by excessively fast restoration speed.

[0081] While this application provides the method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in this embodiment is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the methods shown in this embodiment or the accompanying drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0082] like Figure 8 As shown in the figure, this application embodiment also provides a dynamic adjustment device 800 for an AR user interface. The device includes: a feature extraction module 801, a switching module 802, and a determination module 803, as detailed below.

[0083] The feature extraction module 801 is used to extract features from the real-time detected user head motion data to obtain the head motion state. The head motion state includes the head motion rate and the head motion direction.

[0084] The switching module 802 is used to switch the interface mode of the AR user interface based on the head movement rate. The size of the AR user interface is different for different interface modes.

[0085] The determination module 803 is used to determine the motion of the AR user interface based on the head movement direction and head movement rate.

[0086] Some modules in the apparatus described in this application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0087] The apparatus or module described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. For ease of description, the above apparatus is described by dividing it into various modules according to their functions. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0088] The methods, apparatus, or modules described in this application can be implemented in a computer-readable program code manner. The controller can be implemented in any suitable manner, such as a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of a memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code manner, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included within it for implementing various functions can also be considered as structures within the hardware component. Alternatively, the device used to implement various functions can be viewed as either a software module that implements the method or a structure within a hardware component.

[0089] This application also provides an apparatus, the apparatus comprising: a processor; a memory for storing processor-executable instructions; wherein, when the processor executes the executable instructions, it implements the method described in this application.

[0090] This application also provides a non-volatile computer-readable storage medium storing a computer program or instructions thereon, which, when executed, enables the method described in this application embodiment to be implemented.

[0091] Furthermore, in the various embodiments of the present invention, each functional module can be integrated into a processing module, or each module can exist independently, or two or more modules can be integrated into a single module.

[0092] The aforementioned storage media include, but are not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), Cache, Hard Disk Drive (HDD), or Memory Card. The memory can be used to store computer program instructions.

[0093] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, or it can be embodied in the process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0094] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0095] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of this application.

Claims

1. A method for dynamically adjusting an AR user interface, the method comprising: The application comprises: feature extraction is performed on the head movement data of the user in real time to obtain a head movement state; the head movement state comprises a head movement speed and a head movement direction; an interface mode of an AR user interface is switched based on the head movement speed, different interface modes corresponding to AR user interfaces of different sizes; wherein the interface mode comprises a standard mode, a compact mode and a minimalist mode; in the standard mode, the AR user interface is not scaled and all interface elements are displayed; in the compact mode, the AR user interface is scaled at a medium scale and important interface elements are displayed; in the minimalist mode, the AR user interface is scaled at a minimum scale and only key interface elements are displayed; the movement of the AR user interface is determined based on the head movement direction and the head movement speed, comprising: when the head movement speed does not exceed a lower threshold, the relative position of the AR user interface in the field of view of the environment does not change; when the head movement speed exceeds the lower threshold, the AR user interface moves to a preset position in the opposite direction of the head movement direction and then reverses to the initial relative position in the field of view of the environment.

2. The method of claim 1, wherein, the interface mode of the AR user interface is switched based on the head movement speed, comprising: when the head movement speed does not exceed a lower threshold, the interface mode remains the standard mode; when the head movement speed is between the lower threshold and an upper threshold, the interface mode is switched to the compact mode; when the head movement speed reaches the upper threshold, the interface mode is switched to the minimalist mode.

3. The method of claim 1, wherein, when the head movement speed exceeds the lower threshold, the AR user interface moves to a preset position in the opposite direction of the head movement direction, comprising: the movement speed of the AR user interface is positively correlated with the head movement speed.

4. The method of claim 1, wherein, the movement of the AR user interface is determined based on the head movement direction and the head movement speed, comprising: the AR user interface is abstracted as a mass point in a spring-damper-mass model, and the actual position of the current AR user interface is determined by solving the dynamic equation of the spring-damper-mass model; wherein the damping coefficient of the spring-damper-mass model is dynamically determined based on the head movement state; the current interface elements are determined according to the current interface mode, and the AR user interface is rendered in combination with the actual position; if the current head movement data triggers a recovery condition, the AR user interface is recovered to a target position by reducing the damping coefficient.

5. The method of claim 4, wherein, the AR user interface is abstracted as a mass point, which is connected to a target position by a spring and a damper to obtain a spring-damper-mass model; wherein the target position is determined in real time based on the head movement data of the user. after the actual position of the current AR user interface is determined by solving the dynamic equation of the spring-damper-mass model, the method further comprises:

6. The method of claim 4, wherein, if the deviation between the actual position of the current AR user interface and the target position exceeds a preset offset, the actual position of the current AR user interface is forced to stop moving, and then the actual position of the current AR user interface is controlled to move to the target position after the head movement of the user stops. ​ If a deviation between the actual position of the current AR user interface and the target position does not exceed a preset offset, the actual position of the current AR user interface is updated by the dynamic equation.

7. The method of claim 1, wherein, The reverse recovery to the initial relative position in the environmental field of view is triggered based on a recovery condition; The recovery condition includes that a momentary motion rate does not exceed a lower threshold, a motion acceleration is less than a preset acceleration threshold, and a motion angle is less than a preset angle threshold, and a duration exceeds a first time.

8. The method of claim 4, wherein, The recovery of the AR user interface to the target position by reducing the damping coefficient includes: The damping coefficient is gradually reduced to a basic damping coefficient size, so that the AR user interface moves to the target position under the pulling force of the spring; When the real-time position of the AR user interface coincides with the target position, the position of the AR user interface is locked.

9. An apparatus for implementing dynamic adjustment of an AR user interface for the method of any one of claims 1-8, characterized in that, It includes: A feature extraction module is configured to extract features from head motion data of a user detected in real time to obtain a head motion state. The head motion state includes a head motion rate and a head motion direction. A switching module is configured to switch an interface mode of the AR user interface based on the head motion rate, different interface modes corresponding to different sizes of the AR user interface; wherein the interface mode includes a standard mode, a compact mode, and a minimalist mode; in the standard mode, the AR user interface is not scaled and displays all interface elements; in the compact mode, the AR user interface is scaled at a medium scale and displays important interface elements; and in the minimalist mode, the AR user interface is scaled at a minimum scale and displays only key interface elements. A determination module is configured to determine motion of the AR user interface based on the head motion direction and the head motion rate, including: when the head motion rate does not exceed a lower threshold, the relative position of the AR user interface in the environmental field of view is unchanged; and when the head motion rate exceeds the lower threshold, the AR user interface moves to a preset position in the opposite direction of the head motion direction, and then reversely recovers to the initial relative position in the environmental field of view.

Images