Screen following method and device of intelligent wearable equipment, equipment and storage medium

Abstract

The invention provides a screen following method and device for intelligent wearable equipment, computer equipment and a storage medium, and the method comprises the steps: judging whether a target window is located in a current display picture or not according to a projection coordinate of a current position coordinate of the target window in a plane where the current display picture is located and a coordinate region of the current display picture; and the accuracy of window position detection is improved. According to the current position coordinate and the picture center coordinate, the moving path of the target window moving from the current position to the current display picture is calculated, automatic triggering and returning of the window are achieved, the operation burden of a user is reduced, and usability and flexibility of three-degree-of-freedom equipment are improved; and meanwhile, according to the moving path, the target window is moved into the current display picture, and the target window can smoothly return to the current display picture, so that the window return is more dynamic, the discomfort of a user is reduced, and the smoothness and flexibility of the window return are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a screen following method, device, equipment and storage medium for a smart wearable device. Background Art

[0002] In the field of virtual reality and augmented reality, 3Dof (three degrees of freedom) devices (such as AR glasses, VR glasses, smart helmets, etc.) are increasingly used. 3Dof devices usually do not allow users to perform rotation operations (pitch, yaw, roll), but do not support translation operations (front and back, left and right, up and down).

[0003] In the related art, 3Dof systems usually adopt a fixed window position when moving a window outside the user's field of view. However, when the user's field of view moves, the user needs to manually adjust the viewing angle to see the window again, which increases the operation burden.

[0004] Therefore, how to improve the window movement flexibility of the 3Dof device has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a screen following method, apparatus, device and storage medium for a smart wearable device, aiming to improve the window movement flexibility of a 3Dof device.

[0006] In a first aspect, the present application provides a screen following method for a smart wearable device, the method comprising:

[0007] Obtaining the center coordinates of the current display screen and the projection coordinates of the current position coordinates of the target window in the plane where the current display screen is located;

[0008] When the projection coordinates are outside the coordinate area where the current display screen is located, based on the current position coordinates and the screen center coordinates, calculating a moving path of the target window from the current position to the current display screen;

[0009] Based on the moving path, the target window is moved from a current position to the current display screen, so as to display the target window in the current display screen.

[0010] In a second aspect, the present application further provides a screen following device for a smart wearable device, the screen following device for the smart wearable device comprising:

[0011] A coordinate acquisition module, used to acquire the center coordinates of the current display screen and the projection coordinates of the current position coordinates of the target window in the plane where the current display screen is located;

[0012] A moving path calculation module, used for calculating the moving path of the target window from the current position to the current display screen based on the current position coordinates and the screen center coordinates when the projection coordinates are outside the coordinate area where the current display screen is located;

[0013] The interpolation coordinate calculation module is used to move the target window from the current position to the current display screen based on the moving path, so as to display the target window in the current display screen.

[0014] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the screen following method of the smart wearable device as described above are implemented.

[0015] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the screen following method of the smart wearable device as described above are implemented.

[0016] The present application provides a screen following method, device, computer device and storage medium for a smart wearable device. The present application method determines the relative position of the target window and the current display screen according to the projection coordinates of the current position coordinates of the target window on the plane where the current display screen is located, and the coordinate area of ​​the current display screen, thereby judging whether the target window is located in the current display screen, and improving the accuracy of window position detection. According to the current position coordinates and the center coordinates of the screen, the moving path of the target window from the current position to the current display screen is calculated, and the automatic triggering return of the window is realized, so that the window can move with the movement of the current display screen, and the automatic following of the window is realized, which reduces the user's operating burden and improves the ease of use and flexibility of the three-degree-of-freedom device; at the same time, according to the moving path, the target window is moved to the current display screen, and the target window can smoothly return to the current display screen, making the window return more dynamic, reducing user discomfort, and improving the smoothness and flexibility of the window return. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1A schematic diagram of a flow chart of a first embodiment of a screen following method for a smart wearable device provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of a second embodiment of a screen following method for a smart wearable device provided in an embodiment of the present application;

[0020] Figure 3 This is a structural schematic diagram of a first embodiment of a screen following device for a smart wearable device provided by the present application;

[0021] Figure 4 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0022] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0025] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0026] Please refer to Figure 1 , Figure 1 A schematic flow chart of a first embodiment of a screen following method for a smart wearable device provided in an embodiment of the present application.

[0027] like Figure 1 As shown, the screen following method of the smart wearable device includes steps S101 to S103.

[0028] S101, obtaining the center coordinates of the current display screen and the projection coordinates of the current position coordinates of the target window in the plane where the current display screen is located;

[0029] In one embodiment, when the target user wears a 3Dof device (such as AR glasses, head-mounted smart helmet, etc.), the current display screen corresponding to the current field of view of the target user is obtained. The position of the 3Dof device can be used as the origin (the midpoint of the baseline between the human eyes can be used as the origin), and the three-dimensional coordinates of the center point of the current display screen relative to the origin position of the target user's eyes are calculated to obtain the center coordinates of the current display screen.

[0030] In one embodiment, the starting position when the target user starts the 3Dof device can be obtained, and the initial posture of the display screen of the 3Dof device at this time can be obtained, and the initial position coordinates of the key points (such as midpoints, corner points, etc.) of the target window in the display screen of the initial posture can be obtained, and then the current position coordinates of the initial position coordinates of the target window in the world coordinate system corresponding to the three-dimensional display space of 3Dof are calculated according to the starting position and the current position of the 3Dof device, that is, the initial position coordinates of the target window relative to the current relative coordinates of the 3Dof device.

[0031] In one embodiment, because the target window does not move with the movement of the 3Dof device, when the 3Dof device is displaced, the position coordinates of the target window relative to the 3Dof device change, but the starting position coordinates of the target window relative to the 3Dof device are relatively unchanged, and the position coordinates of the 3Dof device after displacement are based on the starting position coordinates. Therefore, the displacement direction and displacement distance of the 3Dof device can be determined, and the relative position coordinates of the current position of the 3Dof device after displacement relative to the starting position can be calculated; that is, the target window and the current position of the 3Dof device after displacement share the same reference coordinates, namely, the starting position coordinates of the 3Dof device. Therefore, the starting position coordinates of the 3Dof device can be used as reference coordinates, the relative position of the initial position coordinates of the target window relative to the starting position coordinates of the 3Dof device, and the relative position of the current position coordinates of the 3Dof device after displacement relative to the starting position coordinates, and the relative position of the initial position coordinates of the target window relative to the current position coordinates of the 3Dof device after displacement can be calculated, so as to obtain the position coordinates of the target window in the three-dimensional space coordinate system with the current position coordinates as the origin.

[0032] For example, assuming that the starting position coordinates of the 3Dof device are (0, 0, 0, the initial position coordinates of the target window when the 3Dof device is started are (1, 1, 1, and the current position of the 3Dof device is (0, 0, 1) relative to the starting position coordinates, then at this time, the position coordinates of the target window relative to the current position (current origin) of the 3Dof device are (1, 1, 0.

[0033] In one embodiment, the projection coordinates of the current position coordinates of the target window on the plane where the current display screen is located can be calculated by coordinate conversion. Specifically, it can include obtaining the projection matrix between the plane where the current display screen is located and the world coordinate system, and then converting the current position coordinates (3D coordinates) of the target window in the world coordinate system into three-dimensional coordinates in the clipping coordinate system, and then converting the three-dimensional coordinates in the clipping coordinate system into the NDC (Normalized Device System) coordinate system to obtain NDC coordinates, and finally, converting the NDC coordinates into the screen coordinate system of the plane where the current display screen is located to obtain 2D screen coordinates, i.e., projection coordinates.

[0034] Among them, the clipping coordinate system is a concept in computer graphics. It is an intermediate coordinate system used to clip the three-dimensional world coordinates before converting them into two-dimensional screen coordinates. The conversion from world coordinates to clipping coordinates is completed through the model view matrix and the projection matrix. The model view matrix is ​​used to convert the world coordinates into coordinates from the observer's perspective, while the projection matrix is ​​used to project these coordinates into the clipping coordinate system. In perspective projection, the coordinates in the clipping coordinate system are normalized to the range of -1 to 1 through perspective division. Perspective division is achieved by dividing the X, Y, and Z coordinates by the W coordinate (the fourth component of the homogeneous coordinate).

[0035] Generally, in the clipping coordinate system, only those points whose coordinate values ​​are in the range of -1 to 1 will be retained, and these points are considered to be within the clipping window. Points outside this range will be clipped.

[0036] Furthermore, the projection matrix of the plane where the current display screen is located is obtained; based on the projection matrix, the current position coordinates of the target window are converted into clipping coordinates; based on the screen resolution of the current display screen, the clipping coordinates are converted into coordinates in the plane where the current display screen is located to obtain the projection coordinates.

[0037] In one embodiment, the projection matrix may be a perspective projection matrix or an orthogonal projection matrix.

[0038] Exemplarily, the formula of the perspective projection matrix P is as follows:

[0039]

[0040] Where n is the distance to the near clipping plane, f is the distance to the far clipping plane, r and l are the right and left boundaries of the field of view, and t and b are the top and bottom boundaries of the field of view.

[0041] Generally, the coordinates of the clipping coordinate system are usually between -1 and 1. In this coordinate system, -1 usually corresponds to one side of the viewing cone, and 1 corresponds to the other side. For the Z axis, 0 usually corresponds to the near clipping plane and 1 corresponds to the far clipping plane. The clipping plane is a plane defined in perspective projection to determine the visible range. The field of view is the angle that describes the range of 3D space that the observer can see, defining the widest and highest boundaries that the observer can see.

[0042] Using the model view matrix M0 and the projection matrix P, the world coordinates (X, Y, Z, W) are transformed into clip coordinates (x c ,y c , z c , w c ):

[0043]

[0044] Among them, W refers to the fourth component in homogeneous coordinates. Points or vectors in 3D space usually have three coordinate components, namely (X, Y, Z. In order to facilitate the mathematical processing of projections and transformations, the fourth component W is introduced so that these points and vectors can be represented and operated in four-dimensional space.

[0045] In one embodiment, the model view matrix M0 is a matrix used to convert model coordinates into coordinates from the observer's perspective, which consists of two parts: a model matrix and a view matrix. Among them, the model matrix is ​​also called the world matrix, which defines the position, rotation, and scaling of the object in the world space. The model matrix may include translation (moving the object to a specific position), rotation (rotating the object around a specific axis), and scaling (changing the size of the object). The view matrix, also known as the camera matrix, defines the position and direction of the observer. The view matrix typically includes a transformation from world coordinates to observer coordinates, which involves moving the observer (such as a virtual camera) to the origin and aligning the observer's line of sight to the negative Z axis.

[0046] Exemplarily, the model view matrix M0 can be expressed as:

[0047] M0=M1×M2

[0048] Among them, M1 is the model matrix and M2 is the view matrix.

[0049] Exemplarily, the model matrix M1 can be expressed as:

[0050] M1=T×R×S

[0051] Among them, T is the translation matrix, R is the rotation matrix, and S is the scaling matrix.

[0052] Exemplarily, the model matrix M2 can be expressed as:

[0053] M2=E×L

[0054] Among them, E is the translation matrix that moves the observer (such as a virtual camera) to the origin, that is, the observer is defined as the origin; L is the rotation matrix that aligns the viewing direction to the negative Z axis.

[0055] Generally, the coordinates in the clip coordinate system are converted to the Normalized Device Coordinates (NDC) after perspective division. The X and Y coordinates of the NDC coordinate system range from -1 to 1, and the Z coordinate range is from 0 to 1. Finally, the NDC coordinates will be converted to the screen coordinate system, usually by mapping the NDC coordinates to the pixel coordinates of the screen. This needs to take into account the screen resolution and viewport settings.

[0056] In one embodiment, when performing coordinate transformation, a 3D point (X, Y, Z) is typically expanded to a four-dimensional vector (X, Y, Z, 1, where W is initially set to 1. After the coordinate transformation, if W is not 0, the coordinates can be normalized back to 3D space by dividing by W: (X / W, Y / W, Z / W. In perspective division, W can be used to convert NDC coordinates to screen coordinates, where W is subtracted from the NDC coordinates and then divided by W to obtain the screen coordinates.

[0057] In this embodiment, in perspective projection, the use of homogeneous coordinates can easily convert 3D points to 2D screen space while maintaining the geometric properties of the projection. In matrix multiplication, the use of homogeneous coordinates can avoid division operations because when the matrix is ​​multiplied by the homogeneous coordinate vector, the fourth component W will be used for normalization. In clipping space, homogeneous coordinates are used to define whether a point is within the clipping window and how to perform clipping operations.

[0058] In one embodiment, perspective division is performed on the basis of the homogeneous clipping space, and the resulting coordinates are called NDC space coordinates. Therefore, the clipping coordinates (x c ,y c , z c , w c ) is the NDC coordinate.

[0059] Among them, the homogeneous clipping space is a four-dimensional space used to store the homogeneous coordinates of points and vectors before the clipping operation. In the homogeneous clipping space, points and vectors are represented as quaternions (X, Y, Z, W, where W is the fourth component of the homogeneous coordinates. In the homogeneous clipping space, the world coordinates are converted to clipping coordinates through the projection matrix. These clipping coordinates are normalized to normalized device coordinates (NDC) after perspective division, and their X, Y, and Z components are also limited to the range of [-1, 1]. Perspective division is achieved by dividing the X, Y, and Z components of the clipping coordinates by the W component.

[0060] In one embodiment, if w c ≠0, the clipping coordinates can be normalized to obtain the NDC coordinates:

[0061]

[0062] In one embodiment, the NDC coordinates are converted into pixel coordinates in the pixel coordinate system of the current display image, that is, screen coordinates (x s ,y s ), at this time, the screen resolution (width, height) of the current display needs to be considered, so we can get:

[0063]

[0064] Among them, width is the pixel width of the current display screen, and height is the pixel height of the current display screen.

[0065] In one embodiment, the screen coordinates (x s ,y s ) is the projection coordinate of the target window on the plane where the current display screen is located. Thus, the current position coordinates (3D coordinates) of the target window are converted to the projection coordinates (2D coordinates) of the plane where the current display screen is located, so as to facilitate the determination of whether the target window is within the coordinate area of ​​the current display window.

[0066] S102, when the projection coordinates are outside the coordinate area where the current display screen is located, calculating a moving path of the target window from the current position to the current display screen based on the current position coordinates and the screen center coordinates;

[0067] In one embodiment, the coordinate area range of the current display screen can be determined based on the screen size (pixel height and pixel width) of the current display screen and the screen coordinates of the center of the current display screen, and then the relative position of the projection coordinates and the coordinates of the center of the screen are compared, or the relative position of the projection coordinates and the boundary points of the coordinate area range (such as corner points, midpoints of side lines, etc.) are compared to obtain the relative position of the projection coordinates and the coordinate area of ​​the current display screen, and determine whether the projection coordinates are located in the coordinate area where the current display screen is located.

[0068] For example, if the screen coordinates of the center of the current display screen are (u0, v0), and the screen size of the current display screen is (w0, h0), then if

[0069]

[0070] It can be determined that the projection coordinates are within the coordinate area of ​​the current display screen.

[0071] On the contrary, if the horizontal distance between the screen coordinates of the center of the picture and the projection coordinates is greater than half the width of the screen size, or the vertical distance is greater than half the height of the screen size, or the horizontal distance is greater than half the width of the screen size and the vertical distance is greater than half the height of the screen size, it can be determined that the projection coordinates are outside the coordinate area currently displayed.

[0072] In one embodiment, the static state of the current display screen of the 3Dof device can be determined by collecting inertial measurement data of the 3Dof device using an inertial measurement unit (IMU).

[0073] In one embodiment, the inertial measurement unit IMU is a device for measuring the three-axis attitude angle (or angular rate) and acceleration of an object. Generally, an IMU includes three single-axis accelerometers and three single-axis gyroscopes. The accelerometer detects the acceleration signal of the object in the independent three axes of the carrier coordinate system, and the gyroscope detects the angular velocity signal of the carrier relative to the navigation coordinate system, measures the angular velocity and acceleration of the object in three-dimensional space, and calculates the attitude of the object.

[0074] In one embodiment, the inertial measurement data may include three-axis attitude angles or angular rates, acceleration, and the like.

[0075] Furthermore, each time the current display screen is refreshed, inertial measurement data of the inertial measurement unit IMU is collected to obtain an inertial measurement data set; based on the data change amount of each inertial measurement data in the inertial measurement data set, it is determined whether the current display screen is in a static state; when the data change amount of any two of the inertial measurement data in the inertial measurement data set is less than a data change threshold, it is determined that the current display screen is in the static state.

[0076] It can be understood that the 3Dof device is worn on the target user's head, and the postures of the 3Dof device and the target user's head are synchronized. Therefore, the current display screen and the target user's head are also in a state of unchanged relative posture, that is, when the user's head remains stationary, that is, the 3Dof device is in a stationary state, and the current display screen is also in a stationary state; when the user's head deflects, moves, etc., the 3Dof device will also move or rotate accordingly, and the current display screen will be in motion. Therefore, it is possible to determine whether the current display screen is in a stationary state by detecting the inertial measurement data of the 3Dof device.

[0077] In one embodiment, the inertial measurement data set may include at least two inertial measurement data.

[0078] In one embodiment, each time the current display screen is refreshed, the inertial measurement data of the inertial measurement unit is collected, and a set of inertial measurement data sets is formed according to the screen refresh sequence or time sequence. Wherein, each time the screen is refreshed, the inertial measurement data of the inertial measurement unit is collected, and each collected inertial measurement data may include one or more of the three-axis attitude angle, angular velocity, acceleration and other data.

[0079] In another embodiment, the inertial measurement data of the inertial measurement unit can be collected once every time the current display screen refreshes a fixed number of times. For example, when the refresh frequency of the current display screen is high, such as the refresh frequency is 60 Hz, that is, the screen is refreshed 60 times per second, at this time, the inertial measurement data can be collected once every 10 times the screen is refreshed (which can be configured independently according to actual needs), that is, 6 inertial measurement data can be collected per second.

[0080] In one embodiment, the data change amount of any two inertial measurement data in each set of inertial measurement data sets, such as the change amount of the three-axis attitude angle, the change amount of acceleration, etc., can be calculated. If the data change amount of any two inertial measurement data is less than a preset data change threshold, it is considered that the current display screen is in a static state. For example, if the acceleration change amount is less than the acceleration change threshold, it can be considered that the current display screen is in a uniform speed state or a static state. At this time, if the change amount of the three-axis attitude angle is also less than the attitude angle change threshold, it can be determined that the current display screen is in a static state.

[0081] In this embodiment, the inertial measurement data of the 3Dof device is detected by the inertial measurement unit IMU, so as to judge whether the current display screen is in a static state according to the data change amount of the inertial measurement data, thereby improving the recognition accuracy of the static state of the current display screen.

[0082] Furthermore, when the current display screen is in a static state, the amount of data in the inertial measurement data set and the screen refresh frequency of the current display screen are obtained; based on the data amount and the screen refresh frequency, the static time of the current display screen is calculated; when the static time of the current display screen is greater than or equal to a preset time length, based on the projection coordinates and the coordinate area where the current display screen is located, the relative position of the projection coordinates and the current display screen is determined.

[0083] In one embodiment, inertial measurement data is collected every time the current display screen is refreshed a certain number of times (once or more). Therefore, the static time of the current display screen can be calculated based on the screen refresh frequency of the current display screen and the data volume of the inertial measurement data.

[0084] For example, if the refresh frequency of the current display screen is F (unit: Hz, i.e., the number of refreshes per second), when the current display screen is in a static state, the amount of data in the inertial measurement data set is G, and it is stipulated that the inertial measurement data is collected once every b refreshes of the current display screen, then the static time of the current display screen can be calculated as:

[0085]

[0086] Among them, T ′ is the static time in seconds (s). The ratio of b to F indicates the time interval between the acquisition of two adjacent inertial measurement data.

[0087] In a specific embodiment, assuming that when the current display screen is in a static state, the IMU collects 120 data points, the screen refresh frequency is 60 Hz, and the inertial measurement data is collected once each time the current display screen is refreshed, then the static time of the current display screen is:

[0088]

[0089] That is, the duration for which the current display image remains static is 2 seconds.

[0090] In one embodiment, when the projection coordinates are outside the coordinate area of ​​the current display screen, a spherical linear interpolation (SLERP) method may be used to calculate a moving path of the target window from the current position coordinates to the center coordinates of the current display screen.

[0091] In one embodiment, spherical linear interpolation is a method of interpolating on a sphere, and is generally used to calculate a smooth transition of rotation or direction. The basic idea of ​​SLERP is to smoothly connect two unit quaternions (representing rotation) by interpolation.

[0092] In one embodiment, the interpolation coordinates are the coordinates of points on the moving path. In order to make the target window move from the current position coordinates to the center coordinates of the screen along the moving path more smoothly, the window movement process can be divided into multiple stages, multiple jump points are taken from the moving path, and the coordinates of each point, i.e., the interpolation coordinates, are calculated. The target window can jump to the position corresponding to each interpolation coordinate in sequence from the current position coordinates to the center coordinates of the screen until it moves to the center coordinates of the screen, thereby completing the smooth movement of the target window.

[0093] For example, assuming that the current position coordinate is P0, the center coordinate of the screen is P1, and the interpolation parameter p changes from 0 to 1, it represents the transition process from the current position coordinate to the center coordinate of the screen. The interpolation coordinate is calculated using the calculation formula of spherical linear interpolation:

[0094]

[0095] Where θ is the angle between P0 and P1. As p increases from 0 to 1, the interpolated coordinate P(p) will smoothly transition from P0 to P1.

[0096] S103: Based on the moving path, move the target window from the current position to the current display screen, so as to display the target window in the current display screen.

[0097] Furthermore, based on the moving path, at least one interpolation coordinate is obtained; based on a rendering engine, the interpolation coordinate is rendered to generate a window return animation; based on the window return animation, the target window is moved to the interpolation coordinate until the target window is moved to the current display screen.

[0098] In one embodiment, the moving path includes one or more interpolation coordinates. After each interpolation coordinate is calculated according to the spherical linear interpolation algorithm, the position of the target window is updated to P(p). When the interpolation parameter p is gradually increased from 0 to 1, each time the value of the interpolation parameter p is increased, the interpolation coordinate P(p) is recalculated and the position of the target window is synchronously updated. When the interpolation parameter p=1, the target window will be completely moved to the position where the center coordinate of the screen is located, thereby moving the target window from the current position to the current display screen for display.

[0099] In one embodiment, the midpoint coordinates of the moving path may be the coordinates of the center of the screen, or any screen coordinates in the current display screen, as long as the target window can be moved to the current display screen and displayed completely.

[0100] In one embodiment, the rendering engine can be OpenGL, DirectX, Unity, Unreal Engine, Three.js and other engines. Among them, OpenGL is a widely used cross-language, cross-platform graphics API (Application Programming Interface). DirectX is mainly used for graphics and audio APIs on Windows platforms. Unity is a popular game development engine that supports 3D and 2D game development. Unreal Engine is an advanced game development engine that provides rich visual effects and animation tools. Three.js is a JavaScript library for WebGL that allows 3D graphics to be created in a browser.

[0101] In one embodiment, the window return animation is a series of images or positions generated by interpolation at a series of time points. The rendering engine can configure the animation duration, frame rate, and interpolation parameters for each frame, and then use the spherical linear interpolation formula to calculate the target position (interpolation coordinates) of each update of the target window.

[0102] Exemplarily, the process of window return animation may include creating a window or rendering surface, then calculating the interpolation coordinates of the target window currently refreshed, and moving the target window to the interpolation coordinates of the current refresh, then clearing the interpolation coordinates of the target window last refreshed, drawing the currently refreshed image, and finally displaying the rendered image at the currently refreshed interpolation coordinates.

[0103] In a specific embodiment, the starting state and the ending state of the window return animation are first determined. For example, the target window may move from the interpolation coordinates of the last refresh to the interpolation coordinates of the current refresh, or expand from a small window to a large window at the interpolation coordinates of the current refresh. Then, according to the needs of animation display, a suitable animation effect can be selected, such as fade in and fade out, zoom, slide, etc. The animation effect is realized using a programming language and a corresponding library or framework. During the animation display process, the interpolation coordinates need to be calculated and applied to the target window. The rendering engine draws the display content of the target window according to the interpolation coordinates, and displays the display content of the target window to the interpolation coordinates according to the settings of the animation effect, etc.

[0104] This embodiment provides a screen following method for a smart wearable device, which determines the relative position of the target window and the current display screen according to the projection coordinates of the current position coordinates of the target window on the plane where the current display screen is located, and the coordinate area of ​​the current display screen, thereby judging whether the target window is located in the current display screen, and improving the accuracy of window position detection. According to the comparison between the static time of the current display screen and the preset time length, the posture state of the current display screen is judged, and then when it is determined that the current display screen is in a static state, the moving path of the target window from the current position to the current display screen is calculated according to the current position coordinates and the center coordinates of the screen, so as to realize the automatic triggering return of the window, reduce the operation burden of the user, and improve the ease of use and flexibility of the three-degree-of-freedom device; at the same time, according to the moving path, the target window is moved to the current display screen, and the target window can smoothly return to the current display screen, making the window return more dynamic, reducing the user's discomfort, and improving the smoothness and flexibility of the window return.

[0105] Please refer to Figure 2 , Figure 2 A schematic flow chart of a second embodiment of a screen following method for a smart wearable device provided in the present application.

[0106] In this embodiment, Figure 2 As shown, based on the above Figure 1 In the illustrated embodiment, step S102 specifically includes:

[0107] S201, determining a deflection angle between the current position coordinates and the picture center coordinates based on the current position coordinates and the picture center coordinates;

[0108] In one embodiment, a line connecting the current position coordinates and the current origin (i.e., the position coordinates of the 3Dof device) is one side, and a line connecting the center coordinates of the screen and the current origin is the other side, forming a deflection angle with the current origin as the vertex, and the angle of the deflection angle is the deflection angle between the current position coordinates and the center coordinates of the screen.

[0109] Specifically, according to the current position coordinates, the center coordinates of the screen and the current origin, the straight-line distances between the three coordinates are calculated respectively as the three side lengths of the triangle formed by the three coordinate points, and then the angle of the vertex where the current origin is located is calculated according to the angle calculation formula of the triangle to obtain the deflection angle.

[0110] In one embodiment, the angle calculation formula of a triangle can adopt a conventional calculation method in the art, and the embodiment of the present application will not be described in detail here.

[0111] S202, calculating the position offset angle of the target window at each screen refresh based on the screen refresh frequency of the current display screen, the deflection angle, and the window movement time;

[0112] Furthermore, based on the screen refresh frequency, the time interval for each screen refresh is determined; based on the time interval and the window movement time, the number of refreshes of the currently displayed screen is determined; based on the refresh number and the deflection angle, the position offset angle of the target window during each screen refresh is calculated.

[0113] In one embodiment, in AR glasses, the target window is calculated to smoothly transition from a current position coordinate to the current display screen, and SLERP (Spherical Linear Interpolation) can be used to determine the position offset angle of the window at different time points.

[0114] For example, it is assumed that there are two angles, a starting angle θ0 and a target angle θ1, and a time parameter T0 (the range of T0 is [0, 1], where 0 represents the starting time and 1 represents the ending time), wherein the starting angle θ0 is the angle of the current position coordinate of the target window relative to the position coordinate of the 3Dof device (the current origin), and the target angle θ1 is the position offset angle of the center coordinate of the screen relative to the position coordinate of the 3Dof device. The formula according to spherical linear interpolation is as follows:

[0115]

[0116] Among them, Δθ is the difference between the two angles, but it is necessary to ensure that this difference is the angle difference on the shortest path, that is:

[0117] Δθ=min(|θ1-θ0|, 2π-|θ1-θ0|)

[0118] Among them, 2π is a complete circle angle, and the minimum value of the difference between the two angles can be taken to ensure that the interpolation is performed along the shortest path.

[0119] For example, if the refresh rate (e.g., 60 Hz means 60 refreshes per second) and the total transition time (e.g., 2 seconds) are known, the t value for each refresh can be calculated. Assuming the total number of refreshes is N, the time interval Δt for each refresh is:

[0120]

[0121] For the nth refresh, the value of t will be:

[0122]

[0123] Using this t value, the position offset angle θ (t n ).

[0124] S203: Calculate at least one interpolation coordinate based on the position offset angle to obtain the moving path.

[0125] In one embodiment, the screen coordinates corresponding to each position offset angle on the plane of the current display image are calculated according to the position offset angle, that is, the interpolation coordinates on the moving path.

[0126] Exemplarily, assuming that the minimum moving path of the current position coordinates and the center coordinates of the screen is a straight line between the two, the length of the straight line can be calculated based on the two coordinates, and at the same time, the angle of the vertex of the current origin is calculated based on the triangle formed by the current position coordinates, the center coordinates of the screen, and the current origin, that is, the deflection angle. By calculating the angle of the target window at each refresh, the angle between the interpolated coordinates and the current position coordinates (with the current origin as the vertex) can be obtained at each refresh. According to the ratio of the angle and the deflection angle, and the length of the straight line between the current position coordinates and the center coordinates of the screen, the moving distance of the interpolated coordinates obtained by refreshing relative to the current position coordinates at each refresh can be known, and then the corresponding point coordinates, that is, the interpolated coordinates, can be found on the straight line between the current position coordinates and the center coordinates of the screen.

[0127] In one embodiment, the interpolation coordinates are the point coordinates of each interpolation point on the moving path. According to a series of interpolation coordinates calculated in time sequence, the moving path of the target window from the current position coordinates to the center coordinates of the screen can be constructed.

[0128] This embodiment can ensure that the target window moves more smoothly, reduce the jumping feeling, and improve the user experience by calculating the position offset angle at each refresh. By using interpolated coordinates to calculate the moving path, the movement of the target window can be controlled more accurately to avoid display problems caused by inaccurate movement. By reasonably allocating the number of refreshes and deflection angles, the consumption of computing resources can be reduced while ensuring the display effect, thereby improving the operating efficiency of the system.

[0129] See also Figure 3 , Figure 3 It is a structural schematic diagram of a first embodiment of a screen following device for a smart wearable device provided in the present application, wherein the screen following device for the smart wearable device is used to execute the screen following method for the smart wearable device described above.

[0130] like Figure 3 As shown, the screen following device 300 of the smart wearable device includes: a coordinate acquisition module 301, a moving path calculation module 302 and an interpolation coordinate calculation module 303.

[0131] A coordinate acquisition module 301 is used to acquire the center coordinates of the current display screen and the projection coordinates of the current position coordinates of the target window in the plane where the current display screen is located;

[0132] A moving path calculation module 302, configured to calculate a moving path of the target window from the current position to the current display screen based on the current position coordinates and the screen center coordinates when the projection coordinates are outside the coordinate area of ​​the current display screen;

[0133] The interpolation coordinate calculation module 303 is used to move the target window from the current position to the current display screen based on the moving path, so as to display the target window in the current display screen.

[0134] In one embodiment, the coordinate acquisition module 301 includes:

[0135] A projection matrix acquisition unit, used to acquire the projection matrix of the plane where the current display screen is located;

[0136] A clipping coordinate acquisition unit, used for converting the current position coordinates of the target window into clipping coordinates based on the projection matrix;

[0137] The projection coordinate acquisition unit is used to convert the clipping coordinates into coordinates in the plane where the current display picture is located based on the screen resolution of the current display picture, so as to obtain the projection coordinates.

[0138] In one embodiment, the screen following device 300 of the smart wearable device further includes a screen state recognition module, and the screen state recognition module specifically includes:

[0139] a data acquisition unit, configured to acquire, when the current display screen is in a static state, the amount of data in the inertial measurement data set and the picture refresh frequency of the current display screen;

[0140] The static time calculation unit is used to calculate the static time of the currently displayed picture based on the data volume and the picture refresh frequency.

[0141] In one embodiment, the screen state recognition module specifically includes:

[0142] A data acquisition unit, used to collect inertial measurement data of an inertial measurement unit IMU each time the current display screen is refreshed, to obtain an inertial measurement data set;

[0143] a state judgment unit, configured to judge whether the current display screen is in a static state based on a data change amount of each of the inertial measurement data in the inertial measurement data set;

[0144] The static state determining unit is configured to determine that the current display screen is in the static state when the data change amount between any two of the inertial measurement data in the inertial measurement data set is less than a data change threshold.

[0145] In one embodiment, the moving path calculation module 302 includes:

[0146] a deflection angle determining unit, configured to determine a deflection angle between the current position coordinates and the picture center coordinates based on the current position coordinates and the picture center coordinates;

[0147] A position offset angle calculation unit, configured to calculate the position offset angle of the target window at each screen refresh based on the screen refresh frequency of the currently displayed screen, the deflection angle, and the window movement time;

[0148] The moving path obtaining unit is used to calculate at least one interpolation coordinate based on the position offset angle to obtain the moving path.

[0149] In one embodiment, the position offset angle calculation unit includes:

[0150] A time interval determination subunit, used to determine the time interval of each screen refresh based on the screen refresh frequency;

[0151] A refresh number determination subunit, used to determine the refresh number of the current display image based on the time interval and the window movement time;

[0152] The position offset angle calculation subunit is used to calculate the position offset angle of the target window each time the screen is refreshed based on the refresh times and the deflection angle.

[0153] In one embodiment, the interpolation coordinate calculation module 303 includes:

[0154] an interpolation coordinate calculation unit, configured to obtain at least one interpolation coordinate based on the movement path;

[0155] An animation rendering unit, used for rendering the interpolation coordinates based on a rendering engine to generate a window return animation;

[0156] A window moving unit is used to move the target window to the interpolation coordinates based on the window return animation until the target window is moved into the current display screen.

[0157] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned screen following method embodiment of the smart wearable device, and will not be repeated here.

[0158] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 4 Runs on the computer device shown.

[0159] See also Figure 4 , Figure 4 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0160] See also Figure 4 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0161] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any screen following method of the smart wearable device.

[0162] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0163] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any screen following method of the smart wearable device.

[0164] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0165] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0166] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0167] Obtaining the center coordinates of the current display screen and the projection coordinates of the current position coordinates of the target window in the plane where the current display screen is located;

[0168] When the projection coordinates are outside the coordinate area where the current display screen is located, based on the current position coordinates and the screen center coordinates, calculating a moving path of the target window from the current position to the current display screen;

[0169] Based on the moving path, the target window is moved from a current position to the current display screen, so as to display the target window in the current display screen.

[0170] In one embodiment, when the processor implements the step of acquiring the projection coordinates of the current position coordinates of the target window in the plane where the current display screen is located, the processor is used to implement:

[0171] Obtaining the projection matrix of the plane where the current display image is located;

[0172] Based on the projection matrix, convert the current position coordinates of the target window into clipping coordinates;

[0173] Based on the screen resolution of the current display picture, the clipping coordinates are converted into coordinates in the plane where the current display picture is located to obtain the projection coordinates.

[0174] In one embodiment, before the processor implements the step of calculating the moving path of the target window from the current position to the current display screen based on the current position coordinates and the screen center coordinates when the projection coordinates are outside the coordinate area where the current display screen is located, the processor is further configured to implement:

[0175] When the current display screen is in a static state, acquiring the data volume in the inertial measurement data set and the screen refresh frequency of the current display screen;

[0176] Based on the data volume and the picture refresh frequency, the static time of the currently displayed picture is calculated.

[0177] In one embodiment, before the processor implements acquiring the amount of data in the inertial measurement data set and the picture refresh frequency of the current display picture when the current display picture is in a static state, it is further configured to implement:

[0178] Each time the current display screen is refreshed, inertial measurement data of an inertial measurement unit IMU is collected to obtain an inertial measurement data set;

[0179] Based on the data change amount of each of the inertial measurement data in the inertial measurement data set, determining whether the current display screen is in a static state;

[0180] When the data change amount of any two of the inertial measurement data in the inertial measurement data set is less than a data change threshold, it is determined that the current display screen is in the static state.

[0181] In one embodiment, when the processor implements the calculation of the moving path of the target window from the current position to the currently displayed screen based on the current position coordinates and the screen center coordinates, the processor is used to implement:

[0182] Based on the current position coordinates and the picture center coordinates, determining a deflection angle between the current position coordinates and the picture center coordinates;

[0183] Calculating the position offset angle of the target window at each screen refresh based on the screen refresh frequency of the current display screen, the deflection angle, and the window movement time;

[0184] Based on the position offset angle, at least one interpolation coordinate is calculated to obtain the movement path.

[0185] In one embodiment, when the processor calculates the position offset angle of the target window at each screen refresh based on the screen refresh frequency of the currently displayed screen, the deflection angle, and the window movement time, it is configured to implement:

[0186] Based on the picture refresh frequency, determine the time interval for each picture refresh;

[0187] Determining the number of refreshes of the currently displayed image based on the time interval and the window movement time;

[0188] Based on the refresh times and the deflection angle, the position offset angle of the target window is calculated each time the screen is refreshed.

[0189] In one embodiment, when the processor implements the moving of the target window from the current position to the current display screen based on the moving path so as to display the target window in the current field of view, it is configured to implement:

[0190] Based on the movement path, obtaining at least one interpolated coordinate;

[0191] Based on a rendering engine, the interpolated coordinates are rendered to generate a window return animation;

[0192] Based on the window return animation, the target window is moved to the interpolation coordinates until the target window is moved into the current display screen.

[0193] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement a screen following method for any smart wearable device provided in an embodiment of the present application.

[0194] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0195] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A screen following method for a smart wearable device, characterized in that: The method comprises: Obtaining the center coordinates of the current display screen and the projection coordinates of the current position coordinates of the target window in the plane where the current display screen is located; When the projection coordinates are outside the coordinate area where the current display screen is located, based on the current position coordinates and the screen center coordinates, calculating a moving path of the target window from the current position to the current display screen; Based on the moving path, the target window is moved from a current position to the current display screen, so as to display the target window in the current display screen.

2. The screen following method of the smart wearable device according to claim 1, characterized in that: The step of obtaining the projection coordinates of the current position coordinates of the target window in the plane where the current display screen is located includes: Obtaining the projection matrix of the plane where the current display image is located; Based on the projection matrix, convert the current position coordinates of the target window into clipping coordinates; Based on the screen resolution of the current display picture, the clipping coordinates are converted into coordinates in the plane where the current display picture is located to obtain the projection coordinates.

3. The screen following method of the smart wearable device according to claim 1, characterized in that: When the projection coordinates are outside the coordinate area where the current display screen is located, before calculating the moving path of the target window from the current position to the current display screen based on the current position coordinates and the screen center coordinates, the method further includes: When the current display screen is in a static state, acquiring the data volume in the inertial measurement data set and the screen refresh frequency of the current display screen; Calculating the static time of the currently displayed picture based on the data volume and the picture refresh frequency; When the static time of the currently displayed image is greater than or equal to a preset time length, the relative position of the projection coordinates and the coordinate area where the currently displayed image is located is determined based on the projection coordinates and the coordinate area where the currently displayed image is located, so as to judge whether the projection coordinates are within the coordinate area where the currently displayed image is located according to the relative position.

4. The screen following method of the smart wearable device according to claim 3, characterized in that: Before acquiring the data amount in the inertial measurement data set and the picture refresh frequency of the current display picture when the current display picture is in a static state, the method further includes: Each time the current display screen is refreshed, inertial measurement data of an inertial measurement unit IMU is collected to obtain the inertial measurement data set; When the data change amount of any two of the inertial measurement data in the inertial measurement data set is less than a data change threshold, it is determined that the current display screen is in the static state.

5. The screen following method of the smart wearable device according to claim 1, characterized in that: The calculating, based on the current position coordinates and the center coordinates of the screen, a moving path of the target window from the current position to the currently displayed screen includes: Based on the current position coordinates and the picture center coordinates, determining a deflection angle between the current position coordinates and the picture center coordinates; Calculating the position offset angle of the target window at each screen refresh based on the screen refresh frequency of the current display screen, the deflection angle, and the window movement time; Based on the position offset angle, calculating at least one interpolation coordinate; The moving path is constructed based on the current position coordinates, the screen center coordinates and at least one of the interpolated coordinates.

6. The screen following method of the smart wearable device according to claim 5, characterized in that: The calculating the position offset angle of the target window at each screen refresh based on the screen refresh frequency of the current display screen, the deflection angle, and the window movement time includes: Based on the picture refresh frequency, determine the time interval for each picture refresh; Determining the number of refreshes of the current display image based on the time interval and the window movement time; Based on the refresh times and the deflection angle, the position offset angle of the target window is calculated each time the screen is refreshed.

7. The screen following method of the smart wearable device according to claim 1, characterized in that: The step of moving the target window from a current position to the current display screen based on the moving path so as to display the target window in the current field of view includes: Based on the movement path, obtaining at least one interpolated coordinate; Based on a rendering engine, the interpolated coordinates are rendered to generate a window return animation; Based on the window return animation, the target window is moved to the interpolation coordinates until the target window is moved into the current display screen.

8. A screen following device for a smart wearable device, characterized in that: The screen following device of the smart wearable device comprises: A coordinate acquisition module, used to acquire the center coordinates of the current display screen and the projection coordinates of the current position coordinates of the target window in the plane where the current display screen is located; A moving path calculation module, used for calculating the moving path of the target window from the current position to the current display screen based on the current position coordinates and the screen center coordinates when the projection coordinates are outside the coordinate area where the current display screen is located; The interpolation coordinate calculation module is used to move the target window from the current position to the current display screen based on the moving path, so as to display the target window in the current display screen.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the screen following method of the smart wearable device according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the screen following method for a smart wearable device according to any one of claims 1 to 7 are implemented.

Images