Screen following method and apparatus for smart wearable device, and device and storage medium

Abstract

The present application provides a screen following method and apparatus for a smart wearable device, and a device and a storage medium. The method comprises: on the basis of projected coordinates of the current position coordinates of a target window on a current display picture, and a coordinate region of the current display picture, determining whether the target window is located in the current display picture; on the basis of the current position coordinates and picture center coordinates, calculating a movement path of the target window; and on the basis of the movement path, moving the target window into the current display picture.

Description

Screen following method, device and equipment of smart wearable device and storage medium

[0001] This application claims priority to the Chinese patent application No. 2024111200077, filed on August 14, 2024, entitled "Screen following method, device and equipment of smart wearable device and storage medium", and the Chinese patent application No. 2024111398088, filed on August 19, 2024, entitled "Screen following method, device and equipment of smart wearable device and storage medium", the contents of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of data processing, in particular to a screen following method, device and equipment of smart wearable device and storage medium. BACKGROUND

[0003] In the field of virtual reality and augmented reality, the application of 3Dof (three degrees of freedom) devices (such as AR glasses, VR glasses, smart headsets, etc.) is becoming more and more widespread. 3Dof devices generally allow users to perform rotation operations (pitch, yaw, roll), but do not support translation operations (forward, backward, left, right, up and down).

[0004] In related technologies, when the 3Dof system moves the window outside the user's field of view, a fixed window position is generally used. However, with a fixed window position, when the user's field of view moves, the user needs to manually adjust the viewing angle to see the window again, increasing the operational burden.

[0005] Therefore, how to improve the flexibility of window movement of 3Dof devices has become a technical problem to be solved. SUMMARY

[0006] The present application provides a screen following method, device and equipment of smart wearable device and storage medium, aiming to improve the flexibility of window movement of 3Dof devices.

[0007] In a first aspect, the present application provides a screen following method of smart wearable device, the method comprising:

[0008] obtaining a picture center coordinate of a current display picture, and a projection coordinate of a current position coordinate of a target window in a plane where the current display picture is located;

[0009] when the projection coordinate is located outside the coordinate region where the current display picture is located, calculating a movement path of the target window from the current position to the current display picture based on the current position coordinate and the picture center coordinate;

[0010] The interpolation coordinate calculation module is configured to move the target window from the current position to the current display picture based on the movement path, so as to display the target window in the current display picture.

[0011] In a second aspect, the present application provides a screen following device of a smart wearable device, which comprises:

[0012] A coordinate acquisition module is configured to acquire a picture center coordinate of a current display picture and a projection coordinate of a current position coordinate of a target window in a plane where the current display picture is located;

[0013] A movement path calculation module is configured to calculate a movement path of the target window from the current position to the current display picture based on the current position coordinate and the picture center coordinate when the projection coordinate is located outside a coordinate region where the current display picture is located.

[0014] An interpolation coordinate calculation module is configured to move the target window from the current position to the current display picture based on the movement path, so as to display the target window in the current display picture.

[0015] In a third aspect, the present application provides a computer device, which comprises a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the computer program is executed by the processor to implement the steps of the screen following method of the smart wearable device as described above.

[0016] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of the screen following method of the smart wearable device as described above.

[0017] The application provides a screen following method and device of a smart wearable device, a computer device and a storage medium. The application determines the relative position of a target window and a current display picture according to the projection coordinates of the current position coordinates of the target window on the plane of the current display picture and the coordinate region of the current display picture, so as to determine whether the target window is located in the current display picture, and improve the accuracy of window position detection. According to the current position coordinates and the picture center coordinates, the moving path of the target window from the current position to the current display picture is calculated, the automatic triggering return of the window is realized, the window can move following the movement of the current display picture, the automatic following of the window is realized, the operation burden of the user is reduced, and the usability and flexibility of the three-degree-of-freedom device are improved. Meanwhile, according to the moving path, the target window is moved to the current display picture, the target window can be smoothly returned to the current display picture, the window return is more dynamic, the user discomfort is reduced, and the smoothness and flexibility of the window return are improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can also be obtained by those skilled in the art without any creative effort.

[0019] Fig. 1 is a flow diagram of a first embodiment of a screen following method of a smart wearable device provided by the application;

[0020] Fig. 2 is a flow diagram of a second embodiment of a screen following method of a smart wearable device provided by the application;

[0021] Fig. 3 is a structural diagram of a first embodiment of a screen following device of a smart wearable device provided by the application;

[0022] Fig. 4 is a structural diagram of a computer device provided by the embodiment of the application.

[0023] The implementation, functional features and advantages of the application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the application will be described clearly and completely with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are some embodiments of the application, but not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without any creative effort are within the protection scope of the application.

[0025] The flowchart shown in the drawings is only an example and does not necessarily include all the contents and operations / steps, nor does it have to be executed in the order described. For example, some operations / steps can be further divided, combined or partially merged, so the actual execution order can be changed according to the actual situation.

[0026] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0027] Please refer to FIG. 1, which is a flowchart of a first embodiment of a screen following method of a smart wearable device according to an embodiment of the present application.

[0028] As shown in FIG. 1, the screen following method of the smart wearable device includes steps S101-S103.

[0029] S101, obtain the center coordinate of the current display screen and the projection coordinate of the current position coordinate of the target window in the plane where the current display screen is located;

[0030] In an embodiment, when the target user wears a 3Dof device (such as AR glasses, a head-mounted smart helmet, etc.), the current display screen corresponding to the target user's current field of view is obtained, and the center point of the current display screen can be calculated relative to the three-dimensional coordinates of the eye origin position of the target user, taking the position of the 3Dof device as the origin (which can take the midpoint of the baseline between the eyes as the origin), to obtain the center coordinate of the current display screen.

[0031] In an embodiment, the starting position when the target user starts the 3Dof device can be obtained, and the initial pose of the display screen of the 3Dof device at this time is obtained, and the initial position coordinates of the key points (such as the midpoint, corner point, etc.) of the target window in the initial pose of the display screen are obtained. Then, according to the starting position and the current position of the 3Dof device, the current position coordinates of the initial position coordinates of the target window in the world coordinate system corresponding to the three-dimensional display space of the 3Dof device are calculated, i.e. the current relative coordinates of the initial position coordinates of the target window relative to the 3Dof device.

[0032] In an embodiment, because the target window does not move with the 3Dof device, when the 3Dof device is displaced, the position coordinates of the target window relative to the 3Dof device change, but the starting position coordinates of the target window relative to the 3Dof device are relatively unchanged, and the position coordinates of the 3Dof device after displacement are relative to the starting position coordinates, so the displacement direction and distance of the 3Dof device can be determined, and the relative position coordinates of the current position of the 3Dof device after displacement relative to the starting position coordinates can be calculated; that is, the target window and the current position of the 3Dof device after displacement share the same reference coordinates, that is, the starting position coordinates of the 3Dof device. Therefore, the starting position coordinates of the 3Dof device can be taken as the reference coordinates, the relative position of the initial position coordinates of the target window relative to the starting position coordinates of the 3Dof device, and the relative position of the current position coordinates of the 3Dof device after displacement relative to the starting position coordinates, can be taken to calculate the relative position between the initial position coordinates of the target window and the current position coordinates of the 3Dof device after displacement, so as to obtain the position coordinates of the target window in the three-dimensional coordinate system with the current position coordinates as the origin.

[0033] For example, assuming that the starting position coordinates of the 3Dof device are (0, 0, 0), the initial position coordinates of the target window when the 3Dof device is started are (1, 1, 1), and the current position coordinates of the 3Dof device relative to the starting position coordinates are (0, 0, 1), then the position coordinates of the target window relative to the current position (current origin) of the 3Dof device are (1, 1, 0).

[0034] In an embodiment, the current position coordinates of the target window can be calculated in the projection coordinates of the plane on which the current display screen is located by means of coordinate conversion. Specifically, the projection matrix between the plane on which the current display screen is located and the world coordinate system can be obtained, and then the current position coordinates (3D coordinates) of the target window in the world coordinate system are converted to three-dimensional coordinates in the clipping coordinate system, and then the three-dimensional coordinates in the clipping coordinate system are converted to NDC (Normalized Device System, normalized device coordinates) coordinates, and finally the NDC coordinates are converted to the screen coordinate system of the plane on which the current display screen is located, and 2D screen coordinates, that is, projection coordinates, are obtained.

[0035] where the clipping coordinate system is a concept in computer graphics, which is an intermediate coordinate system used to perform clipping operations on the coordinates before converting them from three-dimensional world coordinates to two-dimensional screen coordinates. The conversion from world coordinates to clipping coordinates is done using a model-view matrix and a projection matrix. The model-view matrix is used to convert world coordinates to coordinates from the perspective of an observer, while the projection matrix is used to project these coordinates into the clipping coordinate system. In perspective projection, the coordinates in the clipping coordinate system are normalized to the range -1 to 1 through perspective division. Perspective division is achieved by dividing the X, Y, Z coordinates by the W coordinate (the fourth component of homogeneous coordinates).

[0036] Generally, in the clipping coordinate system, only points whose coordinate values are within the range of -1 to 1 are retained, and these points are considered to be within the clipping window. Points outside this range will be clipped.

[0037] Further, a projection matrix of a plane where the current display screen is located is obtained; based on the projection matrix, the current position coordinates of the target window are converted into clipping coordinates; based on the screen resolution of the current display screen, the clipping coordinates are converted into coordinates in the plane where the current display screen is located, to obtain the projection coordinates.

[0038] In an embodiment, the projection matrix can be a perspective projection matrix or an orthogonal projection matrix.

[0039] For example, the formula of the perspective projection matrix P is as follows:

[0040] where n is the distance of the near clipping plane, f is the distance of the far clipping plane, r and l are the right and left boundaries of the field of view, and t and b are the top and bottom boundaries of the field of view.

[0041] Generally, the coordinates of the clipping coordinate system are usually between -1 and 1. In this coordinate system, -1 usually corresponds to one side of the viewing frustum, and 1 corresponds to the other side. For the Z axis, 0 usually corresponds to the near clipping plane, and 1 corresponds to the far clipping plane. The clipping plane is a plane defined in perspective projection for determining the visible range, and the field of view is an angle describing the range of 3D space that an observer can see, defining the widest and highest boundaries that the observer can see.

[0042] Using the model-view matrix M0 and the projection matrix P, the world coordinates (X, Y, Z, W) are converted into clipping coordinates (x c , y c , z c , w c ):

[0043] where W refers to the fourth component in Homogeneous Coordinates. Points or vectors in 3D space usually have three coordinate components, i.e. (X, Y, Z). To facilitate mathematically handling of projection and transformation, a fourth component W is introduced so that these points and vectors can be represented and operated in four-dimensional space.

[0044] In an embodiment, the modelview matrix M0 is a matrix used to convert model coordinates to coordinates in the viewer's perspective, which is composed of two parts, a model matrix and a view matrix. The model matrix, also known as the world matrix, defines the position, rotation, and scale of an object in the world space. The model matrix can contain translation (moving the object to a specific position), rotation (turning the object around a specific axis), and scaling (changing the size of the object). The view matrix, also known as the camera matrix, defines the position and orientation of the viewer. The view matrix usually contains a transformation from world coordinates to viewer coordinates, which involves moving the viewer (e.g. a virtual camera) to the origin and aligning the viewer's line of sight to the negative Z axis.

[0045] For example, the modelview matrix M0 can be expressed as: M0 = M1 x M2

[0046] where M1 is the model matrix and M2 is the view matrix.

[0047] For example, the model matrix M1 can be expressed as: M1 = T x R x S

[0048] where T is the translation matrix, R is the rotation matrix, and S is the scaling matrix.

[0049] For example, the model matrix M2 can be expressed as: M2 = E x L

[0050] where E is the translation matrix that moves the viewer (e.g. a virtual camera) to the origin, i.e. defines the viewer as the origin, and L is the rotation matrix that aligns the viewing direction to the negative Z axis.

[0051] Generally, coordinates in the clip coordinate system are converted to the Normalized Device Coordinates (NDC) after perspective division. The X and Y coordinates of the NDC coordinate system range from -1 to 1, and the Z coordinate ranges from 0 to 1. Finally, the NDC coordinates are converted to the screen coordinate system, usually by mapping the NDC coordinates to the pixel coordinates of the screen. This needs to consider the resolution of the screen and the viewport settings.

[0052] In one embodiment, when performing coordinate transformation, a 3D point (X, Y, Z) is usually extended to a four-dimensional vector (X, Y, Z, 1), where W is initially set to 1. After coordinate transformation, if W is not 0, the coordinates can be normalized back to 3D space by dividing by W: (X / W, Y / W, Z / W). In perspective division, W can be used to convert NDC coordinates to screen coordinates, where W is subtracted from the NDC coordinates and then divided by W to get the screen coordinates.

[0053] In this embodiment, in perspective projection, using homogeneous coordinates can facilitate the conversion of 3D points to 2D screen space while preserving the geometric properties of the projection. In matrix multiplication, using homogeneous coordinates can avoid division operations because the fourth component W will be used for normalization when a matrix is multiplied by a homogeneous coordinate vector. In clipping space, homogeneous coordinates are used to define whether a point is within the clipping window and how to perform clipping operations.

[0054] In one embodiment, perspective division is performed on the basis of homogeneous clipping space, and the resulting coordinates are called NDC space coordinates, so the clipping coordinates (x c , y c , z c , w c ) are NDC coordinates.

[0055] where the homogeneous clipping space is a four-dimensional space used to store the homogeneous coordinates of points and vectors before performing clipping operations. In homogeneous clipping space, points and vectors are represented as four-tuples (X, Y, Z, W), where W is the fourth component of the homogeneous coordinates. In homogeneous clipping space, world coordinates are converted to clipping coordinates by a projection matrix. These clipping coordinates are normalized into normalized device coordinates (NDC) after perspective division, and their X, Y, Z components are also limited to the range [-1, 1]. Perspective division is achieved by dividing the X, Y, Z components of the clipping coordinates by the W component.

[0056] In one embodiment, if w c ≠ 0, the clipping coordinates can be normalized to obtain NDC coordinates by normalization:

[0057] In one embodiment, the NDC coordinates are converted to pixel coordinates in the pixel coordinate system of the current display frame, i.e. screen coordinates (x s , y s ). At this time, the screen resolution (width, height) of the current display frame needs to be considered, so we can get:

[0058] wherein width is the pixel width of the current display picture, and height is the pixel height of the current display picture.

[0059] In an embodiment, the screen coordinate (x s , y s ) obtained through the above conversion is the projection coordinate of the target window on the plane where the current display picture is located. Thus, the current position coordinate (three-dimensional coordinate) of the target window is converted to the projection coordinate (two-dimensional coordinate) on the plane where the current display picture is located, so as to facilitate the judgment of whether the target window is within the coordinate area of the current display window.

[0060] S102, when the projection coordinate is located outside the coordinate area where the current display picture is located, based on the current position coordinate and the picture center coordinate, a moving path of the target window from the current position to the current display picture is calculated;

[0061] In an embodiment, the coordinate area range of the current display picture can be determined according to the screen size (pixel height and pixel width) of the current display picture and the screen coordinate of the picture center of the current display picture, and then the relative position of the projection coordinate and the picture center coordinate is compared, or the relative position of the projection coordinate and the boundary point (such as the corner point, the mid-point of the side line, etc.) of the coordinate area range is compared, so as to obtain the relative position of the projection coordinate and the coordinate area of the current display picture, and to judge whether the projection coordinate is located within the coordinate area where the current display picture is located.

[0062] For example, if the screen coordinate of the picture center of the current display picture is (u0, v0), and the screen size of the current display picture is (w0, h0), if

[0063] it can be determined that the projection coordinate is located within the coordinate area range of the current display picture.

[0064] On the contrary, if the horizontal distance between the screen coordinate of the picture center and the projection coordinate is greater than half of the width of the screen size, or the vertical distance is greater than half of the height of the screen size, or the horizontal distance is greater than half of the width of the screen size and the vertical distance is greater than half of the height of the screen size, it can be determined that the projection coordinate is located outside the coordinate area range of the current display picture.

[0065] In an embodiment, the static state of the current display picture of the 3Dof device can be judged by collecting the inertial measurement data of the 3Dof device through the inertial measurement unit (IMU).

[0066] In an embodiment, the inertial measurement unit (IMU) is a device that measures the three-axis attitude angle (or angular rate) and acceleration of an object. Generally, an IMU includes three single-axis accelerometers and three single-axis gyroscopes. The accelerometers detect the acceleration signals of the object in the independent three-axis of the carrier coordinate system, and the gyroscopes detect the angular velocity signals of the carrier relative to the navigation coordinate system. The angular velocity and acceleration of the object in three-dimensional space are measured, and the attitude of the object is calculated based on the angular velocity and acceleration.

[0067] In an embodiment, the inertial measurement data can include three-axis attitude angle or angular rate, acceleration, and the like.

[0068] Further, the inertial measurement data of the inertial measurement unit (IMU) is collected every time the current display screen is refreshed to obtain a set of inertial measurement data. Based on the data variation of each inertial measurement data in the set of inertial measurement data, it is determined whether the current display screen is in a stationary state. When the data variation of any two inertial measurement data in the set of inertial measurement data is less than a data variation threshold, it is determined that the current display screen is in the stationary state.

[0069] It can be understood that the 3Dof device is worn on the head of the target user, and the attitude of the 3Dof device and the head of the target user is synchronized. Therefore, the current display screen and the head of the target user are also in a relative pose-invariant state. That is, when the head of the user remains stationary, that is, the 3Dof device is in a stationary state, the current display screen is also in a stationary state. When the head of the user is deflected, moved, or the like, the 3Dof device will also move or rotate, and the current display screen will be in a motion state. Therefore, whether the current display screen is in a stationary state can be determined by detecting the inertial measurement data of the 3Dof device.

[0070] In an embodiment, the set of inertial measurement data can include at least two inertial measurement data.

[0071] In an embodiment, the inertial measurement data of the inertial measurement unit (IMU) is collected every time the current display screen is refreshed, and a set of inertial measurement data is formed according to the order of screen refresh or the order of time. Wherein, the inertial measurement data of the inertial measurement unit (IMU) is collected every time the screen is refreshed, and the inertial measurement data collected each time can include one or more of three-axis attitude angle, angular rate, acceleration, and the like.

[0072] In another embodiment, the inertial measurement data of the inertial measurement unit can also be collected once every fixed number of refreshes of the current display screen, such as when the refresh frequency of the current display screen is high, for example, the refresh frequency is 60 Hz, that is, the screen is refreshed 60 times per second. At this time, the inertial measurement data can be collected once every 10 refreshes of the screen (which can be configured independently according to actual needs), that is, 6 inertial measurement data can be collected per second.

[0073] In an embodiment, the data change amount of any two inertial measurement data in each set of inertial measurement data can be calculated, such as the change amount of three-axis attitude angle, the change amount of acceleration, and the like. If the data change amount of any two inertial measurement data is less than a preset data change threshold, it is considered that the current display screen is in a stationary state. For example, if the acceleration change amount is less than the acceleration change threshold, it can be considered that the current display screen is in a uniform speed state or a stationary state. At this time, if the change amount of the three-axis attitude angle is also less than the attitude angle change threshold, it can be determined that the current display screen is in a stationary state.

[0074] In the embodiment, the inertial measurement data of the 3Dof device is detected by the inertial measurement unit IMU, so as to determine whether the current display screen is in a stationary state according to the data change amount of the inertial measurement data, and improve the identification accuracy of the stationary state of the current display screen.

[0075] Further, when the current display screen is in a stationary state, the data amount in the inertial measurement data set and the screen refresh frequency of the current display screen are obtained; the stationary time of the current display screen is calculated based on the data amount and the screen refresh frequency; and when the stationary time of the current display screen is greater than or equal to a preset time length, the relative position of the projection coordinate and the current display screen is determined based on the projection coordinate and the coordinate region where the current display screen is located.

[0076] In an embodiment, the inertial measurement data is collected once every certain number of refreshes (one or more times) of the current display screen, so that the stationary time of the current display screen can be calculated according to the screen refresh frequency of the current display screen and the data amount of the inertial measurement data.

[0077] For example, if the screen refresh frequency of the current display screen is F (unit: Hz, that is, the number of refreshes per second), the data amount in the inertial measurement data set is G when the current display screen is in a stationary state, and it is specified that the inertial measurement data is collected once every b refreshes of the current display screen, then the stationary time of the current display screen can be calculated as:

[0078] T = G / F / b ′The static time is in seconds (s). The ratio of b and F represents the interval of the collection time between two adjacent inertial measurement data.

[0079] In an embodiment, assuming that 120 data points are collected by the IMU when the current display screen is in a static state, the screen refresh frequency is 60 Hz, and the inertial measurement data is collected once per screen refresh, the static time of the current display screen is:

[0080] That is, the duration of the current display screen remaining in the static state is 2 seconds.

[0081] In an embodiment, when the projection coordinates are located outside the coordinate region of the current display screen, a spherical linear interpolation (SLERP) method can be used to calculate the movement path of the target window moving from the current position coordinates to the center coordinates of the current display screen.

[0082] In an embodiment, spherical linear interpolation is a method of interpolating on a sphere, and is usually used to calculate the smooth transition of rotation or direction. The basic idea of SLERP is to smoothly connect two unit quaternions (representing rotation) through interpolation.

[0083] In an embodiment, the interpolation coordinates are point coordinates on the movement path. In order to make the target window move from the current position coordinates to the center coordinates of the screen along the movement path more smoothly, the window movement process can be divided into multiple stages, multiple jumping points are taken from the movement path, and the coordinates of each point, i.e., the interpolation coordinates, are calculated. The target window can jump to the positions corresponding to the interpolation coordinates in order according to the interpolation coordinates in the direction from the current position coordinates to the center coordinates of the screen, until it moves to the center coordinates of the screen, completing the smooth movement of the target window.

[0084] For example, assuming that the current position coordinates are P0, the center coordinates of the screen are P1, and the interpolation parameter p varies from 0 to 1, representing the transition process from the current position coordinates to the center coordinates of the screen. The interpolation coordinates are calculated using the calculation formula of spherical linear interpolation:

[0085] Where θ is the angle between P0 and P1. As p increases from 0 to 1, the interpolation coordinates P(p) will smoothly transition from P0 to P1.

[0086] S103, based on the movement path, moving the target window from the current position to the current display screen to display the target window in the current display screen.

[0087] Further, based on the movement path, at least one interpolation coordinate is obtained; based on a rendering engine, the interpolation coordinate is rendered to generate a window return animation; and based on the window return animation, the target window is moved to the interpolation coordinate until the target window is moved to the current display picture.

[0088] In an embodiment, the movement path includes one or more interpolation coordinates. After each interpolation coordinate is calculated according to the spherical linear interpolation algorithm, the position of the target window is updated to P(p). When the interpolation parameter p is gradually increased from 0 to 1, each time the value of the interpolation parameter p is increased, the interpolation coordinate P(p) is recalculated and the position of the target window is updated synchronously. Until the interpolation parameter p = 1, the target window will be completely moved to the position of the picture center coordinate, so as to move the target window from the current position to the current display picture for display.

[0089] In an embodiment, the midpoint coordinate of the movement path can be the picture center coordinate, or any screen coordinate in the current display picture, as long as the target window can be moved to the current display picture and displayed completely.

[0090] In an embodiment, the rendering engine can be OpenGL, DirectX, Unity, Unreal Engine, Three.js, etc. Among them, OpenGL is a widely used cross-language and cross-platform graphics API (Application Programming Interface). DirectX is mainly used for graphics and audio API on Windows platform. Unity is a popular game development engine that supports 3D and 2D game development. Unreal Engine is a high-level game development engine that provides rich visual effects and animation tools. Three.js is a JavaScript library for WebGL that allows 3D graphics to be created in the browser.

[0091] In an embodiment, the window return animation is a series of images or positions generated by interpolation at a series of time points. The rendering engine can configure the animation duration, frame rate, and interpolation parameter of each frame, and then use the spherical linear interpolation formula to calculate the target position of each update of the target window (interpolation coordinate).

[0092] For example, the process of the window return animation can include creating a window or rendering surface, then calculating the interpolation coordinate of the current refresh of the target window, and moving the target window to the interpolation coordinate of the current refresh, then clearing the interpolation coordinate of the last refresh of the target window, drawing the current refresh image, and finally displaying the rendered image at the interpolation coordinate of the current refresh.

[0093] In a specific embodiment, first, the starting state and the ending state of the window return animation are determined, for example, the target window can move from the interpolation coordinates of the last refresh to the interpolation coordinates of the current refresh, or expand from a small window to a large window at the interpolation coordinates of the current refresh. Then, according to the needs of the animation display, a suitable animation effect can be selected, such as fade-in and fade-out, zoom, sliding, etc. The animation effect is implemented using the morphing language and the corresponding library or framework. During the animation display process, the interpolation coordinates need to be calculated and applied to the target window. The rendering engine draws the display content of the target window according to the interpolation coordinates, and displays the display content of the target window to the interpolation coordinates according to the animation effect setting.

[0094] The embodiment provides a screen following method of a smart wearable device. The method determines the relative position of a target window and a current display picture according to the projection coordinates of the current position coordinates of the target window on the plane of the current display picture and the coordinate region of the current display picture, so as to determine whether the target window is located in the current display picture, and improve the accuracy of window position detection. The pose state of the current display picture is determined according to the comparison between the static time of the current display picture and the preset time length, and then the moving path of the target window from the current position to the current display picture is calculated according to the current position coordinates and the picture center coordinates when it is determined that the current display picture is in a static state, so as to realize the automatic triggering return of the window, reduce the operation burden of the user, and improve the ease of use and flexibility of the three-degree-of-freedom device. Meanwhile, the target window is moved to the current display picture according to the moving path, and the target window can be smoothly returned to the current display picture, so that the window return is more dynamic, the user's discomfort is reduced, and the smoothness and flexibility of the window return are improved.

[0095] Please refer to FIG. 2, which is a flowchart of a second embodiment of a screen following method of a smart wearable device provided by the present application.

[0096] In the embodiment, as shown in FIG. 2, based on the above-mentioned embodiment shown in FIG. 1, the step S102 specifically includes:

[0097] S201, determining the deflection angle between the current position coordinates and the picture center coordinates based on the current position coordinates and the picture center coordinates.

[0098] In an embodiment, the line connecting the current position coordinates and the current origin (i.e., the position coordinates of the 3Dof device) is taken as one side, and the line connecting the picture center coordinates and the current origin is taken as the other side, to form a deflection angle with the current origin as the vertex. The angle of the deflection angle is the deflection angle between the current position coordinates and the picture center coordinates.

[0099] Specifically, according to the current position coordinates, the picture center coordinates and the current origin, the straight line distances between the three coordinates are calculated respectively as the three side lengths of the triangle formed by the three coordinates, and then the angle of the top angle at the current origin is calculated according to the angle calculation formula of the triangle to obtain the deflection angle.

[0100] In an embodiment, the angle calculation formula of the triangle can be calculated in a conventional manner in the art, and the embodiments of the present application will not be described in detail here.

[0101] S202, based on the picture refresh frequency of the current display picture, the deflection angle and the window moving time, the position offset angle of the target window at each picture refresh is calculated;

[0102] Further, based on the picture refresh frequency, the time interval of each picture refresh is determined; based on the time interval and the window moving time, the refresh number of the current display picture is determined; based on the refresh number and the deflection angle, the position offset angle of the target window at each picture refresh is calculated.

[0103] In an embodiment, in the AR glasses, the calculation of the smooth transition of the target window from a current position coordinate to the current display picture can use SLERP (Spherical Linear Interpolation) to determine the position offset angle of the window at different time points.

[0104] For example, assuming there are two angles, a starting angle θ0 and a target angle θ1, and a time parameter T0 (the range of T0 is [0, 1], where 0 represents the starting time and 1 represents the end time), wherein the starting angle θ0 is the angle of the current position coordinates of the target window relative to the position coordinates of the 3Dof device (the current origin), and the target angle θ1 is the position offset angle of the picture center coordinates relative to the position coordinates of the 3Dof device. According to the formula of spherical linear interpolation is as follows:

[0105] Where Δθ is the difference between the two angles, but it needs to be ensured that this difference is the angle difference on the shortest path, that is: Δθ = min(|θ1-θ0|, 2π-|θ1-θ0|)

[0106] Where 2π is a complete circumferential angle, which can take the minimum value of the difference between the two angles to ensure interpolation along the shortest path.

[0107] For example, if the refresh frequency (for example, 60Hz means 60 refreshes per second) and the total transition time (for example, 2 seconds) are known, the t value at each refresh can be calculated. Assuming that the total refresh number is N, then the time interval Δt of each refresh is:

[0108] For the n-th refresh, the value of t will be:

[0109] Using this t value, the position offset angle of the target window at each refresh can be calculated as θ(t n ).

[0110] S203, based on the position offset angle, calculate at least one interpolation coordinate to obtain the movement path.

[0111] In an embodiment, according to the position offset angle, the interpolation coordinates on the movement path corresponding to the screen coordinates of the current display screen on the plane can be calculated.

[0112] For example, assuming that the minimum movement path between the current position coordinate and the picture center coordinate is a straight line between the two coordinates, the length of the straight line can be calculated according to the two coordinates, and the angle of the vertex at the current origin in the triangle formed by the current position coordinate, the picture center coordinate and the current origin, i.e. the deflection angle, can be calculated. The angle of the target window at each refresh can be calculated, and the included angle between the interpolation coordinate and the current position coordinate at each refresh (with the current origin as the vertex) can be obtained. According to the ratio of the included angle and the deflection angle, and the length of the straight line between the current position coordinate and the picture center coordinate, the movement distance of the interpolation coordinate relative to the current position coordinate at each refresh can be known, and the corresponding point coordinate on the straight line between the current position coordinate and the picture center coordinate, i.e. the interpolation coordinate, can be found.

[0113] In an embodiment, the interpolation coordinates are the point coordinates of each interpolation point on the movement path, and the movement path of the target window from the current position coordinate to the picture center coordinate can be constructed according to a series of interpolation coordinates calculated in time sequence.

[0114] By calculating the position offset angle at each refresh, the embodiment can ensure that the movement of the target window is smoother, reduce the sense of jumping, and improve the user experience. By calculating the movement path using the interpolation coordinates, the movement of the target window can be more accurately controlled, and display problems caused by inaccurate movement can be avoided. By reasonably allocating the number of refreshes and the deflection angle, the consumption of computing resources can be reduced while ensuring the display effect, and the running efficiency of the system can be improved.

[0115] Please refer to FIG. 3, which is a structural schematic diagram of a first embodiment of a screen following device of a smart wearable device provided by the present application. The screen following device of the smart wearable device is used to execute the screen following method of the smart wearable device as described above.

[0116] As shown in FIG. 3, the screen following device 300 of the smart wearable device comprises a coordinate acquisition module 301, a moving path calculation module 302 and an interpolation coordinate calculation module 303.

[0117] The coordinate acquisition module 301 is configured to acquire a picture center coordinate of a current display picture and a projection coordinate of a current position coordinate of a target window in a plane where the current display picture is located.

[0118] The moving path calculation module 302 is configured to, when the projection coordinate is located outside a coordinate region where the current display picture is located, calculate a moving path of the target window from a current position to the current display picture based on the current position coordinate and the picture center coordinate.

[0119] The interpolation coordinate calculation module 303 is configured to move the target window from the current position to the current display picture based on the moving path, so as to display the target window in the current display picture.

[0120] In an embodiment, the coordinate acquisition module 301 comprises:

[0121] A projection matrix acquisition unit is configured to acquire a projection matrix of the plane where the current display picture is located.

[0122] A clipping coordinate acquisition unit is configured to convert the current position coordinate of the target window into a clipping coordinate based on the projection matrix.

[0123] A projection coordinate acquisition unit is configured to convert the clipping coordinate into a coordinate in the plane where the current display picture is located based on a screen resolution of the current display picture, so as to obtain the projection coordinate.

[0124] In an embodiment, the screen following device 300 of the smart wearable device further comprises a picture state identification module, and the picture state identification module specifically comprises:

[0125] A data acquisition unit is configured to, when the current display picture is in a static state, acquire a data amount in the inertial measurement data set and a picture refresh frequency of the current display picture.

[0126] A static time calculation unit is configured to calculate a static time of the current display picture based on the data amount and the picture refresh frequency.

[0127] In an embodiment, the picture state identification module specifically further comprises:

[0128] A data acquisition unit is configured to, when the current display picture is refreshed once, acquire inertial measurement data of an inertial measurement unit (IMU), so as to obtain an inertial measurement data set.

[0129] a state determining unit, configured to determine whether the current display picture is in a static state based on data variation of each inertial measurement data in the inertial measurement data set;

[0130] a static state determining unit, configured to determine that the current display picture is in the static state when data variation of any two inertial measurement data in the inertial measurement data set is less than a data variation threshold.

[0131] In an embodiment, the moving path calculation module 302 comprises:

[0132] a deflection angle determining unit, configured to determine a deflection angle between the current position coordinate and the picture center coordinate based on the current position coordinate and the picture center coordinate;

[0133] a position offset angle calculation unit, configured to calculate a position offset angle of the target window at each picture refresh based on a picture refresh frequency of the current display picture, the deflection angle and a window moving time;

[0134] a moving path obtaining unit, configured to calculate at least one interpolation coordinate based on the position offset angle, and obtain the moving path.

[0135] In an embodiment, the position offset angle calculation unit comprises:

[0136] a time interval determining sub-unit, configured to determine a time interval of each picture refresh based on the picture refresh frequency;

[0137] a refresh number determining sub-unit, configured to determine a refresh number of the current display picture based on the time interval and the window moving time;

[0138] a position offset angle calculation sub-unit, configured to calculate a position offset angle of the target window at each picture refresh based on the refresh number and the deflection angle.

[0139] In an embodiment, the interpolation coordinate calculation module 303 comprises:

[0140] an interpolation coordinate calculation unit, configured to obtain at least one interpolation coordinate based on the moving path;

[0141] an animation rendering unit, configured to render the interpolation coordinate based on a rendering engine, and generate a window return animation;

[0142] a window moving unit, configured to move the target window to the interpolation coordinate based on the window return animation, until the target window is moved to the current display picture.

[0143] It should be noted that the skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the foregoing screen following method embodiments of the intelligent wearable device, and will not be described here.

[0144] The device provided in the above embodiments can be implemented in the form of a computer program, which can run on the computer device shown in FIG. 4.

[0145] Please refer to FIG. 4, which is a structural schematic block diagram of a computer device provided in an embodiment of the present application. The computer device can be a server.

[0146] Referring to FIG. 4, the computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory can include a non-volatile storage medium and an internal memory.

[0147] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, which, when executed, can cause the processor to perform any one of the screen following methods of the intelligent wearable device.

[0148] The processor is configured to provide computing and control capabilities to support the operation of the entire computer device.

[0149] The internal memory provides an environment for the running of the computer program in the non-volatile storage medium, which, when executed by the processor, can cause the processor to perform any one of the screen following methods of the intelligent wearable device.

[0150] The network interface is configured to perform network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that the structure shown in FIG. 4 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0151] It should be appreciated that the processor can be a central processing unit (CPU), the processor can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0152] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0153] obtaining a projection coordinate of a current position coordinate of the target window in a plane where the current display screen is located, and a screen center coordinate of the current display screen;

[0154] when the projection coordinate is located outside a coordinate region where the current display screen is located, calculating a moving path of the target window from the current position to the current display screen based on the current position coordinate and the screen center coordinate;

[0155] moving the target window from the current position to the current display screen based on the moving path, so as to display the target window in the current display screen.

[0156] In one embodiment, when implementing the step of obtaining the projection coordinate of the current position coordinate of the target window in the plane where the current display screen is located, the processor is configured to implement:

[0157] obtaining a projection matrix of the plane where the current display screen is located;

[0158] converting the current position coordinate of the target window into a clipping coordinate based on the projection matrix;

[0159] converting the clipping coordinate into a coordinate in the plane where the current display screen is located based on a screen resolution of the current display screen, to obtain the projection coordinate.

[0160] In one embodiment, before implementing the step of, when the projection coordinate is located outside the coordinate region where the current display screen is located, calculating the moving path of the target window from the current position to the current display screen based on the current position coordinate and the screen center coordinate, the processor is further configured to implement:

[0161] acquiring an amount of data in the inertial measurement dataset and a frame refresh frequency of the current display frame when the current display frame is in a static state;

[0162] calculating a static time of the current display frame based on the amount of data and the frame refresh frequency.

[0163] In an embodiment, before the processor implements the acquiring the amount of data in the inertial measurement dataset and the frame refresh frequency of the current display frame when the current display frame is in a static state, the processor is further configured to implement:

[0164] collecting inertial measurement data of an inertial measurement unit (IMU) to obtain an inertial measurement dataset when the current display frame is refreshed once;

[0165] judging whether the current display frame is in a static state based on a data variation of each of the inertial measurement data in the inertial measurement dataset;

[0166] determining that the current display frame is in the static state when a data variation of any two of the inertial measurement data in the inertial measurement dataset is less than a data variation threshold.

[0167] In an embodiment, before the processor implements the calculating the moving path of the target window from the current position to the current display frame based on the current position coordinate and the frame center coordinate, the processor is configured to implement:

[0168] determining a deflection angle between the current position coordinate and the frame center coordinate based on the current position coordinate and the frame center coordinate;

[0169] calculating a position offset angle of the target window at each frame refresh based on a frame refresh frequency of the current display frame, the deflection angle and a window moving time;

[0170] calculating at least one interpolation coordinate based on the position offset angle to obtain the moving path.

[0171] In an embodiment, before the processor implements the calculating a position offset angle of the target window at each frame refresh based on a frame refresh frequency of the current display frame, the deflection angle and a window moving time, the processor is configured to implement:

[0172] determining a time interval of each frame refresh based on the frame refresh frequency;

[0173] determining a refresh number of the current display frame based on the time interval and the window moving time;

[0174] Based on the refresh number and the deflection angle, a position offset angle of the target window at each time of picture refresh is calculated.

[0175] In an embodiment, when implementing the moving the target window from the current position to the current display picture based on the moving path to display the target window in the current view, the processor is configured to implement:

[0176] Based on the moving path, at least one interpolation coordinate is obtained;

[0177] Based on a rendering engine, the interpolation coordinate is rendered to generate a window return animation;

[0178] Based on the window return animation, the target window is moved to the interpolation coordinate until the target window is moved to the current display picture.

[0179] In an embodiment of the present application, a computer readable storage medium is also provided, which stores a computer program. The computer program includes program instructions. The processor executes the program instructions to implement any of the screen following methods of the intelligent wearable device provided in the embodiments of the present application.

[0180] The computer readable storage medium can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. The computer readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0181] The above merely illustrates the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed in the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A screen following method of a smart wearable device, the method comprising: obtaining a projection coordinate of a current position coordinate of a target window in a plane where a current display picture is located, and a picture center coordinate of the current display picture; when the projection coordinate is located outside a coordinate region where the current display picture is located, calculating a moving path of the target window from a current position to the current display picture based on the current position coordinate and the picture center coordinate; moving the target window from the current position to the current display picture based on the moving path, to display the target window in the current display picture. 2.The screen following method of the smart wearable device of claim 1, wherein, The obtaining of the projection coordinate of the current position coordinate of the target window in the plane where the current display picture is located comprises: obtaining a projection matrix of the plane where the current display picture is located; converting the current position coordinate of the target window into a clip coordinate based on the projection matrix; converting the clip coordinate into a coordinate in the plane where the current display picture is located based on a screen resolution of the current display picture, to obtain the projection coordinate. 3.The screen following method of the smart wearable device of claim 1, wherein, Before the calculating of the moving path of the target window from the current position to the current display picture based on the current position coordinate and the picture center coordinate when the projection coordinate is located outside the coordinate region where the current display picture is located, the method further comprises: when the current display picture is in a static state, obtaining a data amount in an inertial measurement data set and a picture refresh frequency of the current display picture; calculating a static time of the current display picture based on the data amount and the picture refresh frequency; when the static time of the current display picture is greater than or equal to a preset time length, determining a relative position between the projection coordinate and the current display picture based on the projection coordinate and the coordinate region where the current display picture is located, to determine whether the projection coordinate is in the coordinate region where the current display picture is located according to the relative position. 4.The screen following method of the smart wearable device according to claim 3, wherein, Before the obtaining of the data amount in the inertial measurement data set and the picture refresh frequency of the current display picture when the current display picture is in the static state, the method further comprises: collecting inertial measurement data of an inertial measurement unit (IMU) every time the current display picture is refreshed, to obtain the inertial measurement data set; when a data variation amount of any two inertial measurement data in the inertial measurement data set is less than a data variation threshold, determining that the current display picture is in the static state. 5.The screen following method of the smart wearable device according to claim 4, wherein, Before the determining that the current display picture is in the static state when the data variation amount of any two inertial measurement data in the inertial measurement data set is less than the data variation threshold, the method further comprises: collecting inertial measurement data of an inertial measurement unit (IMU) every time the current display picture is refreshed a fixed number of times, to obtain the inertial measurement data set. 6.The screen following method of the smart wearable device according to claim 4, wherein, The inertial measurement data comprises one or more of a three-axis attitude angle, an angular rate and an acceleration. 7.The screen following method of the smart wearable device according to claim 3, wherein, The calculating of the static time of the current display picture based on the data amount and the picture refresh frequency comprises: calculating a still time of a current display screen: Wherein, T' is the static time, G is the data amount in the inertial measurement data set when the current display picture is in the static state, F is the picture refresh frequency of the current display picture, the inertial measurement data is collected once every b times of refresh of the current display picture, and the ratio of b and F represents the collection time interval between adjacent two inertial measurement data. 8.The screen following method of the smart wearable device according to claim 1, wherein, The method further comprises: determining a deflection angle between the current position coordinate and the picture center coordinate based on the current position coordinate and the picture center coordinate; calculating a position offset angle of the target window at each picture refresh based on the picture refresh frequency of the current display picture, the deflection angle and a window moving time; calculating at least one interpolation coordinate based on the position offset angle; constructing the moving path based on the current position coordinate, the picture center coordinate and the at least one interpolation coordinate. 9.The screen following method of the smart wearable device according to claim 8, wherein, The method further comprises: calculating straight line distances between the current position coordinate, the picture center coordinate and a current origin as three side lengths of a triangle formed by the three coordinate points, wherein the current origin is a position coordinate of the smart wearable device; calculating an angle of a vertex at the current origin based on an angle calculation formula of the triangle and the three side lengths of the triangle formed by the three coordinate points to obtain the deflection angle. 10.The screen following method of the smart wearable device according to claim 8, wherein, The method further comprises: determining a time interval of each picture refresh based on the picture refresh frequency; determining a refresh number of the current display picture based on the time interval and the window moving time; calculating the position offset angle of the target window at each picture refresh based on the refresh number and the deflection angle. 11.The screen following method of the smart wearable device according to claim 8, wherein, The method further comprises: calculating an angle of the target window at each refresh to obtain an included angle between the interpolation coordinate and the current position coordinate at each refresh; determining a moving distance of the interpolation coordinate relative to the current position coordinate at each refresh based on a proportion of the included angle between the interpolation coordinate and the current position coordinate and the position deflection angle and a straight line length between the current position coordinate and the picture center coordinate; finding a corresponding point coordinate between the current position coordinate and the picture center coordinate as the interpolation coordinate based on the moving distance. 12.The screen following method of the smart wearable device according to claim 1, wherein, The method further comprises: obtaining at least one interpolation coordinate based on the moving path; generating a window return animation by rendering the interpolation coordinate based on a rendering engine; based on the window return animation, moving the target window to the interpolation coordinate until the target window is moved to the current display screen. 13.The screen following method of the smart wearable device according to claim 12, wherein, The moving the target window to the interpolation coordinate based on the window return animation until the target window is moved to the current display screen comprises: calculating the interpolation coordinate of the current refresh of the target window, and moving the target window to the interpolation coordinate of the current refresh; drawing the image of the current refresh; displaying the rendered image at the interpolation coordinate of the current refresh. 14.The screen following method of the smart wearable device according to claim 1, wherein, Before the obtaining the projection coordinate of the current position coordinate of the target window in the plane of the current display screen, the method further comprises: obtaining a starting position when the target user starts the smart wearable device, and obtaining an initial pose of the display screen of the smart wearable device; obtaining an initial position coordinate of the key point of the target window in the display screen of the initial pose; calculating the current position coordinate of the initial position coordinate of the target window in the world coordinate system corresponding to the three-dimensional display space of the smart wearable device according to the starting position and the current position of the smart wearable device. 15.The screen following method of the smart wearable device according to claim 1, wherein, Before the calculating the moving path of the target window from the current position to the current display screen based on the current position coordinate and the screen center coordinate when the projection coordinate is located outside the coordinate region of the current display screen, the method further comprises: determining the coordinate region range of the current display screen according to the screen size of the current display screen and the screen coordinate of the screen center of the current display screen; comparing the relative positions of the projection coordinate and the boundary points of the coordinate region range to obtain the relative position of the projection coordinate and the current display screen; judging whether the projection coordinate is located in the coordinate region of the current display screen based on the relative position of the projection coordinate and the current display screen. 16.The screen following method of the smart wearable device according to claim 15, wherein, The judging whether the projection coordinate is located in the coordinate region of the current display screen based on the relative position of the projection coordinate and the current display screen comprises: if the horizontal distance between the screen coordinate of the screen center and the projection coordinate is greater than half of the width of the screen size, it is determined that the projection coordinate is located outside the coordinate region range of the current display screen; or if the vertical distance between the screen coordinate of the screen center and the projection coordinate is greater than half of the height of the screen size, it is determined that the projection coordinate is located outside the coordinate region range of the current display screen; or if the horizontal distance between the screen coordinate of the screen center and the projection coordinate is greater than half of the width of the screen size and the vertical distance is greater than half of the height of the screen size, it is determined that the projection coordinate is located outside the coordinate region range of the current display screen. 17.The screen following method of the smart wearable device according to claim 1, wherein, The obtaining the screen center coordinate of the current display screen comprises: obtaining the current display screen corresponding to the current field of view of the target user when the target user wears the smart wearable device; calculating the three-dimensional coordinate of the center point of the current display screen relative to the eye original position of the target user with the position of the smart wearable device as the origin, and obtaining the screen center coordinate of the current display screen.

18. A screen following device of a smart wearable device, the screen following device of the smart wearable device comprising: The coordinate obtaining module is configured to obtain a picture center coordinate of a current display picture and a projection coordinate of a current position coordinate of a target window in a plane where the current display picture is located; The moving path calculating module is configured to, when the projection coordinate is located outside a coordinate region where the current display picture is located, calculate a moving path of the target window from the current position to the current display picture based on the current position coordinate and the picture center coordinate; The interpolation coordinate calculating module is configured to move the target window from the current position to the current display picture based on the moving path, so as to display the target window in the current display picture. 19.A computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the computer program, when executed by the processor, implements steps of the screen following method of the smart wearable device according to any one of claims 1 to 17. 20.A computer readable storage medium, having a computer program stored therein, wherein the computer program, when executed by a processor, implements steps of the screen following method of the smart wearable device according to any one of claims 1 to 17.

Images