Method and device for controlling motion of focus window of intelligent glasses

Abstract

The invention provides a method and device for controlling motion of a focus window of smart glasses. The method comprises the following steps: in response to the rotating motion of the intelligent glasses, determining that the field of view of a camera of the intelligent glasses rotates; the focus window is controlled to move towards the view field center position of the current view field of the camera along a movement spherical surface with the camera as the center and with the radius of a preset value, and the movement of the focus window comprises the process of acceleration first and then deceleration.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a method, device, electronic device, and medium for controlling the movement of a focus window of smart glasses. Background Art

[0002] As an emerging technology product, smart glasses (e.g., augmented reality (AR) glasses or mixed reality (XR) glasses, etc.) are also important factors in the competitiveness of the product in terms of interaction and UI design. The interaction and display methods between smart glasses and the focus (flat windows, 3D models, etc.) in the virtual world are also hotly discussed aspects in the industry. For example, the image acquisition device (e.g., camera) provided on the smart glasses can be used to capture scenes and / or user interaction gestures in the real environment, etc., and feed them back to the processor in the smart glasses for corresponding rendering processing, so as to generate relevant information that can be displayed using the optical machine (display hardware) of the smart glasses. During the rendering process, an AR virtual space can be constructed, and the image displayed on the display screen of the smart glasses is a virtual image generated in the AR virtual space, and the virtual image ultimately seen by the user is related to the field of view (FOV) of the camera and the distance of the virtual image.

[0003] In the application of smart glasses, there is a focus in the virtual world that the smart glasses interact with (e.g., a flat window or a 3D model, etc.). The focus window can be used to provide enhanced information or enable user interaction, etc. The focus window can be considered as the current active window or interactive window. Ideally, after being rendered, it can occupy a part of the display screen of the smart glasses and cover the displayed image (associated with the camera's FOV), so that useful information can be presented to the user or better interaction can be achieved. Summary of the invention

[0004] According to one aspect of the present application, a method for controlling the movement of a focus window of smart glasses is provided. The method may include: in response to the rotational movement of the smart glasses, determining that the field of view of the camera of the smart glasses undergoes rotational movement; and controlling the focus window to move along a moving sphere with a radius having a preset value and centered on the camera, toward the center position of the field of view of the current field of view of the camera; wherein the movement of the focus window includes a process of first accelerating and then decelerating.

[0005] According to an embodiment of the present application, the movement of the focus window is initiated in response to at least a portion of the focus window being outside a current field of view of the camera.

[0006] According to an embodiment of the present application, the movement of the focus window is controlled based on an elastic rope model, wherein one end of the elastic rope of the elastic rope model is connected to the center position of the field of view of the current field of view of the camera, and the other end is connected to the window center position of the focus window, wherein the window center position is updated according to each frame.

[0007] According to an embodiment of the present application, the focus window is controlled to move along a moving sphere that is closely attached to the camera and has a preset radius, toward the center position of the field of view of the current field of view of the camera, including: determining the elastic force of the elastic rope between the center position of the field of view and the center position of the window at the start time of the current frame; determining the arc displacement of the focus window within the current frame based on the elastic force and the movement speed of the focus window at the start time of the current frame, the arc displacement being a portion of the arc between the center position of the field of view and the center position of the window at the start time of the current frame on the moving sphere; determining the window center position of the focus window at the end time of the current frame based on the arc displacement; and moving the focus window to the determined window center position of the focus window at the end time of the current frame.

[0008] According to an embodiment of the present application, the arc displacement of the focus window within the current frame is determined based on the elastic force and the movement speed of the focus window at the start time of the current frame, including: determining the acceleration of the focus window within the frame period corresponding to the current frame based on the elastic force and a preset maximum static friction resistance; determining the movement speed at the end time of the current frame and the arc displacement within the frame period corresponding to the current frame based on the movement speed of the focus window at the start time of the current frame and the acceleration.

[0009] According to an embodiment of the present application, determining the window center position of the focus window at the end moment of the current frame based on the arc displacement includes: determining the ratio of the arc displacement to the arc on the moving sphere between the field of view center position and the window center position at the start moment of the current frame; and determining the window center position of the focus window at the end moment of the current frame based on the field of view center position, the window center position and the ratio.

[0010] According to an embodiment of the present application, determining the window center position of the focus window at the end time of the current frame includes: based on a first vector between the position of the camera and the center position of the field of view at the start time of the current frame, a second vector between the position of the camera and the window center position at the start time of the current frame, and the ratio, determining the vector between the position of the camera and the window center position of the focus window at the end time of the current frame as the window center position of the focus window at the end time of the current frame.

[0011] According to an embodiment of the present application, controlling the focus window to move along a moving sphere with a preset radius closely centered on the camera, toward the center position of the field of view of the current field of view of the camera, also includes: using an interpolation algorithm, based on a first rotation amount corresponding to the window center position of the focus window at the start time of the current frame and a second rotation amount corresponding to the position when it coincides with the center position of the field of view, determining the rotation amount of the focus window at the end time of the current frame; and rotating the focus window to the determined rotation amount at the end time of the current frame.

[0012] According to an embodiment of the present application, determining the rotation amount of the focus window at the end moment of the current frame includes: determining an interpolation ratio based on the movement speed of the focus window at the start moment of the current frame, wherein the greater the movement speed, the smaller the interpolation ratio; and based on the interpolation ratio, determining the rotation amount of the focus window at the end moment of the current frame according to the first rotation amount and the second rotation amount.

[0013] According to an embodiment of the present application, the interpolation algorithm is based on the following quaternion interpolation formula:

[0014]

[0015] Wherein, q0 is the first rotation amount expressed in quaternion corresponding to the window center position of the focus window at the start time of the current frame, q1 is the second rotation amount expressed in quaternion when the focus window is expected to be at the center position of the field of view at the start time of the current frame, and q delta is the interpolation ratio, and θ q It is the angle between a first vector between the position of the camera and the center position of the field of view at the start time of the current frame and a second vector between the position of the camera and the center position of the window of the focus window at the start time of the current frame.

[0016] According to an embodiment of the present application, the functional relationship between the interpolation ratio and the motion speed at the start time of the current frame is:

[0017] qdelta =1-(-m -V +1) n

[0018] Among them, V is the motion speed at the beginning of the current frame, q delta is the interpolation ratio, and m and n are empirical values.

[0019] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors; and one or more memories storing instructions executable by the processors, wherein when the processor executable instructions are executed by the one or more processors, the one or more processors execute the method as described above.

[0020] According to another aspect of the present application, a computer program product is provided, on which a computer program is stored. When the computer program is executed by one or more processors, the method described above is implemented.

[0021] According to another aspect of the present application, a non-volatile computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by one or more processors, the method described above is implemented.

[0022] According to various embodiments of the present application, by controlling the movement of the focus window on the moving sphere as the camera rotates, the user can see the focus window outside the field of view quickly move into the field of view through the display screen of the smart glasses. In addition, the position of the focus window at the end of each frame (for example, the three-dimensional coordinates in the world coordinate system) and the rotation amount can be determined by the elastic rope model, and the focus window is controlled according to the position and rotation amount that the focus window should have at the end of each frame, so that the window center position of the presented focus window can be relatively close to the center position F of the field of view and basically face the observer, thereby achieving the effect of elastic animation, so that when the focus window is outside the field of view, it will accelerate first and then decelerate smoothly to move into the field of view. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings illustrate various embodiments of various aspects of the present application, and together with the description, they are used to explain the principles of the present application. It is understood by those skilled in the art that the specific embodiments shown in the accompanying drawings are only exemplary, and they are not intended to limit the scope of the present application.

[0024] Figure 1A A schematic diagram showing an exemplary environment for application of smart glasses according to an embodiment of the present disclosure.

[0025] Figure 1BA schematic diagram of the scene and a display preview effect diagram are shown when the focus window is within the viewing cone in front of the camera.

[0026] Figure 1C A schematic diagram of a scene and a display preview effect diagram are shown when the focus window is outside the viewing cone in front of the camera.

[0027] Figure 2 A schematic flow chart of a method for controlling the movement of a focus window of smart glasses according to an embodiment of the present application is shown.

[0028] Figure 3 A schematic diagram of the elastic cord model is shown.

[0029] Figure 4 The embodiment according to the present application is shown Figure 2 A sub-step of the step of controlling the movement of the focus window.

[0030] Figure 5 A schematic diagram showing the movement of a focal window around a moving sphere according to an embodiment of the present application is shown.

[0031] Figure 6 The figure shows the function curve of the interpolation ratio changing with the movement speed when m=1.5 and n=2.

[0032] Figure 7 A structural block diagram of a device for controlling the movement of a focus window of smart glasses according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] Figure 1A A schematic diagram showing an exemplary environment for application of smart glasses according to an embodiment of the present disclosure.

[0035] refer to Figure 1A , an exemplary environment 100 is shown. The exemplary environment 100 includes smart glasses 102, a network 104 (optional), and a server 106 (optional). The smart glasses 102 may include an image acquisition device 102a and a display device 102b. The image acquisition device 102a may be a camera or other image sensor. The smart glasses 102 and the server 106 may be communicatively coupled via the network 104. Optionally, the smart device may also include one or more processors 102c.

[0036] Further shown is the field of view (hereinafter referred to as "FOV") 108 of the image acquisition device 102a and the focus window 110 located within the FOV 108, which can be presented on the display screen of the display device. The FOV parameters of the image acquisition device 102a (including horizontal FOV parameters (such as an angle of 60°) and vertical FOV parameters (such as an angle of 60°)), the aspect ratio of the display screen of the display device 102b, the virtual image distance (the distance from the image acquisition device when the focus window is directly in front of the image acquisition device in the virtual space), and the size of the focus window, etc., can be preset. The image acquisition device 102a can move with the movement of the smart glasses (for example, the rotation of the head of the user wearing the smart glasses), so the orientation can change. When the smart glasses are not moved, for example, the corresponding positions of the FOV and the focus window in the coordinate system of the virtual space (for example, the world coordinate system) are determined according to the initial posture of the image acquisition device and various preset parameters.

[0037] The image acquisition device 102a can capture a sequence of image frames (eg, video) within the FOV. The smart glasses 102 can also be configured to generate media content based on the captured sequence of image frames for display on a display screen of a display device.

[0038] The display device 102b may include appropriate logic, circuitry, and interfaces that may be configured to display a preview (live or recorded) of a sequence of image frames of the FOV and further overlay a focus window on the FOV. Examples of display devices may include, but are not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, a plasma display, or an organic LED (OLED) display technology, or other display devices. According to one embodiment, the display device may be a smart glass device, a see-through display, a projection-based display, an electrochromic display, or a transparent display, and the like.

[0039] The server 106 may include appropriate logic, circuitry, and interfaces, and image frames captured by the image processing device of the smart glasses may be sent to the server for processing, or media content generated by the smart glasses may be stored on the server 108 and / or transmitted to a display device. Alternatively, the processor 102c in the smart glasses 102 and the server 106 may individually or in combination implement a scheme for controlling the movement of the focus window as will be described herein. For example, the processor 102c of the smart glasses may independently process the image frames captured by the image acquisition device, control the position of the focus window, and generate an image to be displayed on the display device, etc., without intervention by the server.

[0040] The server 108 may be implemented as a cloud server that may perform operations via web applications, cloud applications, http requests, repository operations, file transfers, etc. Other examples of the server 106 may include, but are not limited to, a database server, a file server, a web server, a media server, an application server, a host server, a cloud server, or other types of servers.

[0041] like Figure 1B As shown in FIG. 1 , when the focus window (currently active window or interactive window) is within the camera's front viewing cone (PABCD cone) and is at a certain distance from the camera (i.e., the virtual image distance, which can be predetermined), the focus window is visible on the display screen after rendering, and ideally, the center position of the rendered focus window (E, referred to as the window center position herein) can coincide with the center position of the field of view range on the display screen corresponding to the camera's field of view (F, referred to as the field of view center position herein), as shown in FIG. Figure 1B as shown in .

[0042] At present, the focus window position in most smart glasses on the market is fixed in the world coordinate system. Since the field of view of the camera of the smart glasses moves (rotates) with the movement of the user's head (for example, rotation), the focus window with a fixed position may exist outside the field of view of the camera of the smart glasses, so that at least a part of it may exist outside the display screen of the smart glasses when it is displayed on the display screen, such as Figure 1C As shown, the focus window is not completely located in the display screen, so it cannot interact with the user well or the display effect is poor.

[0043] Therefore, a solution is needed that enables the focus window to remain within the field of view of the camera of the smart glasses even when the smart glasses rotate with the user's head.

[0044] In addition, when the user's head rotation causes at least a portion of the focus window to be not on the display screen, the position of the focus window (for example, the position in the world coordinate system) needs to be adjusted so that it can be displayed on the display screen, and when adjusting the position of the focus window, motion smoothness and visual comfort also need to be considered to achieve a better user experience.

[0045] Therefore, an embodiment of the present application provides a solution that enables at least a portion of the focus window to move smoothly into the field of view (or display screen) of the camera when it is outside the field of view (which may also be equivalent to the display screen), thereby greatly improving the motion smoothness and visual comfort.

[0046] Figure 2 FIG. 1 is a flow chart showing a method for controlling the movement of a focus window of smart glasses according to an embodiment of the present application. For example, the method may be Figure 1A The method is executed by a processor in the smart glasses 102 shown in the figure or a server 106 connected to the smart glasses 102 via a network. When the method is executed by the server, the smart glasses can send relevant information (for example, preset parameters as described later and positions and rotation amounts associated with each display frame, etc.) to the server via the network, and receive control information associated with the movement of the focus window from the server and generate display information based on the control information, etc.

[0047] like Figure 2 As shown, in step S210, in response to the smart glasses undergoing rotational movement, it is determined that the field of view of the camera of the smart glasses undergoes rotational movement.

[0048] For example, the smart glasses are worn on the user's head, so when the user's head rotates, the smart glasses will rotate accordingly, and the camera of the smart glasses will also rotate. Since the field of view of the camera is generally directly in front of the human eye, it will also rotate.

[0049] Optionally, the rotational movement of the smart glasses (or its camera) can be sensed by one or more sensors. The rotational movement of the camera can be a rotational movement that can be represented by a rotation quaternion. The fixed position of the camera in the world coordinate system is P (Px, Py, Px), and the initial orientation of the camera is a vector D (Dx, Dy, Dz).

[0050] In step S220, the focus window is controlled to move along a moving sphere with a preset radius centered on the camera toward the center position of the current field of view, wherein the movement of the focus window includes a process of accelerating first and then decelerating.

[0051] Optionally, the movement of the focus window (the width and height of the focus window are both preset values) can be controlled regardless of the rotation amplitude of the smart glasses (and its camera). For example, the focus window can follow the movement of the smart glasses. As long as the smart glasses rotate, the focus window is controlled to move toward the center of the field of view; or in other embodiments, in order to reduce the amount of calculation of the processing device in the smart glasses, the position of the focus window can be adjusted only when at least a part of the focus window in the smart glasses is outside the current field of view of the camera. In this way, when the smart glasses (and its camera) rotate, if the rotation amplitude is small, even if the focus window is not controlled to follow the rotation of the smart glasses, the moving focus window can still be within the current field of view of the camera (the field of view after the current rotation); of course, if the rotation amplitude of the smart glasses (and its camera) is large, at least a part of the focus window will be outside the current field of view of the camera, so it is necessary to control it to move into the current field of view of the camera. Therefore, it is possible to determine whether the movement of the focus window needs to be controlled by setting a rotation amplitude threshold.

[0052] Optionally, when the smart glasses are not rotating, in the virtual space, the center position of the focus window is substantially coincident with the center position of the camera's field of view (e.g. Figure 1B As shown in the figure); after the smart glasses rotate, the rotation quaternion of the camera increases by a certain amount, and since the focus window is generally fixed in the world coordinate system, the center position of the focus window will deviate from the center position of the field of view of the camera after rotation by a certain distance (as ... Figure 1C As shown in the figure, the position of the focus window is adjusted by controlling the movement of the focus window so that it moves along with the rotation of the camera and does not deviate outside the field of view of the rotated camera.

[0053] In addition, the user's head rotation will generally stop, that is, the center position of the field of view may no longer change after a certain moment, and the focus window will move closer to the fixed center position of the field of view. Since the focus window needs to be presented in a part of the display screen of the smart glasses as the displayed focus window image, the movement process of the focus window needs to be as smooth as possible to improve visual comfort, so an effect similar to elastic animation can be achieved by accelerating first and then decelerating.

[0054] For example, the position update of the focus window can be updated with the display frame rate of the image (e.g., from a camera) on the display screen of the smart glasses, so the time interval for updating the position of the focus window is very small. In addition, the camera may also be in continuous rotation, and the center position of its field of view may also be updated accordingly. In some embodiments, the movement of the focus window can be controlled after the camera stops rotating (detecting that the position no longer changes within a preset time period), that is, after the center position of the field of view is fixed, and the center position of the field of view at the beginning of each frame can be considered to be the same position or a fixed position, that is, the center position of the field of view after the camera stops rotating, and the center position of the window is updated with the movement of the focus window. In addition, in other embodiments, the movement of the focus window can be controlled while the camera is rotating, and the center position of the field of view and the center position of the window at the beginning of each frame will be updated.

[0055] Through the method of controlling the movement of the focus window of the present application, the user can see through the display screen of the smart glasses that the focus window outside the field of view moves quickly into the field of view.

[0056] Optionally, in order to realize the movement process of the focus window around the moving sphere that first accelerates and then decelerates, an elastic rope model can be introduced. The elastic rope model takes into account the idea that the elastic force will change with the stretching length of the elastic rope, and the speed of the target object can be controlled by using this change, and introduces it into the process of controlling the movement of the focus window. For example, the model can be built as follows: one end of the elastic rope of the elastic rope model is connected to the center position F of the field of view of the current field of view of the camera, and the other end is connected to the window center position E of the focus window, wherein the window center position can be updated according to each frame. Therefore, the stretching length of the elastic rope at the beginning of different frames may be different, so the elastic force can be calculated for each frame. In addition, the maximum static friction resistance Fn between the focus window and the moving sphere is also introduced, and the mass M of the focus window is small, so the preset inertia force positively correlated with the mass is much smaller than Fn, for example, the difference is more than 20 times, so that even if the speed of the focus window is large, the focus window will quickly decelerate to stillness when the rotational movement of the camera stops, and there will be no phenomenon of exceeding the center position of the field of view or jumping back and forth. Figure 3 A schematic diagram of the elastic rope model is shown, which shows that the tension of the elastic rope is Fp, which can act on the weight together with the friction force (ie, the maximum static friction resistance) Fn.

[0057] The following combination Figure 4-Figure 7 More details of controlling the movement of the focus window based on an elastic rope model are described.

[0058] Figure 4 The embodiment according to the present application is shown Figure 2The following sub-steps are performed for each frame after it is determined that the movement of the focus window needs to be controlled, that is, for different frames, the current center position of the field of view and the center position of the window may change.

[0059] like Figure 4 As shown, in sub-step S220-1, for the start time of the current frame, the elastic force of the elastic rope between the center position of the field of view and the center position of the window at the start time of the current frame is determined.

[0060] For example, the deformation length of the elastic rope can be determined based on the length of the straight line or arc between the center position of the field of view at the start time of the current frame (in context, the center position of the field of view may be a fixed position, such as performing the operation after the camera rotation stops, or it may be an updated position, such as performing the operation during the camera rotation movement) and the center position of the window at the start time of the current frame and the original length of the elastic rope; and the elastic force is determined based on the elastic coefficient of the elastic rope and the deformation length of the elastic rope.

[0061] For example, the total length of the elastic rope and the elastic coefficient of the elastic rope model are known, and the center position of the field of view and the center position of the window at the beginning of each frame (the initial position at the first display frame or the updated position thereafter (as will be described later)) can be determined, so the stretching length of the elastic rope at the beginning of the current frame can be determined based on these two positions.

[0062] For example, when the current frame starts, the two ends of the elastic rope are connected to the center position F of the field of view and the center position E of the window, respectively. The elastic force Fe between position F and position E is:

[0063] Fe=K×d L (1)

[0064] Where K is the elastic coefficient, d L is the elastic rope stretch length, d L =S L -L,S L is the total length of the elastic rope, S L The length of the straight line from F to E can be used to approximate the calculation:

[0065]

[0066] Or to calculate the length of an arc:

[0067]

[0068] vector is the vector between the camera position P and the window center position E at the start of the current frame. is the vector between the camera position P and the center position F of the field of view at the beginning of the current frame, R is the distance of the focus window from the camera when it is directly in front of the camera (i.e., the preset virtual image distance), and the surface on which the focus window moves around the camera is called the motion sphere and its radius is R.

[0069] When S L When using straight line length, the straight line length L between position F and position E can be calculated; when S L When the arc length is used, the arc length L between position F and position E can be calculated.

[0070] In sub-step S220-2, based on the elastic force and the movement speed of the focus window at the start time of the current frame, the arc displacement of the focus window within the current frame is determined, and the arc displacement is a part of the arc between the center position of the field of view and the center position of the window on the moving sphere at the start time of the current frame.

[0071] For example, when the current frame is the first display frame after the focus window starts to move, the movement speed of the focus window at the start of the current frame is 0; when the current frame is the second display frame after the focus window starts to move and the display frames thereafter, the movement speed of the focus window at the start of the current frame is the movement speed at the end of the previous frame.

[0072] For example, in the previous steps, the elastic force of the elastic rope at the start time of the current frame is determined, and the position of the focus window (and / or the center position of the field of view) is updated as the display frame is refreshed, and the corresponding elastic force is also updated as the display frame is refreshed. Therefore, it can be considered that the elastic force in each frame period is constant, but the elastic force between different frames may change. Therefore, the movement of the focus window in each frame period can be considered as a movement based on the action of a fixed elastic force. For example, the acceleration of the focus window in the frame period corresponding to the current frame can be determined based on the elastic force and the preset maximum static friction resistance, and the movement speed at the end time of the current frame and the arc displacement within the frame duration corresponding to the current frame are determined based on the movement speed of the focus window at the start time of the current frame and the acceleration. That is, the focus window performs uniformly accelerated motion in the frame period corresponding to the current frame. For example, when the acceleration is greater than 0, the focus window is in accelerated motion, and when the acceleration is less than or equal to 0, the focus window decelerates or stops.

[0073] For example, in the frame period corresponding to the current frame, the acceleration (a) of the focus window is a=(Fe-Fn) / M, and the real-time speed (V) of the focus window is V=V0+a×t, Fe is the determined elastic force, Fn is the maximum static friction resistance preset in the elastic rope model (which can be set to be much larger than the inertia force of the focus window (based on the preset mass M of the focus window in the elastic rope model)), so that even if the speed of the focus window is large, the focus window will quickly decelerate to a stationary state when the camera stops rotating, and will not exceed the center position of the field of view or jump back and forth; V0 is the movement speed at the start time of the current frame, t is the real-time duration elapsed in the current frame and does not exceed the duration T of the frame period. Therefore, it can be determined that the movement speed of the focus window at the end time of the current frame is V=V0+a×T, and the movement displacement is S=VT+1 / 2*a*T 2 To simplify the calculation, the motion displacement can also be calculated approximately according to S=VT. Since the focus window moves around the moving sphere toward the center of the field of view, the motion displacement is on the arc between the center of the window and the center of the field of view, such as Figure 5 shown.

[0074] In sub-step S220-3, the window center position of the focus window at the end moment of the current frame is determined based on the arc displacement.

[0075] For example, the arc displacement and the center position of the field of view on the spherical surface at the start time of the current frame (eg Figure 5 F1) and the window center position (e.g. Figure 5 The ratio of the arc between E1) on the moving sphere, and based on the center position of the field of view, the center position of the window and the ratio, the window center position of the focus window at the end of the current frame is determined (for example Figure 5 of E2).

[0076] Optionally, the position of the camera may be based on Figure 5 P) and the center position of the field of view at the beginning of the current frame (e.g. Figure 5 F1), the first vector between the position of the camera (e.g. Figure 5 P) and the window center position at the beginning of the current frame (e.g. Figure 5 The second vector between the position of the camera and the window center position of the focus window at the end time of the current frame and the ratio are determined as the window center position of the focus window at the end time of the current frame (e.g. Figure 5 of F2).

[0077] For example, the above process may be performed based on an interpolation algorithm, and thus the determined ratio may be used as an interpolation ratio.

[0078] For example, the interpolation scale could be:

[0079]

[0080] Among them, the arc Total length is S L The focus window moves along the frame period corresponding to the current frame The length to be advanced is S, then delta is the displacement of the motion in the frame period corresponding to the current frame. The ratio of the total length of the arc, and the value is between 0 and 1.

[0081] Alternatively, the interpolation formula can be as follows:

[0082]

[0083] The interpolation ratio delta∈[0,1], when delta=0, When delta=1, θ is the same as formula (3), and The angle between.

[0084] That is and The interpolation ratio between them is substituted into formula (4) to calculate the new position of the focus window at the end of the current frame (3D coordinates in the world coordinate system):

[0085] newP=v(delta) (6)

[0086] In sub-step S220-4, the focus window is moved to the determined window center position of the focus window at the end time of the current frame, so as to generate and display the focus window image of the current frame.

[0087] In the above sub-steps, the position that the focus window needs to reach at the end of the current frame is determined, so the content that needs to be presented in the focus window on the display screen for the current frame can be generated at the determined position by internally executing the rendering process to obtain the focus window image. In addition, the window center position of the focus window at the end of the current frame can be used as the window center position at the start of the next frame to determine the updated position of the focus window for the next frame using a similar method as above.

[0088] Optionally, after determining the position that the focus window needs to reach at the end of the current frame (eg Figure 5After E2), since the focus window should face the operator and the focus window needs to be tangent to the sphere when it moves along the moving sphere with a preset radius centered on the camera, the focus window's position in the coordinate system (e.g., the world coordinate system) in the virtual space will change, that is, it will rotate, and its position at each position during the movement can be represented by a rotation quaternion (corresponding to the rotation amount). Figure 5 As shown, the focus window moves from the window center position E1 to a position that coincides with the field of view center position F1, and the rotation amount corresponding to the posture of the focus window changes.

[0089] Assumptions Figure 5 The position E1 in is the window center position of the focus window determined at the start time of the current frame, the position E2 is the window center position of the focus window determined at the end time of the current frame, and the position F1 is the field of view center position determined at the start time of the current frame. The rotation amount of the focus window at position E1 (related to the posture and expressed as a rotation quaternion) is called the first rotation amount and can be the initial rotation amount (for the start time of the first frame, at this time, the window center position coincides with the field of view center position of the camera, and can be consistent with the rotation quaternion of the camera when the field of view center position of the camera is at this position, which is a known value) or the rotation amount of the focus window determined at the end of the previous frame, and the rotation amount at position F1 is called the second rotation amount and can be consistent with the rotation quaternion of the camera when the field of view center position of the camera is at this position, so based on the known first rotation amount and second rotation amount, the rotation amount of the focus window at position E2 at the end time of the current frame can be determined.

[0090] For example, the rotation amount of the focus window that makes the focus window tangent to the moving sphere at the end of the current frame can be determined, and the rotation amount is associated with the posture of the focus window. Optionally, an interpolation algorithm can be used to determine the rotation amount of the focus window at the end of the current frame (equivalent to the posture of the focus window or the rotation quaternion) based on the interpolation ratio, the window center position of the focus window and the center position of the field of view at the beginning of the current frame, the first rotation amount at the window center position of the focus window (position E1) at the beginning of the current frame, and the second rotation amount at the position when it coincides with the center position of the field of view (position F1). Then, at the end of the current frame, the focus window is rotated to have the determined rotation amount to generate and display the focus window image of the current frame.

[0091] Optionally, the interpolation algorithm needs to determine the interpolation ratio, which cannot be determined by the displacement-related interpolation delta determined by the above formula (4). This is because when the camera rotates in one direction, due to the existence of the maximum friction resistance, the center position of the focus window may maintain a certain distance from the center position of the field of view even under the tension of the elastic rope, and the camera cannot move over even if it stops rotating. Therefore, until the focus window is stationary, the displacement interpolation delta is much smaller than 1. However, for the rotation amount, when the focus window slides to a stationary state, it should basically face the observer, that is, the rotation amount is close to the second rotation amount. Therefore, the interpolation value of this step needs to be close to 1, and therefore needs to be re-determined.

[0092] Considering that when the focus window decelerates and slides to a standstill, the speed V gradually approaches 0, so the speed can be used to affect the rotation interpolation q of the focus window delta The faster the speed (for example, when the focus window just starts to move), the higher the interpolation ratio q delta The closer it is to 0, the slower the speed (for example, the focus window is about to finish moving), delta The closer to 1.

[0093] Therefore, the interpolation ratio can be determined based on the movement speed of the focus window associated with the current frame. Optionally, considering that the movement speed at the start of the current frame has one less estimation calculation process than the movement speed at the end, the interpolation ratio can be determined based on the movement speed at the start of the current frame, wherein the greater the movement speed, the smaller the interpolation ratio. Then, based on the interpolation ratio, the rotation amount of the focus window at the end of the current frame can be determined according to the first rotation amount and the second rotation amount.

[0094] For example, the interpolation algorithm is based on the following quaternion interpolation formula:

[0095]

[0096] Wherein, q0 is the first rotation amount represented by the rotation quaternion of the center position of the focus window at the start time of the current frame, q1 is the second rotation amount represented by the rotation quaternion when the focus window is expected to be at the center position of the field of view at the start time of the current frame (that is, consistent with the rotation quaternion corresponding to the center position of the field of view of the camera at the start time of the current frame), q delta is the interpolation ratio, and θ q It is the angle between a first vector between the position of the camera and the center position of the field of view at the start time of the current frame and a second vector between the position of the camera and the center position of the window in the focus window at the start time of the current frame.

[0097] In addition, as an example, the functional relationship between the interpolation ratio and the motion speed at the start time of the current frame is:

[0098] q delta =1-(-m -V +1) n

[0099] Among them, V is the motion speed at the beginning of the current frame, q delta is the interpolation ratio, and m and n are empirical values.

[0100] For example, Figure 6 The figure shows the function curve of the interpolation ratio changing with the movement speed when m=1.5 and n=2, where the horizontal axis represents the movement speed V and the vertical axis represents the interpolation ratio q delta , you can adjust the values ​​of m and n to make the focus window rotation motion more natural and beautiful.

[0101] Therefore, the position (for example, the three-dimensional coordinates in the world coordinate system) and the rotation amount of the focus window at the end time of each frame can be determined, and the focus window can be controlled according to the position and the rotation amount that the focus window should have at the end time of each frame, so that the window center position of the presented focus window can be relatively close to the center position of the field of view and basically face the observer, thereby achieving the effect of elastic animation, so that when the focus window is outside the field of view, it will accelerate first and then decelerate smoothly to move into the field of view, and its movement smoothness and visual comfort are greatly improved.

[0102] According to another aspect of the present application, a device for controlling the movement of a focus window of smart glasses is also provided.

[0103] Figure 7 A structural block diagram of a device for controlling the movement of a focus window of smart glasses according to an embodiment of the present application is shown. Figure 7 The device shown can be used as Figure 1A The server shown in or is part of or includes Figure 1A The server shown.

[0104] like Figure 7 As shown, the apparatus 700 may include a motion determination module 710 and a motion control module 720 .

[0105] The motion determination module 710 can be used to determine that the field of view of the camera of the smart glasses undergoes rotational motion in response to the smart glasses undergoing rotational motion. The motion control module 720 can be used to control the focus window to move along a motion sphere with a preset radius centered on the camera toward the center position of the current field of view of the camera; wherein the movement of the focus window includes a process of first accelerating and then decelerating.

[0106] Optionally, the motion control module 720 may initiate the motion of the focus window in response to at least a portion of the focus window being outside a current field of view of the camera.

[0107] Optionally, the motion control module 720 can be used to control the movement of the focus window based on an elastic rope model, wherein one end of the elastic rope of the elastic rope model is connected to the center position of the field of view of the current field of view of the camera, and the other end is connected to the window center position of the focus window, wherein the window center position is updated according to each frame.

[0108] Optional, such as Figure 7 As shown in , the motion control module 720 can be further divided into an elastic force determination unit 720 - 1 , a displacement determination unit 720 - 2 , a position determination unit 720 - 3 and a movement unit 720 - 4 .

[0109] For example, the elastic force determination unit 720-1 may be configured to determine the elastic force of the elastic rope between the center position of the field of view and the center position of the window at the start time of the current frame.

[0110] The displacement determination unit 720-2 can be configured to determine the arc displacement of the focus window within the current frame based on the elastic force and the movement speed of the focus window at the start time of the current frame, and the arc displacement is a part of the arc between the center position of the field of view and the center position of the window on the moving sphere at the start time of the current frame.

[0111] The position determination unit 720-3 may be configured to determine the window center position of the focus window at the end moment of the current frame based on the arc displacement.

[0112] The moving unit 720 - 4 may be configured to move the focus window to the determined window center position of the focus window at the end time of the current frame.

[0113] Figure 7 For more details on the various modules and units in the Figure 2-Figure 6 described, so it will not be repeated here.

[0114] In addition, although Figure 7 The above modules and submodules are shown in an exemplary manner, but it should be understood that the device 700 can be divided into more or fewer modules according to different functions, or each module can be divided into further more or fewer submodules. In some example embodiments, the module or its submodule can be implemented with electronic hardware (e.g., a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc.), computer software (e.g., can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable ROM (EPROM), etc.), or a combination of the two.

[0115] According to another aspect of the present application, an electronic device is also provided. The electronic device may include Figure 1A Smart glasses or server shown.

[0116] The electronic device may include one or more processors, one or more memories, and an optional network interface, input device, and display screen connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the terminal stores an operating system and may also store a computer executable program or a computer readable code. When the computer executable program or the computer readable code is executed by the processor, the processor may implement the reference described above. Figure 2-5 The internal memory may also store a computer executable program or a computer readable code, which, when executed by the processor, enables the processor to perform the reference operation described above. Figure 2-5 Various operations of the method are described.

[0117] The processor can be an integrated circuit chip with signal processing capabilities. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can be an X84 architecture or an ARM architecture.

[0118] The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable categories of memory.

[0119] The optional input device of the electronic device may be a touch layer covered on the display screen, a key, a trackball or a touch pad provided on the terminal housing, or an external keyboard, a touch pad or a mouse, etc. The display screen may be any applicable display screen in the current technology, such as a liquid crystal display screen, an organic light emitting diode (LED) display screen, etc.

[0120] According to another aspect of the present application, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor executes the reference Figure 2-5 The various operations of the described method. The computer readable storage medium may be a non-volatile storage medium.

[0121] According to another aspect of the present application, a computer program product is also provided, including a computer program, which implements the reference method as described above when the computer program is executed by a processor. Figure 2-5 Various operations of the method are described.

[0122] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the methods and devices according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains at least one executable instruction for realizing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0123] The exemplary embodiments of the present application described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present application, and such modifications should fall within the scope of the present application.

Claims

1. A method for controlling the movement of a focus window of smart glasses, comprising: In response to the smart glasses undergoing rotational movement, determining that a field of view of a camera of the smart glasses undergoes rotational movement; as well as Controlling the focus window to move along a moving sphere with a preset radius and centered on the camera toward a center position of a current field of view of the camera; The movement of the focus window includes a process of first accelerating and then decelerating.

2. The method according to claim 1, wherein: Controlling the movement of the focus window is based on an elastic rope model, Among them, one end of the elastic rope of the elastic rope model is connected to the field of view center position of the current field of view of the camera, and the other end is connected to the window center position of the focus window, wherein the window center position is updated according to each frame.

3. The method according to claim 2, wherein: Controlling the focus window to move along a moving sphere with a preset radius and centered on the camera toward a center position of a current field of view of the camera, comprising: For the start time of the current frame, determine the elastic force of the elastic rope between the center position of the field of view and the center position of the window at the start time of the current frame; Determine, based on the elastic force and the movement speed of the focus window at the start time of the current frame, an arc displacement of the focus window in the current frame, wherein the arc displacement is a portion of an arc between a center position of the field of view and a center position of the window at the start time of the current frame on the moving sphere; Determining the window center position of the focus window at the end time of the current frame based on the arc displacement; and The focus window is moved to the determined window center position of the focus window at the end time of the current frame.

4. The method according to claim 3, wherein: Determining the window center position of the focus window at the end time of the current frame based on the arc displacement includes: Determine the ratio of the arc displacement to the arc on the moving sphere between the center position of the field of view and the center position of the window at the start time of the current frame; and Based on the field of view center position, the window center position and the ratio, the window center position of the focus window at the end time of the current frame is determined.

5. The method according to claim 4, wherein: Determining the window center position of the focus window at the end time of the current frame includes: Based on the first vector between the position of the camera and the center position of the field of view at the start time of the current frame, the second vector between the position of the camera and the center position of the window at the start time of the current frame, and the ratio, the vector between the position of the camera and the window center position of the focus window at the end time of the current frame is determined as the window center position of the focus window at the end time of the current frame.

6. The method according to claim 3, wherein: Controlling the focus window to move along a moving sphere with a preset radius and centered on the camera toward the center position of the current field of view of the camera, further comprising: Determining a rotation amount of the focus window that makes the focus window tangent to the moving sphere at the end time of the current frame, wherein the rotation amount is associated with the position and posture of the focus window; and The focus window is rotated to have the determined rotation amount at the end time of the current frame.

7. The method according to claim 6, wherein: Determining the rotation amount of the focus window so that the focus window is tangent to the moving spherical surface at the end time of the current frame includes: Determine the rotation amount of the focus window at the end time of the current frame by using an interpolation algorithm based on an interpolation ratio, the window center position of the focus window and the center position of the field of view at the start time of the current frame, a first rotation amount corresponding to the window center position of the focus window at the start time of the current frame, and a second rotation amount corresponding to the position when the focus window coincides with the center position of the field of view, The interpolation ratio is related to the movement speed of the focus window at the start time of the current frame, and the greater the movement speed, the smaller the interpolation ratio.

8. An electronic device comprising: one or more processors; as well as One or more memories storing thereon instructions executable by a processor, wherein when the instructions are executed by the one or more processors, the one or more processors are caused to perform the method according to any one of claims 1 to 7.

9. A computer program product having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by one or more processors, the method according to any one of claims 1 to 7 is implemented.

Images