Augmented reality (AR) device and method for predicting posture therein

Abstract

An augmented reality (AR) device and a method for predicting the posture of the AR device are provided. In an augmented reality (AR) device, an inertial measurement unit (IMU) value corresponding to the movement of the AR device is obtained at an IMU rate; an intermediate 6D (6 degrees of freedom) posture of the AR device is estimated based on the IMU value and an image around the AR device via a visual inertial simultaneous localization and mapping (VI-SLAM) module, and a posture prediction model for predicting the relative 6D posture of the AR device is generated by performing learning using a deep neural network.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the priority benefit of Korean Patent Application No. 10-2020-0046268 filed on April 16, 2020 in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Background Art 1. Technical Field

[0004] Example embodiments of the present disclosure relate to an augmented reality (AR) device and a method of predicting a gesture in the AR device.

[0005] 2. Description of Related Technology

[0006] With recent technological developments, various types of wearable devices that can be worn on the human body have been developed. Glasses-type wearable devices are worn on the user's head and can provide augmented reality services to the user by displaying visual information about virtual objects on a display of the device.

[0007] According to display technology such as augmented reality (AR) technology, the image output on the device can be perceived by the user as real. Therefore, the user can experience situations that cannot be encountered in real life as if they really exist through AR technology display devices, etc. In order to facilitate such an experience, it is important to provide the output image to the user in real time through the AR device. The output image must be closely related to the posture of the wearable device worn on the user, and therefore the user's posture needs to be accurately detected. When the output image is not displayed in real time, the user may feel uncomfortable due to errors caused by the difference between the detected posture and the output image. Therefore, it is necessary to use accurate posture prediction methods using various digital processing technologies such as computer vision or simultaneous localization and mapping (SLAM). Summary of the invention

[0008] An augmented reality (AR) device and a method for predicting a posture of an AR device are provided. The technical objectives to be achieved by the present disclosure are not limited to the above objectives, and other technical objectives can be inferred from the following embodiments.

[0009] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the embodiments presented in the disclosure.

[0010] According to one aspect of the present disclosure, a method for predicting the posture of an augmented reality (AR) device is provided, the method comprising: obtaining an IMU value corresponding to the movement of the AR device at an IMU rate of a first frequency using an inertial measurement unit (IMU) provided in the AR device; estimating an intermediate 6-degree-of-freedom (6D) posture of the AR device based on the obtained IMU value and an image around the AR device, the image being obtained at a frame rate of a second frequency by a camera provided in the AR device; and generating a posture prediction model for predicting the relative 6D posture of the AR device by performing learning based on the obtained IMU value and the intermediate 6D posture using a deep neural network through a processor.

[0011] According to another aspect of the present disclosure, a non-transitory computer-readable recording medium is provided, having a program recorded thereon for executing a method for predicting a posture of an augmented reality (AR) device in a computer, the method comprising: obtaining an IMU value corresponding to a movement of the AR device at an IMU rate of a first frequency using an inertial measurement unit (IMU) sensor provided in the AR device; estimating an intermediate 6-degree-of-freedom (6D) posture of the AR device based on the obtained IMU value and an image around the AR device, the image being obtained at a frame rate of a second frequency by a camera provided in the AR device; and performing learning based on the obtained IMU value and the intermediate 6D posture using a deep neural network by a processor to generate a posture prediction model for predicting a relative 6D posture of the AR device.

[0012] According to another aspect of the present disclosure, an augmented reality (AR) device is provided, comprising: an inertial measurement unit (IMU) sensor configured to obtain IMU values ​​corresponding to the movement of the AR device at an IMU rate of a first frequency; a camera configured to obtain images around the AR device at a frame rate of a second frequency; a visual-inertial simultaneous localization and mapping (VI-SLAM) module configured to estimate an intermediate 6-degree-of-freedom (6D) posture of the AR device based on the obtained IMU values ​​and the images around the AR device; and a processor configured to generate a posture prediction model for predicting the relative 6D posture of the AR device by performing learning based on the obtained IMU values ​​and the intermediate 6D posture using a deep neural network.

[0013] According to another aspect of the present disclosure, a method for predicting posture in a processor is provided, the method comprising: obtaining an IMU value corresponding to the movement of the AR device at an IMU rate of a first frequency using an inertial measurement unit (IMU) sensor provided in an augmented reality (AR) device; obtaining an intermediate 6-degree-of-freedom (6D) posture of the AR device estimated based on the obtained IMU value and an image around the AR device, the image being obtained at a frame rate of a second frequency using a camera provided in the AR device; and performing learning based on the obtained IMU value and the intermediate 6D posture using a deep neural network to generate a posture prediction model for predicting the relative 6D posture of the AR device.

[0014] According to another aspect of the present disclosure, a computer-implemented method for training a neural network for posture prediction is provided, the method comprising: collecting inertial measurement unit (IMU) values ​​corresponding to the movement of an augmented reality (AR) device, collecting the IMU values ​​at a first frequency; collecting images around the AR device, collecting the images at a second frequency; performing one or more operations on the collected IMU values ​​and the collected images to create an intermediate 6-degree-of-freedom (6D) posture of the AR device; and training the neural network based on the collected IMU values ​​and the intermediate 6D posture.

[0015] According to another aspect of the present disclosure, a device for training a neural network for posture prediction is provided, comprising: a memory storing one or more instructions; and a processor configured to execute one or more instructions to: collect inertial measurement unit (IMU) values ​​corresponding to the movement of an augmented reality (AR) device, collecting the IMU values ​​at a first frequency; collect images around the AR device, collecting the images at a second frequency; perform one or more operations on the collected IMU values ​​and the collected images to create an intermediate 6-degree-of-freedom (6D) posture of the AR device; and train the neural network based on the collected IMU values ​​and the intermediate 6D posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other aspects, features and advantages of certain example embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0017] Figure 1 An augmented reality (AR) device according to an example embodiment is shown;

[0018] Figure 2 is a block diagram of electronic hardware provided in an AR device according to an example embodiment;

[0019] Figure 3 is a timing diagram of time points for obtaining IMU values ​​via an inertial measurement unit (IMU) sensor according to an example embodiment;

[0020] Figure 4 is a timing diagram of time points for obtaining an image via a camera according to an example embodiment;

[0021] Figure 5A is a timing diagram of time points for estimating intermediate six degree (6D) poses via a virtual inertial simultaneous localization and mapping (VI-SLAM) module according to an example embodiment;

[0022] Figure 5B is a timing diagram of time points for estimating an intermediate 6D pose via a virtual inertial simultaneous localization and mapping (VI-SLAM) module according to another example embodiment;

[0023] Figure 6 is a timing diagram of a method of generating a posture prediction model according to an example embodiment;

[0024] Fig. 7A is a timing diagram of a method of performing posture prediction in a posture prediction model by a sliding window method according to an example embodiment;

[0025] Figure 7B is a timing diagram of a method for performing posture prediction in a posture prediction model by a sliding window method according to another example embodiment;

[0026] Figure 8 is a diagram of a posture prediction model implemented by a recurrent neural network (RNN) using a long short-term memory (LSTM) based on a deep neural network (DNN) according to an example embodiment;

[0027] Fig. 9 and Fig.10 is a block diagram of performing gesture prediction in a processor according to some example embodiments;

[0028] Fig.11A and Fig. 11B is a time diagram of performing inference of posture prediction in a posture prediction model based on a DNN that has completed learning according to some example embodiments;

[0029] Fig.12 is a flowchart of a method of predicting a gesture in an AR device according to example embodiments. DETAILED DESCRIPTION

[0030] Reference will now be made in detail to embodiments, examples of which are shown in the accompanying drawings, wherein the same reference numerals represent the same elements throughout. In this regard, some example embodiments may have different forms and should not be construed as being limited to the descriptions set forth herein. Therefore, example embodiments are described below with reference to the accompanying drawings to explain various aspects. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Expressions such as "at least one of...", when following a list of elements, modify the entire list of elements without modifying the individual elements in the list.

[0031] Considering the functions in the present disclosure, the terms used in the example embodiments have been selected from the currently widely used general terms. However, these terms may vary according to the intentions of ordinary technicians in the field, case precedents, and the emergence of new technologies. In addition, for special cases, the meanings of the terms selected by the applicant are described in detail in the description section. Therefore, the terms used in the present disclosure are defined based on their meaning relative to the content discussed in the entire specification rather than their simple meanings.

[0032] In the exemplary embodiments, although terms such as "first", "second", etc. may be used to describe various components, such components must not be limited to the above terms. The above terms are only used to distinguish one component from another component.

[0033] The terms used in some embodiments are only used to describe the example embodiments and are not intended to limit the present disclosure. Unless it has a significantly different meaning in the context, an expression in the singular includes an expression in the plural form. In addition, in the example embodiments, when a constituent element is "connected" or "connected" to another constituent element, the constituent element is not only directly in contact with or connected to the other constituent element, but also electrically contacted or connected to the other constituent element through at least one of the other constituent elements inserted between the two. When a part may "include" a certain constituent component, unless otherwise specified, it may not be interpreted as excluding another constituent element, but may be interpreted as further including other constituent elements.

[0034] With respect to the example embodiments, "above" and similar instructions, particularly as used in the claims, may refer to both the singular and the plural. Furthermore, the operations or steps of all methods according to the example embodiments described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The example embodiments are not limited to the order in which the operations or steps are described.

[0035] The present disclosure can be described in terms of function block components and various processing steps. Such function blocks can be implemented by any number of hardware and / or software components configured to perform a specified function. For example, the function blocks associated with the example embodiments can be implemented by one or more microprocessors or circuit elements for specific functions. In addition, the function blocks can be implemented in various programming or scripting languages. In addition, the function blocks can be implemented in algorithms executed on one or more processors. The example embodiments can employ any number of related technologies for electronic configuration, signal processing and / or control, data processing, etc.

[0036] Furthermore, the connecting lines or connectors shown in the various figures are intended to represent functional relationships and / or physical or logical couplings between the various elements. It should be noted that many alternative or additional functional relationships, physical connections or logical connections may exist in a practical device.

[0037] The exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. However, the exemplary embodiments are not limited thereto, and it will be understood that various changes in form and details may be made therein without departing from the spirit and scope of the appended claims.

[0038] Figure 1 An augmented reality (AR) 10 device is shown according to an example embodiment.

[0039] refer to Figure 1 , the AR device 10 may be a wearable electronic device worn by a user. For example, the AR device 10 may be a glasses-type wearable device, but the embodiment is not limited thereto. For example, according to another exemplary embodiment, the AR device 10 may be a different type of wearable device to be worn on other human body parts of the user, such as an AR helmet.

[0040] The AR device 10 may provide an AR service through a display on the AR device 10, which fuses real-world information around the user and digital or virtual object information. For example, when an AR object (or AR scene) is displayed on the user's eyes through the AR device 10, the AR object may be displayed together with the scene of the real world seen at the current posture of the AR device 10. According to an example embodiment, the AR object may be displayed, overlaid, or superimposed on the scene of the real-world environment perceived by the user at the current posture of the AR device 10. Although the user can see the AR object through the AR device 10, the AR object does not exist in the real world.

[0041] The real world is a real scene that an observer or user can see through the AR device 10, and may include real-world objects. On the other hand, an AR object (or scene) is an image generated by graphic processing, and may correspond to a static image or a dynamic image. For example, an AR object (or scene) may be an image overlaid on a real scene to provide information about a real object in the real scene or information about the operation of the AR device 10 or a control menu.

[0042] like Figure 1 As shown, the AR device 10 may correspond to Figure 1 The head-mounted wearable device shown in the figure can move together accordingly when the user's head moves. Therefore, since the posture of the AR device 10 can be continuously changed according to the movement of the user's head, it is necessary to continuously update the posture of the AR device 10 in consideration of the change in the posture of the user's head. For example, when a user wearing the AR device 10 walks on a road, the real world scene seen through the AR device 10 keeps changing. Therefore, the AR object (or AR scene) provided by the AR device 10 can be re-rendered in consideration of the current posture, so that a seamless perspective of the AR object (or AR scene) existing in the real world scene is provided to the user.

[0043] According to an example embodiment, part of the internal space of the AR device 10 may be equipped with electronic hardware (HW) 100, which includes various types of sensing modules, such as an inertial measurement unit (IMU) sensor, a simultaneous localization and mapping (SLAM) module, an accelerometer or a compass, an image capture module such as a camera, a microphone, a GPS module, a communication interface, a processing unit, or a battery. According to an example embodiment, the electronic HW may be equipped in part of the internal space of the AR device 10. The AR device 10 may be further equipped with an optical engine or an optical element that displays an AR object. Figure 1 The electronic HW 100 is shown to be disposed in a frame on one side of the AR device 10, the position of the electronic HW 100 in the AR device 10 is not limited thereto, and the electronic HW 100 may be disposed at various positions on the AR device 10, and each of the hardware constituent elements may be disposed in the same space or in different spaces.

[0044] Figure 2 is a block diagram of electronic hardware provided in an AR device according to an embodiment.

[0045] refer to Figure 2 , the electronic HW 100 may include an IMU sensor 110, a camera 120, a visual inertial simultaneous positioning and mapping (VI-SLAM) module 130, a processor 140, and a memory 150. Figure 2In the electronic HW 100, for convenience of explanation, the hardware constituent elements related to the embodiment of performing gesture prediction are mainly shown. However, according to some example embodiments, the electronic HW 100 may further include other hardware constituent elements, such as a microphone, a global positioning system (GPS) module, a communication interface or an optical engine or other general constituent elements.

[0046] The IMU sensor 110 is a sensor for measuring the speed, direction, gravity, or acceleration of the AR device 10. The IMU sensor 110 can measure the free movement of the AR device 10 in a three-dimensional (3D) space by using a gyroscope, an accelerometer, or a geomagnetic field, so that the acceleration in the forward direction, the lateral direction, and the height direction, as well as the roll, pitch, and yaw angular velocities can be measured. The speed, relative position, or posture angle of the AR device 10 can be calculated by integrating the acceleration and angular velocity obtained from the IMU sensor 110.

[0047] The IMU sensor 110 may obtain data corresponding to the first frame rate (i.e., the IMU rate) at a first frequency. Figure 1 In other words, the time series of IMU values ​​can be obtained from the IMU sensor 110.

[0048] Figure 3 is a timing diagram of time points for obtaining IMU values ​​via an IMU sensor according to an example embodiment. Figure 3 , the IMU sensor 110 may measure acceleration in the forward direction (X-axis), the lateral direction (Y-axis), and the height direction (Z-axis), as well as roll, pitch, and yaw angular velocities at an IMU rate of a first frequency, and output the measured IMU values. According to an example embodiment, the IMU rate may be 500 Hz, but the embodiment is not limited thereto. Figure 3 In the timing diagram 300, each vertical line represents a time point at which an IMU value is obtained. For example, the IMU value obtained at time point I can be represented as IMU i , ..., the IMU value obtained at time point L can be expressed as IMU L , the IMU value obtained at time point L+1 can be expressed as IMU L+1 , and the IMU value obtained at time point L+2 can be expressed as IMU L+2 , where i and L are natural numbers.

[0049] Reference again Figure 2, the camera 120 is an image capture device for photographing a real scene of the environment around the AR device 10. For example, the camera 120 may be a front camera configured to capture the environment in front of the AR device 10. The camera 120 obtains images around the AR device 10 at a frame rate of the second frequency. In other words, a time series of images of the real scene around the AR device 10 can be obtained from the camera 120.

[0050] Figure 4 is a timing diagram of time points for obtaining an image via a camera according to an example embodiment. Figure 4 , the camera 120 obtains the surrounding image of the AR device 10 at a frame rate of the second frequency. According to an example embodiment, the frame rate may be 30fps, but the embodiment is not limited thereto. The second frequency of obtaining images from the camera 120 is lower than the first frequency of obtaining IMU values ​​from the IMU sensor 110. In other words, the IMU rate of the first frequency is a relatively high frequency, and the frame rate of the second frequency is a relatively low frequency. Figure 4 In the timing diagram 400, each vertical line represents a time point at which an image is obtained. For example, an image obtained at time point 0 can be represented as I 0 , ..., the image obtained at time point n can be expressed as I n , the image obtained at time point N-1 can be expressed as I N-1 , and the image obtained at time point N can be expressed as I N , where n and N are natural numbers.

[0051] Refer to Figure 2 According to an example embodiment, the VI-SLAM module 130 is a hardware component (eg, a hardware processor) that performs calculations for estimating a relative position of a camera to a map by fusing visual data and inertial data (ie, visual-inertial fusion).

[0052] When the posture is estimated using only the IMU value of the IMU rate obtained from the IMU sensor 110, the posture may be inaccurate due to noise and slowly changing deviations of the obtained IMU value. In addition, when the posture is estimated by performing SLAM processing only with the image obtained from the camera 120, it may be difficult to perform high-speed posture calculation. Therefore, the VI-SLAM module 130 can improve the accuracy of the SLAM process by performing posture estimation based on visual inertial fusion in a VI-SLAM method of error value compensation, which compensates for the error value by using the IMU value obtained from the IMU sensor 110 and the image obtained from the camera 120.

[0053] For example, due to various factors, such as occlusion due to flat areas (e.g., walls, poor lighting, motion blur / focus issues, high-speed camera motion, or hand shaking), a good camera pose may not be calculated. However, in an example embodiment, since learning the camera pose through a deep neural network (DNN) based on visual-inertial fusion is performed, the accuracy of the pose prediction of the AR device 10 can be improved even when a suitable camera image cannot be obtained.

[0054] The VI-SLAM module 130 may estimate an intermediate 6-degree-of-freedom pose (intermediate 6D pose) of the AR device 10 based on inputs of images (image sequences) obtained by the camera 120 at a frame rate of the second frequency and IMU values ​​(IMU sequences) obtained from the IMU sensor 110. An “intermediate” 6D pose refers to a pose based on an intermediate calculation result before predicting a final 6D pose (ie, a relative 6D pose).

[0055] In one embodiment, whenever an image is obtained at a frame rate of a second frequency, the VI-SLAM module 130 may perform an estimation of an intermediate 6D pose. In this case, the estimation of the intermediate 6D pose may be performed at a frame rate of the second frequency. In other words, the intermediate 6D pose in this case may be first type pose data at a frame rate of the second frequency, which is estimated by performing a SLAM technique for fusing IMU values ​​with images obtained at a frame rate of the second frequency.

[0056] In addition, in another exemplary embodiment, between the time points at which images are obtained at the frame rate of the second frequency image, the VI-SLAM module 130 may perform estimation of the intermediate 6D posture at the IMU rate of the first frequency in the following manner: by calculating the integral value of the angular velocity and the integral value of the acceleration of each axis by a pre-integration method based on the IMU value between the time points at which images are obtained at the frame rate. In other words, the intermediate 6D posture in this case may be the second type of posture data at the IMU rate, which includes posture data corresponding to the integral value, which is calculated by a pre-integration method based on the image-based posture data of the frame rate estimated by performing the above-mentioned SLAM technology according to the IMU value.

[0057] Figure 5A is a timing diagram of time points for estimating intermediate 6D poses in a VI-SLAM module according to an example embodiment. Figure 5A , the VI-SLAM module 130 may receive input of IMU values ​​520 obtained at an IMU rate and images obtained at a frame rate, and estimate an intermediate 6D pose 510 at a frame rate.

[0058] In detail, the VI-SLAM module 130 can calculate the time Δ TRK During the period, the image I obtained at time point 00 And at time point A' 0 At time point A 0 The calculation is performed between the IMU values ​​obtained to obtain the intermediate 6D pose P 0 . Although the middle 6D posture P 0 The calculation is at time point A 0 Finished, middle 6D pose P 0 can be considered to correspond to time point A' 0 In the same way, the VI-SLAM module 130 can calculate the posture of TRK The intermediate 6D pose P is obtained by calculating n 、Intermediate 6D pose P N-1 and the middle 6D pose P N Each of them, and the intermediate 6D pose P obtained n 、Intermediate 6D pose P N-1 and the middle 6D pose P N can be considered to correspond to time point A' n Posture, time point A' N-1 The posture and time point A' N As described below, whenever the VI-SLAM module 130 completes the estimation of the intermediate 6D pose, the estimated intermediate 6D pose is at the actual time point of the image capture, for example, time point A' 0 , ... time point A' n , time point A' N-1 , time point A' N , is updated to the ground truth (GT) and can be used for learning.

[0059] Figure 5B is a timing diagram of time points for estimating intermediate 6D poses via a VI-SLAM module according to another example embodiment. Figure 5B ,and Figure 5A Instead, the VI-SLAM module 130 estimates the intermediate 6D pose 515 at the IMU rate.

[0060] In detail, the VI-SLAM module 130 obtains an integrated value of acceleration and an integrated value of angular velocity of each axis by performing two integrations on the IMU values ​​520 obtained at the IMU rate. The integrated value may correspond to a 6D posture estimated individually from the IMU values ​​520. In other words, the VI-SLAM module 130 performs pre-integration on the IMU values ​​520 based on the intermediate 6D posture (P 0 , P n , P N-1 , P N...) to predict the intermediate 6D pose (..., P') obtained at the IMU rate n_1 , P' n_2 , P' n_3 , ..., P' N_1 , P' N_2 , P' N_3 , ...). In this way, in addition to the intermediate 6D pose (P) based on the camera image estimated by the SLAM process ② 0 , P n , P N-1 , P N ...) and also estimates the intermediate 6D pose (..., P') based on IMU integration through the pre-integration process ① n_1 , P' n_2 , P' n_3 , ..., P' N_1 , P' N_2 , P' N_3 , ...), the VI-SLAM module 130 can continuously output the estimation result of the intermediate 6D pose 515 even when the camera image may not be used smoothly due to problems such as occlusion or blur.

[0061] Refer to Figure 2 The processor 140 is a hardware component that can control the overall operation of the AR device 10 by running an operating system or an application program, and can perform various data processing and operations including posture prediction and AR image generation. The processor 140 may include, for example, a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), an application processor (AP), a neural processing unit (NPU), or a tensor processing unit (TPU), but the embodiment is not limited thereto.

[0062] The processor 140 generates a posture prediction model that predicts the relative 6D posture of the AR device 10 by performing learning based on the input of the intermediate 6D posture and the IMU value obtained by using the DNN. The "relative" 6D posture may mean the posture difference between the intermediate previously predicted 6D posture (original posture) and the currently predicted 6D posture (transformed posture). In addition, the relative 6D posture may mean the final posture finally obtained by calculating from the intermediate 6D posture.

[0063] According to an example embodiment, the pose prediction model can perform DNN learning using multi-rate inputs including IMU inputs at the IMU rate and intermediate 6D pose inputs at the frame rate. In other words, since the IMU rate is a frequency higher than the frame rate, the pose prediction model can perform learning for pose prediction and inference for pose prediction using inputs of different frequencies.

[0064] In addition, according to another example embodiment, the pose prediction model can perform DNN learning with multi-rate input of intermediate 6D pose input, wherein the multi-rate input includes IMU input at IMU rate, IMU integral-based intermediate 6D pose input at IMU rate, and image-based intermediate 6D pose input at frame rate.

[0065] Since the IMU input and the intermediate 6D pose input to the pose prediction model correspond to temporal data sequences, the DNN can be implemented as a recurrent neural network (RNN) using a long short-term memory (LSTM) or a gated recurrent unit (GRU). The pose prediction model can be based on Figure 5A The update of the GT pose described above performs the DNN learning model in a self-supervised manner.

[0066] When the newly obtained IMU value based on the movement of the AR device 10 and the newly estimated intermediate 6D posture based on the movement of the AR device 10 are input into the learned DNN of the posture prediction model, the processor 140 can output the inference result of the predicted relative 6D posture in real time by processing the posture prediction model through the learned DNN.

[0067] The processor 140 may perform DNN learning by using the IMU values ​​and camera images obtained during the use of the AR device 10, or the 6D posture calculated by the VI-SLAM module 130 when the AR device 10 is not used for a certain period of time (for example, in a charging state). When the user wears the AR device 10 again for use, the processor 140 may complete the DNN learning and perform inference of the DNN to predict the posture of the AR device 10.

[0068] The memory 150 may include an internal memory such as a volatile memory or a non-volatile memory. The memory 150 may store various data, programs or applications for driving and controlling the AR device 10, as well as input / output signals or AR data under the control of the processor 140. In addition, the memory 150 may store data related to the DNN. The memory 150 may correspond to a memory device such as a random access memory (RAM), a read-only memory (ROM), a hard disk drive (HDD), a solid-state drive (SSD), a compact flash (CF), a secure digital (SD), a micro secure digital (micro-SD), a mini secure digital (mini-SD), an extreme digital (xD), a memory stick, etc., and the type of the memory 150 is not limited thereto and may be various.

[0069] Figure 6 is a timing diagram of a method for generating a posture prediction model according to an embodiment.

[0070] refer to Figure 6 , the processor 140 receives Figure 2 The IMU sensor 110 obtains the IMU value 520 and the Figure 2 The VI-SLAM module 130 takes the estimated intermediate 6D pose 510 as input.

[0071] Specifically, the processor 140 may perform learning for pose prediction on the DNN using the input of the window that moves in a sliding manner. The window that moves in a sliding manner may include k IMU values, where k is a natural number, and m intermediate 6D poses, where m is a natural number. For example, the window 1610 (WINDOW1610) may be set to include the values ​​from the IMU i To IMU L The IMU value and the P n To P N In other words, the DNN performs learning using multi-rate inputs including IMU input at the IMU rate and intermediate 6D pose input at the frame rate.

[0072] The processor 140 may output the relative 6D posture of the AR device 10 by processing the WINDOW 1610 input to the DNN. 1 .

[0073] Fig. 7A is a timing diagram of a method of performing posture prediction in a posture prediction model by a sliding window method according to an example embodiment.

[0074] refer to Fig. 7A , when the relative 6D pose O is predicted by processing the window 1610 1The window can slide based on the IMU rate. In other words, the window can slide on the time axis in units of time points to obtain an IMU value.

[0075] The processor 140 may output the relative 6D posture of the AR device 10 by processing the WINDOWS-1620 sliding using the DNN. S-1 , and outputs the relative 6D posture of the AR device 10 by processing the WINDOWS 630 sliding using the DNN S In other words, the DNN-based pose prediction model can receive input at multiple rates (IMU rate and frame rate), but output the relative 6D pose corresponding to each window at the IMU rate.

[0076] When the DNN-based posture prediction model performs learning to predict the relative 6D posture, the learning can be performed in a self-supervised learning method (or an unsupervised learning method) based on the update of the GT posture 710. For example, during DNN learning, the intermediate 6D posture at the time matching the time of the output window can be used as the GT posture 710 for the output window.

[0077] Whenever Figure 2 When the VI-SLAM module 130 completes the estimation of the intermediate 6D pose, the estimated intermediate 6D pose can be updated to the GT pose (P 0 (GT)...、P n (GT)...、P M-1 (GT), P N (GT)...) to set GT posture 710.

[0078] The DNN can perform back-propagation through time (BPTT) by reducing the loss function using the updated GT pose, and thus update the weight parameters of the DNN to appropriate values, so that the pose prediction accuracy of the DNN-based pose prediction model can be improved.

[0079] Figure 7B is a timing diagram of a method of performing posture prediction in a posture prediction model by a sliding window method according to another example embodiment.

[0080] refer to Figure 7B ,and Fig. 7A Differently, the processor 140 receives the intermediate 6D pose input 515 as a multi-rate input to the DNN, wherein the intermediate 6D pose input includes the IMU value 520, the IMU rate, the IMU-integrated intermediate 6D pose input (..., P' n_1 , P' n_2 , P' n_3 ...、P' N_1 , P'N_2 , P' N_3 ...) and frame-rate intermediate image-based 6D pose input (P 0 , P n , P N-1 , P N ). Therefore, in Fig. 7A The GT pose 710 is updated at short time intervals and the GT pose 720 is updated, and thus the DDN can relatively accurately output a prediction result of the relative 6D pose even at a moment when the camera image may not be used smoothly due to problems such as occlusion or blur.

[0081] Figure 8 is a diagram of a DNN-based posture prediction model implemented by RNN using LSTM according to an example embodiment.

[0082] According to an example embodiment, since the Figure 1 The input of the posture prediction in the AR device 10 corresponds to the time data sequence, and the posture prediction model can be implemented by RNN. However, even so, the posture prediction model is not necessarily limited to the RNN-based neural network, and different types of neural networks can be used.

[0083] RNN is a neural network that is useful for predicting current information using past information. However, since RNN has problems such as long-term dependency, LSTM type RNN is introduced to solve the problem. LSTM can include a cell state and a hidden layer respectively. In LSTM, the cell state manages the information memory and erases unimportant information (forget gate), and the hidden layer updates only important information to the cell (input gate), so that the long-term dependency problem of RNN can be solved.

[0084] refer to Figure 8 , showing an expanded multi-rate LSTM. For example, IMU values ​​of an IMU rate of a first frequency of 500Hz and an intermediate 6D pose of a frame rate of a second frequency (e.g., 30fps) are input to the LSTM. In this way, when the LSTM performs learning of pose prediction using multi-rate inputs, as shown by the dotted line, the LSTM storing the input (intermediate 6D pose) of the rate of the low frequency (second frequency) can be changed to an inactive state in the middle of learning. The inputs of past IMU values ​​and past intermediate 6D poses stored in the LSTM can be transmitted to the units of other LSTMs to be associated with predicting relative 6D poses at different moments.

[0085] In the pose prediction model in which pose prediction learning is performed by LSTM-type RNN, since the updated GT pose (P 0 ,...P N...) Performing BPTT to reduce the loss function can improve the posture prediction accuracy of the LSTM-based posture prediction model.

[0086] According to another example embodiment, the DNN-based posture prediction model can be implemented by RNN using a gated recurrent unit (GRU) instead of LSTM, and those skilled in the art can understand that the GRU method can operate similarly to the above-mentioned method regarding LSTM.

[0087] Fig. 9 and Fig.10 is a block diagram of performing gesture prediction in a processor according to an example embodiment.

[0088] First reference Fig. 9 , the IMU sensor 110 transmits time data of an IMU sequence ① including time data of IMU values ​​obtained at an IMU rate to the VI-SLAM module 130, and the camera 120 transmits time data of an image sequence ① including images obtained at a frame rate to the VI-SLAM module 130. The IMU sensor 110 transmits the IMU sequence ① to the processor 140 for posture prediction.

[0089] The VI-SLAM module 130 estimates the intermediate 6D pose based on the IMU sequence ① and the image sequence ①, and transmits the estimated intermediate 6D pose ① to the processor 140. In this case, the VI-SLAM module 130 takes the image at the real time point (advance amount Δ TRK ) updates the estimated intermediate 6D pose ① to the GT pose ①, and transmits the updated GT pose ① to the processor 140.

[0090] The processor 140 calculates outputs regarding the IMU sequence ① and the intermediate 6D pose ① by using the DNN-based pose prediction model, and outputs the relative pose ① as a result thereof.

[0091] Next, refer to Fig.10 , the VI-SLAM module 130 estimates the intermediate 6D pose ② based on the IMU sequence ② obtained from the IMU sensor 110 and the image sequence obtained from the camera 120, and transmits the estimated intermediate 6D pose ② to the processor 140. The processor 140 calculates the output about the IMU sequence ② and the intermediate 6D pose ② by using the DNN-based pose prediction model, and outputs the relative pose ② as its result.

[0092] When the relative pose ② is predicted, the processor 140 may update the weight parameters by back propagation (eg, BPTT of DNN) based on the updated GT pose ① and the updated GT pose ②. According to an example embodiment, the processor 140 may repeatedly perform the above process to predict subsequent relative poses.

[0093] exist Fig. 9 and Fig.10 In the figure, symbols ① and ② indicate data at different time points.

[0094] Fig.11A and Fig. 11B is a time diagram of inference of performing posture prediction in a posture prediction model based on a DNN that has completed learning according to some embodiments.

[0095] refer to Fig.11A and Fig. 11B , the learning of pose prediction is performed based on the IMU values ​​and the intermediate 6D pose input to the DNN, and when the parameters of the DNN are updated by the updated GT pose, the DNN learning is completed, and thus a pose prediction model can be generated. Figure 1 When the newly obtained IMU value and the newly estimated intermediate 6D posture of the AR device 10 are input into the posture prediction model generated based on the learned DNN, Figure 2 The processor 140 can output the inference result of the predicted relative 6D posture in real time by processing the posture prediction model through the learned DNN.

[0096] For example, as described above, the IMU values ​​and the intermediate 6D pose may be input to the learned DNN in a sliding window approach. When WINDOW 11110 is input, the processor 140 calculates the 6D pose by calculating the time Δ R During this period, the input in WINDOW11110 is processed to perform inference of the predicted result of the relative 6D posture, and the inference result corresponding to WINDOW11110, that is, the predicted relative 6D posture, is output. Similarly, the processor 140 performs inference on the case where the input has a sliding WINDOW21120 and the case where the input has a sliding WINDOW31130, and outputs the inference result, that is, the predicted relative 6D posture.

[0097] Since the DNN has completed learning for posture prediction based on the input IMU value and the intermediate 6D posture, the processor 140 can not only infer the current and future 6D postures of the AR device 10 based on the results of DNN learning, but also output the prediction results of the future 6D posture of the AR device 10.

[0098] Fig.12 is a flowchart of a method for predicting a gesture in an AR device according to an example embodiment. Fig.12 ,because Figure 1 The posture prediction method in the AR device 10 is related to the example embodiments in the above figures. The contents described in the above figures, even if omitted in the following description, can also be applied to Fig.12 method.

[0099] In operation 1201 , the processor 140 obtains an IMU value corresponding to a movement of the AR device 10 from the IMU sensor 110 at an IMU rate of a first frequency.

[0100] In operation 1202 , the processor 140 may obtain an intermediate 6D pose of the AR device 10 from the VI-SLAM module 130 , wherein the intermediate 6D pose is estimated based on an input of an IMU value and an image around the AR device 10 obtained by the camera 120 at a frame rate of a second frequency.

[0101] In operation 1203 , the processor 140 performs learning using the DNN based on the input of the IMU value and the intermediate 6D pose, thereby generating a pose prediction model that predicts the relative 6D pose of the AR device 10 .

[0102] The above-described exemplary embodiments may be written as a program executable on a computer and may be implemented in a general-purpose digital computer that uses a computer-readable recording medium to operate the program. In addition, the structure of the data used in the above-described embodiments may be recorded on a computer-readable recording medium by various means. The computer-readable recording medium may include a storage medium such as a magnetic storage medium, for example, a ROM, a floppy disk, or a hard disk, or an optical reading medium such as a CD-ROM or a DVD.

[0103] It should be understood that the example embodiments described herein should be considered in a descriptive sense only and not for purposes of limitation. It should generally be considered that the description of features or aspects in each example embodiment can be used for other similar features or aspects in other example embodiments. Although one or more example embodiments have been described with reference to the accompanying drawings, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope as defined by the following claims.

Claims

1. A method for predicting a posture of an augmented reality (AR) device, the method comprising: Using an inertial measurement unit (IMU) sensor provided in the AR device, obtaining an IMU value corresponding to movement of the AR device at an IMU rate of a first frequency; estimating an intermediate 6-DOF (6D) posture of the AR device based on the obtained IMU value and an image around the AR device, the image being obtained at a frame rate of a second frequency by a camera provided in the AR device; generating, by a processor, a posture prediction model for predicting a relative 6D posture of the AR device; Inputting the IMU values ​​as a first input to the pose prediction model and separately inputting the intermediate 6D pose estimated before predicting the relative 6D pose as a second input to the pose prediction model; as well as The relative 6D pose of the AR device is output by performing learning based on the obtained IMU values ​​and the intermediate 6D pose using a deep neural network implemented by the pose prediction model.

2. The method of claim 1, wherein the intermediate 6D pose comprises: The first type of pose data at a frame rate is estimated by using a simultaneous localization and mapping (SLAM) technique that fuses the obtained IMU values ​​with images obtained at a frame rate of a second frequency, or A second type of posture data at an IMU rate of a first frequency includes posture data corresponding to an integrated value calculated by a pre-integration method based on the obtained IMU values ​​and the image-based posture data at the frame rate estimated by using the SLAM technology.

3. The method of claim 1 , wherein the pose prediction model outputs a relative 6D pose for each window that moves in a sliding manner according to learning of the deep neural network, the learning being based on k of the IMU values ​​and m of the intermediate 6D poses, and Wherein, in the window, k is a natural number, and m is a natural number.

4. The method of claim 3, wherein the window slides based on the IMU rate and k IMU values ​​corresponding to the sliding window and m intermediate 6D poses are input to the deep neural network, and The pose prediction model outputs a relative 6D pose for each window corresponding to the IMU rate.

5. The method of claim 1, further comprising: Based on completing the estimation of the intermediate 6D pose, the estimated intermediate 6D pose is updated and set as the ground truth GT pose at the real time point when the image is captured.

6. The method of claim 5, wherein the posture prediction model performs learning using the deep neural network via a self-supervised approach based on the updating of the GT posture.

7. The method of claim 1, wherein the pose prediction model performs learning using the deep neural network using a multi-rate input including IMU values ​​at an IMU rate of a first frequency and the intermediate 6D pose at a frame rate of a second frequency, and The first frequency is greater than the second frequency.

8. The method of claim 1, wherein the deep neural network comprises a recurrent neural network using long short-term memory or a gated recurrent unit.

9. The method of claim 8, wherein the deep neural network performs back-propagation over time by reducing a loss function using the updated ground truth GT pose.

10. The method of claim 1, further comprising: Based on the newly obtained IMU values ​​and the newly estimated intermediate 6D posture according to the movement of the AR device, the learned deep neural network is input into the posture prediction model, and the inference result of the predicted relative 6D posture is output in real time through the processing of the learned deep neural network of the posture prediction model.

11. A non-transitory computer-readable recording medium having a program recorded thereon for executing a method of predicting a posture of an augmented reality (AR) device in a computer, the method comprising: Using an inertial measurement unit (IMU) sensor provided in the AR device, obtaining an IMU value corresponding to movement of the AR device at an IMU rate of a first frequency; estimating an intermediate 6-DOF (6D) posture of the AR device based on the obtained IMU value and an image around the AR device, the image being obtained at a frame rate of a second frequency by a camera provided in the AR device; generating, by a processor, a posture prediction model for predicting a relative 6D posture of the AR device; Inputting the IMU values ​​as a first input to the pose prediction model and separately inputting the intermediate 6D pose estimated before predicting the relative 6D pose as a second input to the pose prediction model; as well as The relative 6D pose of the AR device is output by performing learning based on the obtained IMU values ​​and the intermediate 6D pose using a deep neural network implemented by the pose prediction model.

12. An augmented reality (AR) device, comprising: an inertial measurement unit (IMU) sensor configured to obtain an IMU value corresponding to the movement of the AR device at an IMU rate of a first frequency; a camera configured to obtain an image around the AR device at a frame rate of a second frequency; A visual-inertial simultaneous localization and mapping (VI-SLAM) module configured to estimate an intermediate 6-DOF (6D) posture of the AR device based on the obtained IMU values ​​and an image around the AR device; as well as a processor configured to generate a pose prediction model for predicting a relative 6D pose of the AR device, wherein the IMU value is input as a first input to the pose prediction model, and the intermediate 6D pose estimated before predicting the relative 6D pose is separately input as a second input to the pose prediction model, and Wherein, learning is performed based on the obtained IMU values ​​and the intermediate 6D pose to output the relative 6D pose of the AR device by using a deep neural network implemented by the pose prediction model.

13. The AR device of claim 12 , wherein the pose prediction model outputs a relative 6D pose each corresponding to a window moving in a sliding manner according to learning of the deep neural network, the learning being based on k of the IMU values ​​and m of the intermediate 6D poses, and Wherein, in the window, k is a natural number, and m is a natural number.

14. The AR device of claim 13, wherein the window slides based on the IMU rate and k IMU values ​​corresponding to the sliding window and m intermediate 6D poses are input to the deep neural network, and The pose prediction model outputs a relative 6D pose for each window corresponding to the IMU rate.

15. The AR device of claim 12, wherein the processor is further configured to: based on completing the estimation of the intermediate 6D pose by the VI-SLAM module, update and set the estimated intermediate 6D pose as the ground truth GT pose at a real-time point in time when the image is captured. 16 . The AR device of claim 15 , wherein the pose prediction model performs learning using the deep neural network via a self-supervised method based on the update of the GT pose.

17. The AR device of claim 12, wherein the pose prediction model performs learning using the deep neural network using a multi-rate input including the IMU value at an IMU rate of a first frequency and the intermediate 6D pose at a frame rate of a second frequency, and The first frequency is greater than the second frequency.

18. The AR device of claim 12, wherein the deep neural network comprises a recurrent neural network using long short-term memory or a gated recurrent unit.

19. The AR device of claim 18, wherein the deep neural network performs backpropagation over time by reducing a loss function using the updated ground truth poses.

20. An AR device as described in claim 12, wherein the processor is further configured to: when the newly obtained IMU value and the newly estimated intermediate 6D posture according to the movement of the AR device are input into the learned deep neural network of the posture prediction model, output the inference result of the predicted relative 6D posture in real time through the processing of the learned deep neural network of the posture prediction model.

21. A method for predicting a gesture in a processor, the method comprising: Using an inertial measurement unit (IMU) sensor provided in the augmented reality (AR) device, obtaining an IMU value corresponding to movement of the AR device at an IMU rate of a first frequency; Obtaining an intermediate 6-DOF (6D) posture of the AR device estimated based on the obtained IMU value and an image around the AR device, wherein the image is obtained at a frame rate of a second frequency using a camera provided in the AR device; generating a posture prediction model for predicting a relative 6D posture of the AR device; Inputting the IMU value as a first input into the pose prediction model, and separately inputting the intermediate 6D pose obtained before predicting the relative 6D pose into the pose prediction model as a second input; as well as The relative 6D pose of the AR device is output by performing learning based on the obtained IMU values ​​and the intermediate 6D pose using a deep neural network implemented by the pose prediction model.

Images