Method and system for enhancing IMU data in real time for a head mounted display (HMD) device

Abstract

Embodiments of the present disclosure disclose method and system for enhancing IMU data in real time for a head mounted display (HMD) which comprises initializing a sliding window length, receiving, a first set of IMU values between two consecutive images captured using camera of the HMD and retrieving one or more IMU values prestored in IMU database based on the sliding window length and the first set of IMU values. Method includes generating a second set of IMU values by appending the retrieved one or more IMU values to the first set of IMU values and generating a set of denoised IMU values. Method includes deriving one or more noise parameters present in the first set of IMU values and determining, a modified sliding window length for performing the IMU data enhancement based on the modified sliding window length.

Description

METHOD AND SYSTEM FOR ENHANCING IMU DATA IN REAL TIME FOR A HEAD MOUNTED DISPLAY (HMD) DEVICE

[0001] The present disclosure generally relates to field of head mounted display (HMD) devices. More particularly, the present disclosure relates to method and system for enhancing Inertial Measurement Unit (IMU) data in real time for a head mounted display device.

[0002] Head mounted displays (HMDs) are gaining prominence in the fields of augmented reality (AR) and virtual reality (VR) applications. A critical component of any HMD device is an accurate localization and mapping module, meticulously designed to track user movements within a given scene. Central to this localization and mapping module is a feature detection and matching module, which utilizes visual cues to accurately estimate a user pose.

[0003] The method of calculating these user poses is predominantly dependent upon IMU sensor readings. IMU Data encompasses gyroscope readings indicative of angular velocity and accelerometer readings representing the acceleration of the device. These IMU sensors are significantly noisy and suffer from alignment related issues, leading to inaccuracies in pose estimation thereby affecting the tracking and localization of the user. The Simultaneous Localization and Mapping (SLAM) module are used to achieve accurate user pose estimations if the IMU sensor readings are more precise.

[0004] The information disclosed in this background of the disclosure section is only for enhancement of understanding of the general background of the disclosure and may not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person skilled in the art.

[0005] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description.

[0006] In an embodiment, a method for enhancing IMU data in real time for a head mounted display (HMD) device is disclosed. In an embodiment, the method may include obtaining a first set of IMU values between two consecutive images captured using at least one camera of the HMD. In an embodiment, the method may include obtaining a second set of IMU values by appending previous one or more IMU values to the first set of IMU values wherein the second set of IMU values includes total sample size equal to defined sliding window length which indicates input size of the previous one or more IMU values. In an embodiment, the method may include obtaining a set of denoised IMU values by processing the second set of IMU values. In an embodiment, the method may include determining one or more noise parameters present in the first set of IMU values by correlating the set of denoised IMU values with the first set of IMU values. In an embodiment, the method may include determining a modified sliding window length based on the one or more noise parameters. In an embodiment, the method may include performing the IMU data enhancement on a third set of IMU values based on the modified sliding window length, wherein the third set of IMU values is subsequent to the first set of IMU values.

[0007] In an embodiment, a system for enhancing IMU data in real time for a head mounted display (HMD) device is disclosed. In an embodiment, the system may comprise memory storing instructions. In an embodiment, the system may comprise at least one processor including processing circuitry, memory storing instructions that, when executed by the at least one processor individually or collectively, cause the system to perform the instructions. In an embodiment, the at least one processor may cause the system to obtain a first set of IMU values between two consecutive images captured using at least one camera of the HMD. In an embodiment, the at least one processor may cause the system to obtain a second set of IMU values by appending previous one or more IMU values to the first set of IMU values, wherein the second set of IMU values includes total sample size equal to defined sliding window length which indicates input size of the previous one or more IMU values. In an embodiment, the at least one processor may cause the system to obtain a set of denoised IMU values by processing the second set of IMU values. In an embodiment, the at least one processor may cause the system to determine one or more noise parameters present in the first set of IMU values by correlating the set of denoised IMU values with the first set of IMU values. In an embodiment, the at least one processor may cause the system to determine a modified sliding window length based on the one or more noise parameters and a resource utilization parameters of the HMD. In an embodiment, the at least one processor may cause the system to perform IMU data enhancement on a third set of IMU values based on the modified sliding window length, the third set of IMU values is subsequent to the first set of IMU values.

[0008] One embodiment provides a machine readable medium containing instructions. The instructions, when executed by at least one processor, may cause the at least one processor to perform the method corresponding.

[0009] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate an embodiment and, together with the description, serve to explain the disclosed principles. The same numbers are used throughout the figures to reference like features and components. An embodiment of at least one of device and methods in accordance with an embodiment of the present subject matter are now described, by way of example only, and with reference to the accompanying figures, in which:

[0010] FIG. 1 illustrates an environment for enhancing IMU data in real time for a head mounted display (HMD) device, in accordance with an embodiment of the present disclosure;

[0011] FIG. 2 illustrates pose prediction trajectories comparison, in accordance with an embodiment of the present disclosure;

[0012] FIG. 3A illustrates a block diagram for enhancing IMU data in real time for a head mounted display (HMD) device, in accordance with an embodiment of the present disclosure;

[0013] FIG. 3B illustrates a block diagram for refining IMU values, in accordance with an embodiment of the present disclosure;

[0014] FIG. 3C illustrates a block diagram for deriving one or more noise parameters, in accordance with an embodiment of the present disclosure;

[0015] FIG. 4 illustrates a block diagram of a system for enhancing IMU data in real time for a head mounted display (HMD) device, in accordance with an embodiment of the present disclosure; and

[0016] FIG. 5 illustrates a flowchart for a method for enhancing IMU data in real time for a head mounted display (HMD) device, in accordance with an embodiment of the present disclosure;

[0017] It may be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative systems embodying the principles of the present subject matter. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0018] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0019] While the disclosure is susceptible to various modifications and alternative forms, specific embodiment thereof has been shown by way of example in the drawings and will be described in detail below. It can be understood, however that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover a plurality of modifications, equivalents, and alternative falling within the spirit and the scope of the disclosure.

[0020] The terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a setup, device, or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a device or system or apparatus proceeded by "comprises... a" does not, without more constraints, preclude the existence of other elements or additional elements in the device or system or apparatus.

[0021] In the following detailed description of the embodiments of the disclosure, reference is made to the accompanying drawings that form a part thereof, and in which are shown by way of illustration specific embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0022] The terminology "Artificial intelligence (AI) model" and "neural network" are interchangeably used throughout the specification. The AI module may be a combination of hardware module and software module. The hardware module may comprise necessary circuitry to perform the functionality discussed in the embodiments below.

[0023] The terminology "HMD" and "HMD device" have same meaning and are alternatively used throughout the specification.

[0024] The terminology "SLAM" and "VI SLAM" have same meaning and are alternatively used throughout the specification.

[0025] There is a need for a methodology to improve the noisy IMU data in real-time and reduce alignment errors. In view of the foregoing discussion, there exists a need in the art for an artificial intelligence model strategy that provides IMU data enhancement in real time to aid the efficiency and accuracy of Visual-Inertial (VI) SLAM and overcome the above-mentioned limitations present in the related technology.

[0026] Embodiments of the present disclosure relate to method and system for enhancing IMU data in real time for a head mounted display (HMD) device. According to an embodiment of the present disclosure, the system for enhancing IMU data in real time for a head mounted display (HMD) device may refine noisy IMU values using light-weight deep learning models. Accordingly, the refined IMU values may be provided to a VI SLAM thereby enabling IMU refinement to be done efficiently on the device itself. Thus, the IMU values refined according to the system and method of the present disclosure enables prediction of more accurate and stable poses in real-time even in dynamic environments and even in cases wherein the environment surrounding the HMD device is dark. Therefore, the system and method of the present disclosure balances between efficiency of the HMD as well as accuracy in pose prediction dynamically in any environment.

[0027] FIG. 1 illustrates an environment 100 for pose prediction in real time for a head mounted display (HMD) device, in accordance with an embodiment of the present disclosure. The environment may include an HMD device and a system 110 may be externally connected to the HMD device. The environment 100 depicts a user 101, an IMU sensor 105, a camera 107, a VI Odometry 109, a system 110, a VI Slam 111, and a display 113.

[0028] The IMU sensor 105 may be configured to take a user movement as input from the user 101. In an embodiment, the input may include IMU data which comprises of an angular velocity and linear acceleration corresponding to the user movement. Further, the camera 107 may be configured to capture one or more images of the user's field of view. The VI odometry 109 may be configured to estimate the user pose and the user's movements based on the sequence of images captured by the camera 107.

[0029] The system 110 may be configured to take the input from the IMU sensor 105 and provide the refined IMU values to the VI SLAM 111. The refinement of the IMU values is discussed in further detail in the below embodiments.

[0030] The VI SLAM 111 may be configured to simulate the user's position and movement to display the simulated image or video on the display 113. However, the functionality of the VI SLAM 111 is not limited to above explanation and any other functionality of the VI SLAM 111 is well within the scope of the present disclosure.

[0031] FIG. 2 illustrates pose prediction trajectories comparison, in accordance with an embodiment of the present disclosure.

[0032] As shown in FIG. 2, the trajectory 201a illustrates the user movements simulated by the VI SLAM 111 without the IMU value refinement. In an example, the user may be in a dark environment which may not have sufficient lighting and may also be performing several movements in a short period of time. In such conditions where images captured by the camera are dark and lack detectable features for visual pose estimation, the accuracy of inertial poses obtained from the IMU sensor data may be used for user pose estimation. However, the IMU values captured by the IMU sensor 105 may have several noise parameters which if not refined may lead to the SLAM 111 inaccurately simulating the user movements as shown by the trajectory 201a.

[0033] In an embodiment of the present disclosure, the system 110 for refining one or more of the IMU values may be implemented on an HMD device. These IMU values may have a varied level of noise parameters based on the user movements as well as the surroundings of the user. If the captured IMU values are first refined by the system 110 prior to providing to the SLAM 111 and fused with visual pose, the user movements i.e. final pose of the user may be accurately simulated as shown by trajectory 201b.

[0034] FIG. 3A illustrates a block diagram for enhancing IMU data in real time for a head mounted display (HMD) device, in accordance with an embodiment of the present disclosure.

[0035] As shown in FIG. 3A, the set of raw IMU values denoted as 'k', which are captured between the image frames Ttand Tt+1may be provided as input to the system 300. The visual data Ttand Tt+1corresponds to two consecutive image frames that may be captured by a camera, as discussed in the above an embodiment. The set of raw IMU values 'k' may be captured by an IMU sensor. The IMU data may comprise of angular velocity and linear acceleration values corresponding to a user movement.

[0036] Initially, a pre-determined sliding window length 'L' is initialized. Thereafter, a set of pre-stored IMU values are retrieved by the system 300 from an IMU database 311. The length of the pre-stored IMU values corresponds to 'L-k', where 'k' represents the set of raw IMU values. The pre-stored IMU values 'L-k' are used to complete the sliding window to its full length of 'L'.

[0037] A second set of IMU values is obtained by appending the set of raw IMU values 'k' to the pre-stored IMU values. Thereafter, the second set of IMU values are provided as an input to the first AI model 305. The first AI model 305 processes the second set IMU values to obtain a set of denoised IMU values.

[0038] The set of raw IMU values k and the set of denoised IMU values are provided to a noise calculation unit 309. One or more noise parameters 'μ', 'σ' present in the set of raw IMU values 'k' are determined by the noise calculation unit 309. The noise parameters may include mean 'μ', standard deviation 'σ', etc. The determination of the noise parameters 'μ', 'σ' is discussed in further detail in the below an embodiment. However, the noise parameters are not limited to above example and any other noise parameter known to a person skilled in the art is well with the scope of the present disclosure.

[0039] Thereafter, the set of denoised IMU values are provided by the first AI model 305 to a SLAM (not shown) of the HMD for IMU pre-integration. The denoised set of IMU values and the visual data Ttand Tt+1are used to carry out the visual-inertial alignment and to simulate the user movements. The display to the user is modified on the HMD display based on the visual-inertial alignment.

[0040] Further, the noise parameters 'μ' and 'σ' are provided as input to a second AI model 307 for processing. The second AI model determines an updated sliding window length 'L' based on the noise parameters 'μ', 'σ'. In an embodiment, the second AI model may determine the updated sliding window length 'L' based on the noise parameters and a resource utilization of the HMD. The updated sliding window length 'L' may be used to process the next set of raw IMU values that the system 300 may receive as input. Thus, the sliding window length 'L' may be varied based on the noise present in the captured IMU values of the IMU sensor, thereby improving the response time of the HMD display. In an embodiment, the second AI model may be a regression model. However, the second AI model is not limited to above example and any other model known to a person skilled in the art is well within the scope of the present disclosure.

[0041] FIG. 3B illustrates a block diagram of a first AI model 305, in accordance with an embodiment of the present disclosure. In an embodiment, the first AI model 305 may be a deep learning model composed of dilated causal convolutions or a convolution neural network model.

[0042] In one embodiment, a second set of IMU values obtained by appending the set of raw IMU values 'k' with the pre-stored IMU values 'L-k' are provided as input to the first AI model 305. In an embodiment, the first AI model 305 comprises of five hidden layers which process the noisy IMU values and enhance the IMU values to obtain a set of denoised IMU values. When a new camera frame is received by the system, the second set of IMU values are supplied to the first AI model 305. Based on the range of the sliding window length L, the first AI model 305 may disable one or more convolutional blocks of the first AI model for processing the new input. This helps to keep the inference time of the first AI model 305 in check, even when a large sliding window length needs to be processed for denoising the IMU values.

[0043] Once the set of denoised IMU values are obtained from the first AI model 305, the set of denoised IMU values along with their corresponding set of raw IMU values 'k' are provided to a noise calculation module as input, for determining the noise parameters in the set of raw IMU values. The determination of the noise parameters which may include determination of a mean 'μ' and a standard deviation 'σ' is discussed in further detail in the below an embodiment.

[0044] In one embodiment, the first AI model 305 may be trained to obtain a set of denoised IMU values. A sample set of IMU values and a corresponding sample set of ground truth values comprising translation and rotation information are provided to the first AI model 305. The first AI model is trained based on the sample set of IMU values and the corresponding sample set of ground truth values for estimating angular velocity and linear acceleration of a user in real-time.

[0045] FIG. 3C illustrates a block diagram for deriving one or more noise parameters, in accordance with an embodiment of the present disclosure.

[0046] The set of denoised IMU values obtained by the first AI model 305 and the set of raw IMU values k are provided to a noise calculation unit 309 to determine one or more noise parameters present in the set of raw IMU values. The noise parameters may include a mean deviation 'μ' and a standard deviation 'σ'. The set of raw IMU values k are compared with the set of denoised IMU values for the purpose of error computation to obtain the error vector. Thereafter, feature engineering is applied to the error vector to obtain the noise parameters 'μ' and 'σ'.

[0047] FIG. 4 illustrates a block diagram of a system for enhancing IMU data in real time for a head mounted display (HMD) device, in accordance with an embodiment of the present disclosure. In one embodiment, the system 400 may be similar to the system of FIG. 1 and FIG. 3.

[0048] In an embodiment of the present disclosure, the system 400 may comprise memory 403, at least one processor 401, a first AI model 405, and a second AI model 407 communicatively coupled with each other. In an embodiment, the system 400 may also comprise an input unit (not shown), output module (not shown) and communication interface (not shown).

[0049] It may be noted that, in an embodiment, the system 400 may include more or fewer components than those depicted herein. The various components of the system 400 may be implemented using hardware, software, firmware or any combinations thereof. Further, the various components of the system 400 may be operably coupled with each other. In an embodiment, various components of the system 400 may be capable of communicating with each other using communication channel media (such as buses, interconnects, etc.).

[0050] In one embodiment, the at least one processor 401 may be embodied as a multi-core processor, a single core processor, or a combination of one or more multi-core processors and one or more single core processors. For example, the at least one processor 401 may be embodied as one or more of various processing devices, such as a coprocessor, a microprocessor, a controller, a digital signal processor (DSP), a processing circuitry with or without an accompanying DSP, or various other processing devices including, a microcontroller unit (MCU), a hardware accelerator, a special-purpose computer chip, or the like.

[0051] In one embodiment, the memory 403 is capable of storing machine executable instructions, referred to herein as instructions. In an embodiment, the at least one processor 401 is embodied as an executor of software instructions. As such, the at least one processor 401 is capable of executing the instructions stored in the memory 403 to perform one or more operations described herein.

[0052] The memory 403 can be any type of storage accessible to the at least one processor 401 to perform respective functionalities. For example, the memory 403 may include one or more volatile or non-volatile memories, or a combination thereof. For example, the memory 403 may be embodied as semiconductor memories, such as flash memory, mask ROM, PROM (programmable ROM), EPROM (erasable PROM), RAM (random access memory), etc. and the like.

[0053] In an embodiment of the present disclosure, the at least one processor 401 may be configured to initialize a sliding window length for the first AI model 405. The sliding window length is equivalent to the input size of sample set of IMU values that needs to be provided to the first AI model 405 for accurately denoising the IMU values of the IMU sensor.

[0054] Further, the at least one processor 401 may be configured to obtain a set of raw IMU values from an IMU sensor for pose prediction in real time for a head mounted display (HMD) device. The set of raw IMU values may be captured by the IMU sensor between two consecutive images captured using at least one camera of the HMD device. In an embodiment of the present disclosure, the set of raw IMU values are taken only between two consecutive images for pre-integration with visual data and accurately predict the pose of the user wearing the HMD device.

[0055] The one or more raw IMU values may comprise an angular velocity and a linear acceleration corresponding to a movement of the user. In an embodiment, the set of raw IMU values may be provided through the input unit (not shown) to the system 400.

[0056] The at least one processor 401 may be configured to retrieve one or more pre-stored IMU values from an IMU database of an HMD device. The one or more pre-stored IMU values are retrieved based on the sliding window length and the first set of IMU values. In an embodiment, the pre-stored IMU values may be provided through the input unit (not shown) to the system 400. In an embodiment, the pre-stored IMU values may be stored in the memory 403 and may be retrieved for pose prediction. In an embodiment, the IMU values may be stored in cloud and may be retrieved from cloud for pose prediction.

[0057] The at least one processor 401 may be configured to obtain a second set of IMU values by appending the retrieved pre-stored IMU values to the set of raw IMU values. The second set of IMU values includes total sample size equal to the initialized sliding window length. Further, the at least one processor 401 may be configured to provide the second set of IMU values to the first AI model 405 as input.

[0058] In one embodiment, the first AI model 405 may be configured to obtain a set of denoised IMU values using the procedure discussed in above an embodiment. In an embodiment, the first AI model 405 may be off-the-self AI model being trained for denoising the IMU values of the IMU sensor.

[0059] In an embodiment, the at least one processor 401 may be configured to provide, to the first AI model, a sample set of IMU values and a corresponding sample set of ground truth pose comprising translation and rotation information and train the first AI model based on the sample set of IMU values and the corresponding sample set of ground truth pose for estimating angular velocity and linear acceleration of a user.

[0060] After the set of denoised IMU values have been obtained by the first AI model 405, the at least one processor 401 may be configured to determine one or more noise parameters present in the set of raw IMU values by correlating the set of denoised IMU values with the set of raw IMU values, as shown in FIG. 3C.

[0061] The one or more noise parameters may be determined using the procedure discussed in above an embodiment. The one or more noise parameters may include mean and standard deviation. In an embodiment of the present disclosure, the derivation of the one or more noise parameters may also be done by a noise calculation unit (not shown).

[0062] Once one or more noise parameters are determined, the at least one processor 301 may also be configured to provide the one or more noise parameters as well as a resource utilization of the HMD device on which the said system 400 is deployed as input to the second AI model 407 to determine the modified sliding window length. In an embodiment, the modified sliding window length may only be determined based on the one or more noise parameters. The modified sliding window length may be used by the system 400 to process the next set of raw IMU values that the system 400 may obtain as input after the first set of IMU values.

[0063] Furthermore, based on the modified sliding window length, the at least one processor 401 may be configured to compare the modified sliding window length with a plurality of ranges of sliding window lengths and adjust the number of operational convolution blocks within the first AI model 405 based on the comparison. The adjustment of the number of operational convolution blocks is done to maintain the computational load of the system at an optimal level.

[0064] In one embodiment, the second AI model 407 may be trained using a sample set of noise parameters, a resource utilization parameters, and corresponding optimal sliding window length. The resource utilization parameters may comprise a CPU utilization, memory usage, disk I / O rate, and battery status of the HMD device. However, the resource utilization parameters is not limited to above example and any other resource utilization parameter known to a person skilled in the art is well within the scope of present disclosure.

[0065] In one embodiment, the second AI model 407 may be a light gradient boosting model (LightGBM) that may be trained to generate the optimal sliding window length at a particular time. For example, the second AI model 407 may be trained such that in case of high resource usage demands, the sliding window length may be regressed to lower the computational overhead. In an example, the second AI model 407 may be trained such that in case of high noise in the raw IMU values, the sliding window length may be increased to balance the efficiency of the output as well as maintain the computational overhead at an optimum level. Table 1 below provides an example of sliding window lengths for varied values of the deployed system's resource usage.

[0066]

[0067] The at least one processor 401 may be configured to provide the set of denoised IMU values to the HMD device to perform IMU pre-integration on the set of denoised IMU values. The inertial pose of the user is generated based on the pre-integrated IMU values. Lastly, the inertial pose of the user and visual data captured by the at least one camera of the HMD device are processed by the HMD device to obtain a final pose of the user.

[0068] In an embodiment, the at least one processor 401 may be configured to perform IMU pre-integration on the set of denoised IMU values, generate inertial pose of the user based on the pre-integrated IMU values, and process the inertial pose of the user and visual data captured by the at least one camera of the HMD, to obtain a final pose of the user.

[0069] FIG. 5 illustrates a flowchart for a method for enhancing IMU data in real time for a head mounted display (HMD) device, in accordance with an embodiment of the present disclosure.

[0070] At step 502, the method 500 discloses obtaining a first set of IMU values between two consecutive images captured using at least one camera of the HMD. The first set of IMU values may comprise of an angular velocity and linear acceleration corresponding to a user movement.

[0071] Thereafter, at step 504, the method 500 discloses obtaining a second set of IMU values by appending previous one or more IMU values to the first set of IMU values. The second set of IMU values includes total sample size equal to defined sliding window length. The sliding window length indicates a sample size of Inertial Measurement Unit (IMU) values provided as input to the first AI model for denoising the IMU data. The previous one or more IMU values are retrieved at least based on the sliding window length and the first set of IMU values. The previous one or more IMU values are collected during the capturing of one or more previous images by the at least one camera of the HMD device and are stored in the IMU database.

[0072] In an embodiment of the present disclosure, the method 500 includes collecting a plurality of IMU values during the capturing of one or more previous images using the at least one camera of the HMD and storing the plurality of IMU values in the IMU database.

[0073] At step 506, the method 500 discloses obtaining a set of denoised IMU values by processing the second set of IMU values.

[0074] In an embodiment, the obtaining a set of denoised IMU values by processing the second set of IMU values may be performed using first AI model. The first AI model generates a set of denoised IMU values based on the training. The training of the first AI model comprises providing, to the first AI model, a sample set of IMU values and a corresponding sample set of ground truth pose comprising translation and rotation information and training the first AI model based on the sample set of IMU values and the corresponding sample set of ground truth pose for estimating angular velocity and linear acceleration of a user.

[0075] Further, the method 500 comprises performing IMU pre-integration on the set of denoised IMU values, generating inertial pose of the user based on the pre-integrated IMU values, and processing the inertial pose of the user and visual data captured by the at least one camera of the HMD, to obtain a final pose of the user.

[0076] At step 508, the method 500 discloses determining one or more noise parameters present in the first set of IMU values by correlating the set of denoised IMU values with the first set of IMU values. The noise parameters may include a mean and a standard deviation.

[0077] The deriving of the one or more noise parameters present in the first set of IMU values comprises determining noise in each IMU value of the first set of IMU values based on the set of denoised IMU values with the first set of IMU values and determining a mean and a standard deviation of noise present in the first set of IMU values.

[0078] At step 510, the method 500 discloses determining a modified sliding window length based on the one or more noise parameters. In an embodiment, determining a modified sliding window length based on the one or more noise parameters may be performed using a second AI model. In an embodiment, the determination of the modified sliding window length comprises determining, by the second AI model, a modified sliding window length further based on resource utilization parameters of the HMD.

[0079] In an embodiment of the present disclosure, the method 500 may comprise providing, to the second AI model, a sample set of noise parameters, the resource utilization parameters, and corresponding optimal sliding window length and training the second AI model based on the sample set of noise parameters, the resource utilization parameters, and the corresponding optimal sliding window length. The resource utilization parameters at least comprise CPU utilization, memory usage, disk I / O rate, and battery status of the HMD.

[0080] At step 512, the method 500 discloses performing the IMU data enhancement on a third set of IMU values based on the modified sliding window length. The third set of IMU values is subsequent to the first set of IMU values.

[0081] In an embodiment of the present disclosure, while performing the denoising on the third set of IMU values, the method 500 comprises comparing the modified sliding window length with a plurality of ranges of sliding window length and adjusting number of operational convolution blocks within the first AI model based on the comparison.

[0082] Thus, the method 500 facilitates reducing the inference time of the first AI model for the increased sliding window length for the accurate denoised IMU values.

[0083] The sequence of operations of the method 500 need not be necessarily executed in the same order as they are presented. Further, one or more operations may be grouped together and performed in form of a single step, or one operation may have several sub-steps that may be performed in parallel or in sequential manner.

[0084] The disclosed method with reference to Fig. 5, or one or more operations of the system 400 explained with reference to fig. 5 may be implemented using software including computer-executable instructions stored on one or more computer-readable media (e.g., non-transitory computer-readable media, such as one or more optical media discs, volatile memory components (e.g., DRAM or SRAM), or non-volatile memory or storage components (e.g., hard drives or solid-state non-volatile memory components, such as Flash memory components) and executed on a computer (e.g., any suitable computer, such as a laptop computer, net book, Web book, tablet computing device, smart phone, or other mobile computing device). Such software may be executed, for example, on a single local computer.

[0085] Furthermore, one or more computer-readable storage media may be utilized in implementing embodiments consistent with the present disclosure. A computer-readable storage medium refers to any type of physical memory on which information or data readable by a processor may be stored. Thus, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor(s) to perform steps or stages consistent with the embodiments described herein. The term "computer-readable medium" may be understood to include tangible items and exclude carrier waves and transient signals, i.e., be non-transitory. Examples include Random Access Memory (RAM), Read-Only Memory (ROM), volatile memory, non-volatile memory, hard drives, CD (Compact Disc) ROMs, DVDs, flash drives, disks, and any other known physical storage media.

[0086] It will be understood by those within the art that, in general, terms used herein, and are generally intended as "open" terms (e.g., the term "including" may be interpreted as "including but not limited to," the term "having" may be interpreted as "having at least," the term "includes" may be interpreted as "includes but is not limited to," etc.). For example, as an aid to understanding, the detail description may contain usage of the introductory phrases "at least one" and "one or more" to introduce recitations. However, the use of such phrases may not be construed to imply that the introduction of a recitation by the indefinite articles "a" or "an" limits any particular part of description containing such introduced recitation to disclosure containing only one such recitation, even when the introductory phrases "one or more" or "at least one" and indefinite articles such as "a" or "an" (e.g., "a" and / or "an" may typically be interpreted to mean "at least one" or "one or more") are included in the recitations; the same holds true for the use of definite articles used to introduce such recitations. In addition, even if a specific part of the introduced description recitation is explicitly recited, those skilled in the art will recognize that such recitation may typically be interpreted to mean at least the recited number (e.g., the bare recitation of "two recitations," without other modifiers, typically means at least two recitations or two or more recitations).

[0087] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following detailed description.

[0088] In an embodiment, the method may include initializing a sliding window length for a first artificial intelligence (AI) model for denoising the IMU data, wherein the sliding window length indicates input sample size of Inertial Measurement Unit values provided to the first AI model. In an embodiment, the method may include retrieving one or more IMU values prestored in an IMU database at least based on the sliding window length and the first set of IMU values. In an embodiment, the method may include providing, to the first AI model, the second set of IMU values to generate a set of denoised IMU values.

[0089] In an embodiment, wherein the IMU values comprise an angular velocity and a linear acceleration corresponding to a user movement.

[0090] In an embodiment, the method may include providing, to the first AI model, a sample set of IMU values and a corresponding sample set of ground truth pose comprising translation and rotation information. In an embodiment, the method may include training the first AI model based on the sample set of IMU values and the corresponding sample set of ground truth pose for estimating angular velocity and linear acceleration of a user.

[0091] In an embodiment, the method may include performing IMU pre-integration on the set of denoised IMU values. In an embodiment, the method may include obtaining inertial pose of the user based on the pre-integrated IMU values. In an embodiment, the method may include processing the inertial pose of the user and visual data captured by the at least one camera of the HMD, to obtain a final pose of the user.

[0092] In an embodiment, the method may include providing, to the second AI model, a sample set of noise parameters, the resource utilization parameters, and corresponding optimal sliding window length, wherein the resource utilization parameters at least comprise CPU utilization, memory usage, disk I / O rate, and battery status of the HMD. In an embodiment, the method may include training the second AI model based on the sample set of noise parameters, the resource utilization parameters, and the corresponding optimal sliding window length.

[0093] In an embodiment, the method may include collecting a plurality of IMU values during the capturing of one or more previous images using the at least one camera of the HMD. In an embodiment, the method may include storing the plurality of IMU values in the IMU database.

[0094] In an embodiment, the obtaining a set of denoised IMU values by processing the second set of IMU values and the determining the modified sliding window length based on the one or more noise parameters may be performed using at least one AI model.

[0095] In an embodiment, the method may include comparing the modified sliding window length with a plurality of ranges of sliding window length. In an embodiment, the method may include adjusting number of operational convolution blocks within the AI model based on the comparison.

[0096] In an embodiment, the determining one or more noise parameters present in the first set of IMU values may comprise determining noise in at least one IMU value of the first set of IMU values based on the set of denoised IMU values with the first set of IMU values. In an embodiment, the determining one or more noise parameters present in the first set of IMU values may comprise determining a mean and a standard deviation of the noise present in the first set of IMU values.

[0097] In an embodiment, the determining the modified sliding window length may comprise determining the modified sliding window length further based on resource utilization parameters of the HMD.

[0098] In an embodiment, the system may comprise a first artificial intelligence (AI) model. In an embodiment, the system may comprise a second AI model. In an embodiment, the at least one processor may cause the system to initialize a sliding window length for a first artificial intelligence (AI) model for denoising the IMU data, wherein the sliding window length indicates input sample size of Inertial Measurement Unit (IMU) values provided to the first AI model. In an embodiment, the at least one processor may cause the system to retrieve one or more IMU values prestored in an IMU database at least based on the sliding window length and the first set of IMU values. In an embodiment, the at least one processor may cause the system to provide, to the first AI model, the second set of IMU values to generate a set of denoised IMU values.

[0099] In an embodiment, the IMU values may comprise an angular velocity and a linear acceleration corresponding to a user movement.

[0100] In an embodiment, the at least one processor may cause the system to provide, to the first AI model, a sample set of IMU values and a corresponding sample set of ground truth pose comprising translation and rotation information. In an embodiment, the at least one processor may cause the system to train the first AI model based on the sample set of IMU values and the corresponding sample set of ground truth pose for estimating angular velocity and linear acceleration of a user.

[0101] In an embodiment, the at least one processor may cause the system to perform IMU pre-integration on the set of denoised IMU values. In an embodiment, the at least one processor may cause the system to obtain inertial pose of the user based on the pre-integrated IMU values. In an embodiment, the at least one processor may cause the system to process the inertial pose of the user and visual data captured by the at least one camera of the HMD, to obtain a final pose of the user.

[0102] In an embodiment, the at least one processor may cause the system to provide, to the second AI model, a sample set of noise parameters, the resource utilization parameters, and corresponding optimal sliding window length, wherein the resource utilization parameters at least comprise CPU utilization, memory usage, disk I / O rate, and battery status of the HMD. In an embodiment, the at least one processor may cause the system to train the second AI model based on the sample set of noise parameters, the resource utilization parameters, and the corresponding optimal sliding window length.

[0103] In an embodiment, the at least one processor may cause the system to collect a plurality of IMU values during the capturing of one or more previous images using the at least one camera of the HMD. In an embodiment, the at least one processor may cause the system to store the plurality of IMU values in the IMU database.

[0104] In an embodiment, the at least one processor may cause the system to compare the modified sliding window length with a plurality of ranges of sliding window length. In an embodiment, the at least one processor may cause the system to adjust number of operational convolution blocks within the first AI model based on the comparison.

[0105] In an embodiment, to determine one or more noise parameters present in the first set of IMU values, the at least one processor may cause the system to determine noise in at least one IMU value of the first set of IMU values based on the set of denoised IMU values with the first set of IMU values. In an embodiment, to determine one or more noise parameters present in the first set of IMU values, the at least one processor may cause the system to determine a mean and a standard deviation of the noise present in the first set of IMU values.

[0106] In an embodiment, to determine the modified sliding window length, the at least one processor may cause the system to determine the modified sliding window length further based on resource utilization parameters of the HMD.

Claims

1.A method for enhancing inertial measurement unit (IMU) data in real time for a head mounted display (HMD) device, the method comprising:obtaining a first set of IMU values between two consecutive images captured using at least one camera of the HMD;obtaining a second set of IMU values by appending previous one or more IMU values to the first set of IMU values, wherein the second set of IMU values includes total sample size equal to defined sliding window length which indicates input size of the previous one or more IMU values;obtaining a set of denoised IMU values by processing the second set of IMU values;determining one or more noise parameters present in the first set of IMU values by correlating the set of denoised IMU values with the first set of IMU values;determining a modified sliding window length based on the one or more noise parameters; andperforming the IMU data enhancement on a third set of IMU values based on the modified sliding window length, wherein the third set of IMU values is subsequent to the first set of IMU values.2.The method as claimed in claim 1, wherein the IMU values comprise an angular velocity and a linear acceleration corresponding to a user movement.3.The method any one of claims 1 to 2, further comprising:performing IMU pre-integration on the set of denoised IMU values;obtaining inertial pose of the user based on the pre-integrated IMU values; andprocessing the inertial pose of the user and visual data captured by the at least one camera of the HMD, to obtain a final pose of the user.4.The method any one of claims 1 to 3, further comprising:collecting a plurality of IMU values during the capturing of one or more previous images using the at least one camera of the HMD; andstoring the plurality of IMU values in the IMU database.5.The method any one of claims 1 to 4, wherein the obtaining the set of denoised IMU values by processing the second set of IMU values and the determining the modified sliding window length based on the one or more noise parameters are performed using at least one AI model.6.The method as claimed in claim 5, further comprising:comparing the modified sliding window length with a plurality of ranges of sliding window length; andadjusting number of operational convolution blocks within the at least one AI model based on the comparison.7.The method any one of claims 1 to 6, wherein the determining one or more noise parameters present in the first set of IMU values comprises:determining noise in at least one IMU value of the first set of IMU values based on the set of denoised IMU values with the first set of IMU values; anddetermining a mean and a standard deviation of the noise present in the first set of IMU values.8.The method any one of claims 1 to 7, wherein the determining the modified sliding window length comprises determining the modified sliding window length further based on resource utilization parameters of the HMD.9.A system for enhancing inertial measurement unit (IMU) data in real time for a head mounted display (HMD) device, the system comprising:memory storing instructions; andat least one processor including processing circuitry, memory storing instructions that, when executed by the at least one processor individually or collectively, cause the system to:obtain a first set of IMU values between two consecutive images captured using at least one camera of the HMD;obtain a second set of IMU values by appending previous one or more IMU values to the first set of IMU values, wherein the second set of IMU values includes total sample size equal to defined sliding window length which indicates input size of the previous one or more IMU values;obtain a set of denoised IMU values by processing the second set of IMU values;determine one or more noise parameters present in the first set of IMU values by correlating the set of denoised IMU values with the first set of IMU values;determine a modified sliding window length based on the one or more noise parameters; andperform the IMU data enhancement on a third set of IMU values based on the modified sliding window length, wherein the third set of IMU values is subsequent to the first set of IMU values.10.The system as claimed in claim 9, wherein the IMU values comprise an angular velocity and a linear acceleration corresponding to a user movement.11.The system any one of claims 9 to 10, wherein the at least one processor cause the system to:perform IMU pre-integration on the set of denoised IMU values;obtain inertial pose of the user based on the pre-integrated IMU values; andprocess the inertial pose of the user and visual data captured by the at least one camera of the HMD, to obtain a final pose of the user.12.The system any one of claims 9 to 11, wherein the at least one processor cause the system to:collect a plurality of IMU values during the capturing of one or more previous images using the at least one camera of the HMD; andstore the plurality of IMU values in the IMU database.13.The system any one of claims 9 to 12, wherein to determine one or more noise parameters present in the first set of IMU values, the at least one processor cause the system to:determine noise in at least one IMU value of the first set of IMU values based on the set of denoised IMU values with the first set of IMU values; anddetermine a mean and a standard deviation of the noise present in the first set of IMU values.14.The system any one of claims 9 to 13, wherein to determine the modified sliding window length, the at least one processor cause the system to determine the modified sliding window length further based on resource utilization parameters of the HMD.15.A machine readable medium containing instructions, wherein the instructions, when executed by at least one processor, cause the at least one processor to perform the method of any one of claims 1 to 8.

Images