System and method for a re-localization of a head-mounted display device

Abstract

A method for a re-localization of a HMD device may comprise obtaining feature points of objects identified in their current position. The method may comprise obtaining landmark points associated with each of the identified objects from the feature points corresponding to pre-stored landmark points associated with each of the identified objects. The method may comprise determining whether any objects have moved to the current position from a pre-stored position by comparing the obtained landmark points with the pre-stored landmark points associated with each of the objects. The method may comprise filtering the landmark points by discarding the feature points associated with the objects that have moved to the current position from the pre-stored position. The method may comprise performing re-localization of the HMD device using the filtered landmark points.

Description

SYSTEM AND METHOD FOR A RE-LOCALIZATION OF A HEAD-MOUNTED DISPLAY DEVICE

[0001] The present disclosure relates to image mapping in a dynamic environment and more particularly, relates to a system and a method for enhanced re-localization of a head-mounted display (HMD) device.

[0002] Nowadays, in mapped environments, only localization is needed for pose estimation in a head-mounted display (HMD) device as the environment is known. Typically, the localization may refer to a process of determining a precise position and orientation of the HMD in a given environment. This may be crucial for applications in a virtual reality (VR), augmented reality (AR), and / or mixed reality (MR), where accurate tracking of a user's head movements is essential for an immersive experience. The HMD may utilize simultaneous localization and mapping (SLAM) method to build a map and localize the HMD in the built map at the same time.

[0003] In operation, a typical HMD may capture images and extract features of feature points from the captured images. The features or the feature points may refer to any identifiable or characteristic part of one captured image that can be used to distinguish the image from other captured images. Usually in the captured images, the feature points may correspond to edges and corners. Additionally, the feature points may correspond to shapes, textures, or patterns that change in viewpoint, illumination, and scale. The HMD device may utilize the extracted feature points to construct a map of the environment. This constructed map may evolve as new feature points are observed and incorporated. Simultaneously, the HMD's position and orientation may be estimated relative to the constructed map. This involves comparing the feature points observed by the HMD with the feature points in the constructed map to determine the HMD's pose, i.e., HMD's position and orientation.

[0004] Similarly, re-localization may refer to a process of determining the HMD's position and orientation within a known environment after the HMD has temporarily lost tracking or moved to a new location. Therefore, re-localization is a crucial aspect of the AR and VR experiences, as alignment of the virtual content with the physical world is ensured.

[0005] However, in dynamic environments, such as homes, there are multiple movable objects, such as doors and chairs, which are moved every now and then. Changes in the position of these objects may lead to localization failures in the HMD devices. Further, adding new feature points without removing old ones may often clutter the constructed map. This may further lead to memory overflow, high power consumption, and reduced localization accuracy or localization failure.

[0006] Conventionally, a user of an HMD device may render a virtual application window overlaid at a first position of a real-world environment by mapping feature points on objects present in the environment. There exists a high probability that the objects in the dynamic environment having mapped feature points may move. Therefore, on subsequent use of HMD in the same real-world environment, the HMD device may render the virtual application window misplaced from the first position due to re-localization errors, causing frustration. This re-localization error arises due to the movement of the objects in the dynamic environment having mapped feature points. In severe cases, the application window does not appear at all due to re-localization failure. Such issues hinder users' perception of a product's dynamic home settings which can cause the SLAM algorithm to duplicate feature points, leading to memory overload and reduced accuracy. Further, inefficient mapping in the dynamic environment may increase power usage, decreasing runtime and causing heating problems.

[0007] This summary is provided to introduce a selection of concepts, in a simplified format, that are further described in the detailed description of the disclosure. This summary is neither intended to identify key or essential inventive concepts of the disclosure nor is it intended for determining the scope of the disclosure.

[0008] According to an embodiment of the disclosure, a method for a re-localization of a head-mounted display (HMD) device may be disclosed. The method may comprise obtaining (2102) one or more feature points of one or more objects identified in their current position in a real-world environment. The method may comprise obtaining (2104) one or more landmark points associated with each of the identified one or more objects from the one or more feature points corresponding to one or more pre-stored landmark points associated with each of the identified one or more objects of the real-world environment. The method may comprise determining (2106) whether any one or more objects have moved to the current position from a pre-stored position within the real-world environment by comparing the obtained one or more landmark points with the one or more pre-stored landmark points associated with each of the one or more objects of the real-world environment. The method may comprise filtering (2108) the one or more landmark points by discarding the one or more feature points associated with the one or more objects that have moved to the current position from the pre-stored position. The method may comprise performing (2110) re-localization of the HMD device (1000) using the filtered one or more landmark points.

[0009] According to an embodiment of the disclosure, a system (1002) for performing re-localization a head-mounted display (HMD) device (1000) is provided. The system (1002) may comprise at least one processor (1004) including processing circuitry. In an embodiment, the system (1002) may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system (1002) to obtain one or more current feature points of one or more objects identified in their current position in a real-world environment. In an embodiment, the system (1002) may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system to obtain one or more of landmark points associated with each of the identified one or more objects from the current feature points corresponding to one or more pre-stored landmark points associated with each of the identified one or more objects of the real-world environment. In an embodiment, the system may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the head-mounted display device (1000) to determine whether any one or more objects have moved to the current position from a pre-stored position within the real-world environmentby comparing the obtained one or more landmark points with the one or more pre-stored landmark points associated with each of the one or more objects of the real-world environment. In an embodiment, the system may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system to filter the one or more landmark points by discarding the current feature points associated with the one or more objects that have moved to the current position from the pre-stored position. In an embodiment, the system may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system to perform re-localization of the HMD device (1000) using the filtered one or more landmark points.

[0010] According to an embodiment of the disclosure, a computer-readable medium containing instructions is disclosed. The instructions, when executed by at least one processor, cause the head-mounted display (HMD) device (1000) to perform the method provided.

[0011] To further clarify the advantages and features of the present disclosure, a more particular description of the disclosure will be rendered by reference to specific embodiments thereof, which is illustrated in the appended drawing. It is appreciated that these drawings depict only typical embodiments of the disclosure and are therefore not to be considered limiting its scope. The disclosure will be described and explained with additional specificity and detail with the accompanying drawings.

[0012] The foregoing and other features ofembodiments will become more apparent from the following detailed description of embodiments when read in conjunction with the accompanying drawings. In the drawings, like reference numerals refer to like elements.

[0013] Fig. 1A and Fig. 1B illustrate an exemplary scenario of incorrect rendering of an application window due to objects' movement, in accordance with a related art;

[0014] Fig. 2 illustrates an exemplary scenario for a memory blowout in the HMD, in accordance with a related art;

[0015] Fig. 3 illustrates an example scenario of power consumption and accuracy with respect to feature points, in accordance with a related art;

[0016] Fig. 4A and Fig. 4B illustrate a pictorial depiction of relocation of application windows in a home / work environment, in accordance with a related art;

[0017] Fig. 5 illustrates a pictorial depiction of relocation of rendered application window in a parking space, in accordance with a related art;

[0018] Fig. 6 illustrates an exemplary pictorial depiction of usage of simultaneous localization and mapping (SLAM), in accordance with a related art;

[0019] Fig. 7 illustrates an exemplary scenario for localization using the SLAM, in accordance with a related art;

[0020] Fig. 8 illustrates an exemplary scenario for production of a pose using the localization from the SLAM, in accordance with a related art;

[0021] Fig. 9 illustrates an exemplary scenario for feature matching, in accordance with a related art;

[0022] Fig. 10 illustrates a block diagram of a head-mounted display (HMD) device, in accordance with an embodiment of the present disclosure;

[0023] Fig. 11 illustrates a functional block diagram of localization and mapping of the HMD device, in accordance with an embodiment of the present disclosure;

[0024] Fig. 12 illustrates an example implementation of the functional block diagram of localization and mapping of the HMD device, in accordance with an embodiment of the present disclosure;

[0025] Fig. 13 illustrates a functional block diagram of feature and object information extraction for localization of the HMD device, in accordance with an embodiment of the present disclosure;

[0026] Fig. 14 illustrates a functional block diagram of re-localization and mapping of the HMD device, in accordance with an embodiment of the present disclosure;

[0027] Fig. 15 illustrates an example implementation of the functional block diagram of re-localization and mapping of the HMD device, in accordance with an embodiment of the present disclosure;

[0028] Figs. 16A and 16B illustrate a pictorial depiction of rendering of an application window, in accordance with an embodiment of the present disclosure.

[0029] Fig. 17 illustrates a pictorial depiction of landmark points comparison, in accordance with an embodiment of the present disclosure;

[0030] Figs. 18A and 18B illustrate a pictorial depiction of object identification, in accordance with an embodiment of the present disclosure;

[0031] Fig. 19 illustrates a block diagram of the object information extractor module, in accordance with an embodiment of the present disclosure;

[0032] Fig. 20 illustrates a block diagram of the feature point filter module for landmark matching, in accordance with an embodiment of the present disclosure;

[0033] Fig. 21 illustrates a flow chart of a method for an enhanced re-localization of a head-mounted display (HMD) device, in accordance with an embodiment of the present disclosure; and

[0034] Fig. 22 illustrates an flow chart of a method for an enhanced re-localization of a head-mounted display (HMD) device, in accordance with an embodiment of the present disclosure.

[0035] Further, skilled artisans will appreciate that elements in the drawings are illustrated for simplicity and may not have necessarily been drawn to scale. For example, the flow charts illustrate the method in terms of the most prominent steps involved to help improve understanding of aspects of the present disclosure. Furthermore, in terms of the construction of the device, one or more components of the device may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details that are pertinent to understanding the embodiments of the present disclosure so as not to obscure the drawings with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

[0036] For the purpose of promoting an understanding of the principles of the present disclosure, reference will now be made to the various embodiments and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the present disclosure is thereby intended, such alterations and further modifications in the illustrated system, and such further applications of the principles of the present disclosure as illustrated therein being contemplated as would normally occur to one skilled in the art to which the present disclosure relates.

[0037] It will be understood by those skilled in the art that the foregoing general description and the following detailed description are explanatory of the present disclosure and are not intended to be restrictive thereof.

[0038] Whether or not a certain feature or element was limited to being used only once, it may still be referred to as “one or more features” or “one or more elements” or “at least one feature” or “at least one element.” Furthermore, the use of the terms “one or more” or “at least one” feature or element do not preclude there being none of that feature or element, unless otherwise specified by limiting language including, but not limited to, “there needs to be one or more…” or “one or more elements is required.”

[0039] Reference is made herein to some “embodiments.” It should be understood that an embodiment is an example of a possible implementation of any features and / or elements of the present disclosure. Some embodiments have been described for the purpose of explaining one or more of the potential ways in which the specific features and / or elements of the proposed disclosure fulfill the requirements of uniqueness, utility, and non-obviousness.

[0040] Use of the phrases and / or terms including, but not limited to, “a first embodiment,” “a further embodiment,” “an alternate embodiment,” “one embodiment,” “an embodiment,” “multiple embodiments,” “some embodiments,” “other embodiments,” “further embodiment”, “furthermore embodiment”, “additional embodiment” or other variants thereof do not necessarily refer to the same embodiments. Unless otherwise specified, one or more particular features and / or elements described in connection with one or more embodiments may be found in one embodiment, or may be found in more than one embodiment, or may be found in all embodiments, or may be found in no embodiments. Although one or more features and / or elements may be described herein in the context of only a single embodiment, or in the context of more than one embodiment, or in the context of all embodiments, the features and / or elements may instead be provided separately or in any appropriate combination or not at all. Conversely, any features and / or elements described in the context of separate embodiments may alternatively be realized as existing together in the context of a single embodiment.

[0041] Any particular and all details set forth herein are used in the context of some embodiments and therefore should not necessarily be taken as limiting factors to the proposed disclosure.

[0042] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process or method that comprises a list of steps does not include only those steps but may include other steps not expressly listed or inherent to such process or method. Similarly, one or more devices or sub-systems or elements or structures or components proceeded by “comprises... a” does not, without more constraints, preclude the existence of other devices or other sub-systems or other elements or other structures or other components or additional devices or additional sub-systems or additional elements or additional structures or additional components.

[0043] Embodiments of the present disclosure will be described below in detail with reference to the accompanying drawings.

[0044] For the sake of clarity, the first digit of a reference numeral of each component of the present disclosure is indicative of the Figure number, in which the corresponding component is shown. For example, reference numerals starting with digit “1” are shown at least in Fig. 1. Similarly, reference numerals starting with digit “2” are shown at least in Fig. 2.

[0045] It should be appreciated that the blocks in each flowchart and combinations of the flowcharts may be performed by one or more computer programs which include computer-executable instructions. The entirety of the one or more computer programs may be stored in a single memory or the one or more computer programs may be divided with different portions stored in different multiple memories.

[0046] Any of the functions or operations described herein can be processed by one processor or a combination of processors. The one processor or the combination of processors is circuitry performing processing and includes circuitry like an application processor (AP), a communication processor (CP), a graphical processing unit (GPU), a neural processing unit (NPU), a microprocessor unit (MPU), a system on chip (SoC), an IC, or the like.

[0047] The processor may include various processing circuitry and / or multiple processors. For example, as used herein, including the claims, the term “processor” may include various processing circuitry, including at least one processor, wherein one or more of at least one processor, individually and / or collectively in a distributed manner, may be configured to perform various functions described herein. As used herein, when “a processor”, “at least one processor”, and “one or more processors” are described as being configured to perform numerous functions, these terms cover situations, for example and without limitation, in which one processor performs some of recited functions and another processor(s) performs other of recited functions, and also situations in which a single processor may perform all recited functions. Additionally, the at least one processor may include a combination of processors performing various of the recited / disclosed functions, e.g., in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0048] Fig. 1A and Fig. 1B illustrate an exemplary scenario 100 of incorrect rendering of an application window due to objects’ movement, in accordance with a related art. As shown in Fig. 1A, the scenario 100 depicts a dynamic environment having plurality of easily movable and non-movable objects. A user of an HMD device may open an application window 102 in the HMD device for work. As shown, the application window 102 is rendered at a location in a three-dimensional (3D) space with respect to a plurality of (3D) feature points in the scene, where the plurality of (3D) feature points are associated with the plurality easily movable and non-movable objects.

[0049] Now referring to Fig. 1B, the scenario 100 is depicted with few of the easily movable objects moved from their initial position. For example, this situation may arise when the user may leave the room for some duration, and in absence of the user, at least one object has been moved in the room while cleaning the room. Similarly, the location of the feature points may also change. Therefore, since the application window 102 renders based on all feature points, the application window 102 is rendered in the wrong place.

[0050] Fig. 2 illustrates an exemplary scenario 200 for a memory blowout in the HMD device, in accordance with a related art. As shown, the environment includes multiple movable objects, such as a rug, cushions, pillows, curtains, and the like which tend to often move around. Now, on subsequent rendering of the application window in the HMD device, in an exemplary situation where a few of the movable objects have moved from their initial position, the SLAM may add new feature points for objects found in a new location to a map. By adding new feature points to the map in addition to existing old feature points, the memory usage may exponentially increase. For example, if the memory usage is approximately 500MB, then after 10 minutes of usage of the HMD device in the illustrated scenario 200 of FIG. 2 the HMD device may experience a memory blowout.

[0051] For example, referring to the pillow, as shown in Fig. 2. When the HMD device views the area for the first time, the HMD device may add a few thousand feature points from the pillow to the map. Now, for example, if a person sits on a sofa, the position of the pillow may get shifted. Therefore, on subsequent use of the HMD device in the same exemplary scenario 200, the HMD device may add a few thousand more feature points from the same pillow to the map doubling the map feature points. Since the HMD device may not have the capability to understand that the pillow has been moved, this type of conventional mapping continues to add redundant points to the map, resulting in the memory blowout.

[0052] In an example, referring to the rug as shown in Fig. 2. When the HMD device views the area for the first time, the HMD device may add a few thousand feature points from the rug to the map. If the rug is of size 6ft*2ft, a number of feature points from the rug may be around 10000 to 15000 feature points. Therefore, the rug may consume about 6 to 10 MBs of memory. Hence, any small movement of the rug may increase the memory usage by 10 MB. Over a period of time, the HMD device may keep on adding redundant points to the memory, resulting in the memory blowout.

[0053] Fig. 3 illustrates an example scenario 300 of power consumption and accuracy with respect to feature points, in accordance with a related art. As shown, to identify a single feature point in a map, it takes N comparisons. Here, “N” refers to the total number of points on the map. Therefore, an increase in the number of map feature points increases the number of comparisons between current scene feature points and the map feature points which further increases power usage. Since the feature points associated with the object’s old positions are also present in the map, there is a significant reduction in the HMD device’s pose estimation accuracy.

[0054] For example, if there are 700 map feature points and there are 200 feature points in the current scene. The total number of comparisons required to identify the 200 current scene feature points in the list of map feature points is 700*200 = 140000. Now, if an object moves to a new location in the scene, thereby adding new feature points to the map, for example, addition of new 200 feature points, the total map feature points may increase to 900. Thus, the comparisons required to identify the 200 current scene feature points in the list of map feature points increase to 900 * 200 = 180000, which is approximately a 35% increase. The aforementioned scenario may slow down the pose estimation and also may increase power consumption. The increased power consumption and generated heat due to additional processing may reduce a central processing unit (CPU) clock which eventually reduces the pose estimation accuracy. Further, this effect is cumulative in nature.

[0055] In an example, a user may use the HMD device inside a public transport. In this scenario, due to multiple movable objects in the dynamic environment, new positions of the same object may keep adding new feature points to the map. Due to this, memory and power consumption of the HMD device may increase, thereby, indirectly reducing the pose estimation accuracy.

[0056] Fig. 4A and Fig. 4B illustrate a pictorial depiction 400 of relocation of application windows in a home / work environment, in accordance with an existing art. Referring to Fig. 4A, while using the HMD device, the user may open multiple application windows and arrange them according to the user’s preference. There is a possibility that the place where the user uses the HMD device has more dynamic or movable objects. Now, referring to Fig. 4B, whenever the user removes and wears the HMD device again, the application window’s position may get changed due to the change in position of the movable objects. Therefore, the user has to re-arrange all the application windows.

[0057] Fig. 5 illustrates a pictorial depiction 500 of relocation of the rendered application window in a parking space, in accordance with a related art. For example, the user may render a navigation map application window in a parking space to show the best traveling route before initiating travel. Since the parking lot may have a plurality of movable objects, such as other parked cars, the application window may not be in that same position daily. Further, the application window may drift away, and the user has to re-position the application window every time.

[0058] Fig. 6 illustrates an exemplary pictorial depiction 600 of usage of simultaneous localization and mapping (SLAM), in accordance with a related art. The SLAM is a computational technique used in robotics to construct maps of unknown environments while simultaneously tracking the robot's position within those environments. For example, suppose a mobile robot 602 wishes to find a charging station. The robot 602 first views a park gate 603 and adds the gate to a robot’s map. Then the robot 602 moves forward and finds a swing 604 to the robot’s left, the robot 602 adds the swing to the robot’s map, takes a right, and moves forward to find a park bench 605 to the robot’s right. The robot 602 then adds the park bench to the robot’s map, takes a left, and moves forward to find a lamp 606 to the robot’s left. The robot 602 adds the lamp to the robot’s map, takes a left, and moves forward to find a slide 607 to the robot’s left. The robot 602 adds the slide to the robot’s map, takes a left, and moves forward to find the same swing 604 to the robot’s right. Since, the robot 602 has created the map and localized itself in the map, the robot 602 knows that the robot 602 has to take a right as it has explored other areas. If the SLAM is not performed, the robot 602 might move randomly and may not be able to search the complete area and also the robot 602 may be highly inefficient.

[0059] Fig. 7 illustrates an exemplary scenario 700 for localization using the SLAM, in accordance with an existing art. In an example, consider the same map as shown in Fig. 6, but convert the map into a grid with 1cm as its height and width. Further, each landmark may be represented by an indication, for example, the highlighted box. Now, as the robot 602 is walking around, if the robot 602 is able to see one or more landmarks, the robot 602 will localize itself in the map that the robot 602 has created. In a case where the robot 602 sees two landmarks 701, 702, the robot measures its distance from the landmarks. Then, by comparing the distance values with the map, the robot 602 may find its own location on the map.

[0060] Fig. 8 illustrates an exemplary scenario 800 for production of a pose using the localization from the SLAM, in accordance with a related art. The pose is the result of the localization. The pose has a total of 6 values, where three values represent a position in the x, y, and z-axis, and the other three values represent orientation in the x, y, and z-axis (Roll, Pitch, and Yaw). In an example, a robot may start the SLAM while sitting on a bench to track its head, so the robot first marks an initial pose as (x = 0, y = 0, z = 0, yaw = 0, pitch = 0, roll = 0). The robot first gets up from the bench, so its head position along the z-axis moves up by 5 cm. So now its pose may be at (0, 0, 5, 0, 0, 0). Then the robot walks forward for 4 cm. So now the robot’s pose may be at (4, 0, 5, 0, 0, 0). The robot may then walk left for 6 cm. So now its pose may be at (4, -6, 5, 0, 0, 0). Then the robot bends its head down by 10 degrees. So now the robot’s pose may be at (4, -6, 5, 0, -10, 0). Then the robot tilts its head right by 5 degrees. So now its pose may be at (4, -6, 5, 0, 10, 5). Then the robot turns its head right by 15 degrees. So now its pose may be at (4, -6, 5, 15, 10, 5).

[0061] Fig. 9 illustrates an exemplary scenario 900 for feature matching, in accordance with a related art. Here, feature descriptors may be used for performing feature matching. In an example, suppose a point is given, and a user is asked to find that point in a given image. It is difficult to tell which point is the point in the image, as the image may include many similar-looking points. Hence, a concept of the feature descriptor is introduced, in which information about the surrounding points may also be stored along with the selected point. The feature descriptors are numerical representations of points and surrounding regions. They capture distinctive information about the local appearance of these points, such as gradients, texture, colour, etc. With reference to FIG. 9, if only the point 901 is provided and an attempt is made to identify the corresponding location in the image, it may be difficult to determine the specific point being referred to, as there are many similar black points present. However, if the point 901 is provided together with surrounding information such as descriptors 902, the intended point can be readily identified. The arrow 903 indicates the point being referred to.

[0062] Therefore, in view of the above-mentioned problems, it is advantageous to provide an improved system and method that can overcome the above-mentioned problems and limitations associated with rendering an application window in a dynamic environment.

[0063] The methods and systems disclosed herein have numerous advantages. The present disclosure allows the removal of feature points from the objects that have been moved. By discarding all the feature points or the landmark points from moved objects, re-localization becomes as accurate as in a still environment. The present disclosure enables the HMD device to delete old feature points from the previous position of a moved object and add new ones for its current position, which cuts down on power usage and memory requirements. Further, according to the present disclosure, re-localization never fails in dynamic environments, which ensures that the previously opened application windows are always rendered in the same location. Therefore, according to the present disclosure, the re-localization accuracy increases in dynamic environments, preventing application windows from appearing in the wrong location. Further, according to the present disclosure, with a reduced map size, memory consumption is also reduced. Furthermore, due to the reduced map size and less frequent re-localization, power consumption is also significantly decreased, preventing the HMD device from overheating, and thereby improving device runtime.

[0064] Fig. 10 illustrates a block diagram of a head-mounted display (HMD) device 1000, in accordance with an embodiment of the present disclosure. In an embodiment, the HMD device 1000 (interchangeably referred to as a device 1000) may be configured to support an augmented reality / mixed reality (AR / MR) space, where a virtual object is overlaid in a physical environment, without departing from the scope of the present disclosure. In an embodiment, a system 1002 may be communicatively coupled with the device 1000 and may be located outside the device 1000 in a standalone manner. In an embodiment, the system 1002 may be deployed within the HMD device 1000, without departing from the scope of the present disclosure. In an embodiment, the system 1002 may be configured to enhance re-localization of the HMD device 1000, without departing from the scope of the present disclosure.

[0065] In an embodiment, the system 1002 may include, but is not limited to, at least one processing unit 1004 (referred to here as a processor 1004), a memory 1006, and a plurality of modules 1008 among other examples which are explained in detail in subsequent paragraphs. Further, the system 1002 may include an Input / Output (I / O) interface 1040 and a transceiver 1030.

[0066] In an exemplary embodiment, the processor 1004 may be communicatively coupled with the memory 1006, without departing from the scope of the present disclosure. The processor 1004 may be operatively coupled to each of the I / O interface 1040, the plurality of modules 1008, the transceiver 1030, and the memory 1006. In one embodiment, the processor 1004 may include a graphical processing unit (GPU) and / or an artificial intelligence engine (AIE). In one embodiment, the processor 1004 may include at least one data processor for executing processes in a virtual storage area network. The processor 1004 may include specialized processing units such as integrated system (bus) controllers, memory management control units, floating point units, graphics processing units, digital signal processing units, etc. In one embodiment, the processor 1004 may include a central processing unit (CPU), a graphics processing unit (GPU), or both. The processor 1004 may be one or more general processors, digital signal processors, application-specific integrated circuits, field-programmable gate arrays, servers, networks, digital circuits, analog circuits, combinations thereof, or other now-known or later developed devices for analyzing and processing data. The processor 1004 may execute a software program, such as code generated manually (i.e., programmed) to perform the desired operation.

[0067] The processor 1004 may be disposed of in communication with one or more input / output (I / O) devices via the I / O interface 1040. In some embodiments, the processor 1004 may communicate with the device 1000 using the I / O interface 1040. In some embodiments, the I / O interface 1040 may be implemented within the device 1000. The I / O interface 1040 may employ communication code-division multiple access (CDMA), high-speed packet access (HSPA+), global system for mobile communications (GSM), long-term evolution (LTE), WiMax, or the like. In an embodiment, the I / O interface 1040 may enable input and output to and from the system 1002 using suitable devices such as, but not limited to, display, keyboard, mouse, touch screen, microphone, speaker, and so forth.

[0068] Using the I / O interface 1040, the system 1002 may communicate with one or more I / O devices, specifically, the device 1000, to which the system 1002 provides enhanced re-localization. For example, the input device may be an antenna, microphone, touch screen, touchpad, storage device, transceiver, video device / source, etc. The output devices may be a video display (e.g., cathode ray tube (CRT), liquid crystal display (LCD), light-emitting diode (LED), plasma, Plasma Display Panel (PDP), Organic light-emitting diode display (OLED) or the like), audio speaker, etc.

[0069] The processor 1004 may be disposed of in communication with a communication network via a network interface. In an embodiment, the network interface may be the I / O interface 1040. The network interface may connect to the communication network to enable the connection of the system 1002 with the device 1000. The network interface may employ connection protocols including, without limitation, direct connect, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), transmission control protocol / internet protocol (TCP / IP), token ring, IEEE 802.11a / b / g / n / x, etc. The communication network may include, without limitation, a direct interconnection, local area network (LAN), wide area network (WAN), wireless network (e.g., using Wireless Application Protocol), the Internet, etc. Using the network interface and the communication network, the system 1002 may communicate with other devices. The network interface may employ connection protocols including, but not limited to, direct connect, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), transmission control protocol / internet protocol (TCP / IP), token ring, IEEE 802.11a / b / g / n / x, etc.

[0070] The transceiver 1030 may be configured to receive and / or transmit signals to and from the device 1000. In one embodiment, the database may be configured to store the information as required by the plurality of modules 1008 and the processor 1004 to perform one or more functions for rendering the reflection for the virtual object, on the device 1000.

[0071] In some embodiments, the memory 1006 may be communicatively coupled to the processor 1004. The memory 1006 may be configured to store data, and instructions executable by the processor 1004 to perform the one or more methods disclosed herein throughout the present disclosure. In one embodiment, the memory 1006 may be provided within the device 1000. In an embodiment, the memory 1006 may be provided within the system 1002 being remote from the device 1000. In an embodiment, the memory 1006 may communicate with the processor 1004 via a bus within the system 1002. In an embodiment, the memory 1006 may be located remote from the processor 1004 and may be in communication with the processor 1004 via a network. The memory 1006 may include, but is not limited to, a non-transitory computer-readable storage media, such as various types of volatile and non-volatile storage media including, but not limited to, random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, magnetic tape or disk, optical media and the like.

[0072] In one example, the memory 1006 may include a cache or random-access memory for the processor 1004. In alternative examples, the memory 1006 is separate from the processor 1004, such as a cache memory of a processor, the system memory, or other memory. The memory 1006 may be an external storage device or database for storing data. The memory 1006 may be operable to store instructions executable by the processor 1004. The functions, acts, or tasks illustrated in the figures or described may be performed by the programmed processor 1004 for executing the instructions stored in the memory 1006. The functions, acts, or tasks are independent of the particular type of instruction set, storage media, processor, or processing strategy and may be performed by software, hardware, integrated circuits, firmware, micro-code, and the like, operating alone or in combination. Likewise, processing strategies may include multiprocessing, multitasking, parallel processing, and the like.

[0073] In some embodiments, the plurality of modules 1008 may be included within the memory 1006. The memory 1006 may further include a database to store data. The plurality of modules 1008 may include a set of instructions that may be executed to cause the system 1002, in particular, the processor 1004 of the system 1002, to perform any one or more of the methods / processes disclosed herein. The plurality of modules 1008 may be configured to perform the steps of the present disclosure using the data stored in the database.

[0074] In an embodiment, each of the plurality of modules 1008 may be a hardware unit that may be outside the memory 1006. Further, the memory 1006 may include an operating system for performing one or more tasks of the system 1002, as performed by a generic operating system.

[0075] In one example, the plurality of modules 1008 may include a feature extractor module 1010, an evaluation module 1012, a feature point filter module 1014, a localization module 1016, an object information extractor module 1018, and a mapper module 1020. Each of the modules 1010-1020 may be in communication with each other. Further, each of the modules 1010-1020 may be in communication with the processor 1004.

[0076] Further, the present disclosure contemplates a computer-readable medium that includes instructions or receives and executes instructions responsive to a propagated signal. Further, the instructions may be transmitted or received over the network via a communication port or interface or using a bus (not shown). The communication port or interface may be a part of the processor 1004 or may be a separate component. The communication port may be created in software or may be a physical connection in hardware.

[0077] The communication port may be configured to connect with the network, external media, the display, or any other components in the system, or combinations thereof. The connection with the network may be a physical connection, such as a wired Ethernet connection, or may be established wirelessly. Likewise, the additional connections with other components of the system 1002 may be physical or may be established wirelessly. The network may alternatively be directly connected to a bus. For the sake of brevity, the architecture, and standard operations of the memory 1006, the processor 1004, the transceiver 1030, and the I / O interface 1040 are not discussed in detail.

[0078] In one embodiment, the plurality of modules 1008 may be implemented by processors such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, or the like, and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like.

[0079] In one embodiment, the plurality of modules 1008 may be configured to perform an enhanced re-localization of a head-mounted display (HMD) 1000 device. The feature extractor module 1010 may be configured to obtain (e.g. identify) one or more current feature points of one or more objects present (e.g. identified) in their current position in a real-world environment. In an embodiment, the one or more objects include fixed objects and movable objects. The evaluation module 1012 may be configured to derive one or more of current landmark points associated with each of the objects from the feature points corresponding to one or more pre-stored landmark points associated with each of the one or more objects of the real-world environment. The evaluation module 1012 may further be configured to compare the one or more current landmark points with one or more pre-stored landmark points associated with each of the one or more objects of the real-world environment, to determine whether any one or more objects have moved to the current position from a pre-stored position within the real-world environment. The feature point filter module 1014 may be configured to filter the one or more current landmark points by discarding the feature points associated with the one or more objects that have moved to the current position from the pre-stored position. The localization module 1016 may be configured to perform re-localization of the HMD device 1000 using the filtered one or more current landmark points.

[0080] In an embodiment, the localization module 1016 may be configured to perform pose estimation of the HMD device 1000 using the filtered one or more current landmark points.

[0081] In an embodiment, the object information extractor module 1018 may be configured to categorize each of the one or more objects as one of a fixed object and a movable object based on the size of the corresponding object, by determining whether an estimated weight and computed volume of the corresponding object is greater than a predetermined threshold value.

[0082] In an embodiment, the object information extractor module 1018 may further be configured to categorize each of the one or more objects as one of a fixed object and a movable object based on a placement of the corresponding object, by determining whether the corresponding identified object is placed on at least one of a horizontal surface or a vertical surface.

[0083] In an embodiment, the one or more pre-stored landmark points and prestored feature points associated with each of the one or more objects of the real-world environment are stored in a global map of the real-world environment. The global map may include information related to movability of the one or more objects. Here, the movability corresponds to the one or more objects being fixed objects and movable objects.

[0084] In an embodiment, the one or more pre-stored landmark points associated with the one or more fixed objects and one or more pre-stored landmark points associated with the one or more movable objects are utilized to establish the pre-stored position of each of the one or more fixed and movable objects of the real-world environment.

[0085] In an embodiment, each pre-stored landmark point of a single identified object is at least a predetermined distance apart from another pre-stored landmark point of said single identified object.

[0086] In an embodiment, to compare the one or more current landmark point with one or more pre-stored landmark point of the real-world environment, the evaluation module 1012 is configured to compute a distance between the one or more current landmark points associated with the one or more fixed objects and one or more current landmark points associated with the one or more movable objects. The evaluation module 1012 is further configured to compare the computed distance with a distance between the one or more pre-stored landmark points associated with the one or more fixed objects and the one or more pre-stored landmark points associated with the one or more movable objects.

[0087] In an embodiment, the mapper module 1020 may be configured to receive the current identified feature points from the localization module 1016. Further, mapper module 1020 may be configured to construct the global map of the real-world environment. In an embodiment, the mapper module 1020 may be configured to select a few feature points from each of the one or more objects and mark them as landmark points.

[0088] In an embodiment, the plurality of modules 1008 may be configured to perform an enhanced re-localization of a head-mounted display (HMD) 1000 device. The feature extractor module 1010 may be configured to determine one or more current feature points associated with each of one or more identified objects from a real-world environment. The one or more identified objects include fixed objects and movable objects. The evaluation module 1012 may be configured to retrieve one or more pre-stored feature points, a pre-stored position, and movability associated with each identified object based on a historic localization and mapping of the identified objects by referring to a global map of the real-world environment. The evaluation module 1012 may further be configured to determine that at least one movable object among the one or more identified objects has moved from the associated pre-stored position by correlating the one or more current and pre-stored feature points associated with the movable objects with the one or more current and pre-stored feature points associated with the fixed objects. The feature point filter module 1014 may be configured to filter the one or more current feature points by discarding the feature points associated with the determined movable object that has moved. The localization module 1016 may be configured to perform re-localization of the HMD device 1000 by using the filtered one or more current feature points.

[0089] In an embodiment, the localization module 1016 is configured to perform pose estimation of the HMD device 1000 by using the filtered one or more current feature points.

[0090] In an embodiment, prior to the initiation of the re-localization, the system 1002 performs the historic localization and mapping of the one or more identified objects to the global map of the real-world environment. In an embodiment, the plurality of modules 1008 may be configured to identify the one or more objects in the real-world environment captured by the HMD device 1000. In an embodiment, the plurality of modules 1008 may further be configured to extract one or more feature points associated with each of the one or more identified objects. In an embodiment, the plurality of modules 1008 may further be configured to receive information associated with each of the one or more identified objects. In an embodiment, the plurality of modules 1008 may further be configured to categorize the one or more identified objects as fixed objects and movable objects based on the received information. Further, the plurality of modules 1008 may be configured to perform the localization of the HMD device (1000) based on the extracted one or more feature points, and map the extracted feature points and movability associated with the one or more identified objects to the global map.

[0091] In an embodiment, the object information extractor module 1018 may be configured to categorize each of the one or more objects as one of a fixed object and a movable object based on size of the corresponding object, by determining whether an estimated weight and computed volume of the corresponding identified object is greater than a predetermined threshold value. In embodiment, the object information extractor module 1018 may be configured to categorize each of the one or more objects as one of a fixed object and a movable object based on a placement of the corresponding object, by determination of whether the corresponding identified object is placed on at least one of a horizontal surface or a vertical surface.

[0092] In embodiment, one or more landmark points may be derived from each of the identified objects from the one or more current and pre-stored feature points. The one or more pre-stored landmark points associated with the at least one fixed object and the pre-stored landmark points associated with the at least one movable object are utilized to establish the pre-stored position of each of the one or more fixed and movable objects on the global map of the real-world environment.

[0093] In an embodiment, each landmark of a single identified object is at least a predetermined distance apart from another landmark of said single identified object.

[0094] In an embodiment, to correlate the determined one or more current feature points and the one or more pre-stored feature points, the evaluation module 1012 may be configured to compute a distance between the determined one or more current feature points associated with the one or more fixed objects and the determined one or more current feature points of the one or more movable objects. Further, the evaluation module 1012 may be configured to compare the computed distance with a distance between the one or more pre-stored feature points associated with the one or more fixed objects and the one or more pre-stored feature points associated with the one or more movable objects,

[0095] Fig. 11 illustrates a functional block diagram 1100 of localization and mapping of the HMD device 1000, in accordance with an embodiment of the present disclosure.

[0096] The HMD device 1000 may include a camera sensor configured to capture images of a current real-world environment.

[0097] At block 1102, the feature extractor module 1010 may receive the captured images. The feature extractor module 1010 may extract one or more distinctive feature points associated with one or more identified objects from the received images. In an example, the feature point extraction may be performed using a feature detection algorithm, such as the Harris corner detector, Scale Invariant Feature Transform such as Speeded Up Robust Feature (SURF), or Oriented FAST and Rotated BRIEF (ORB). Further, again at block 1102, the object information extractor module 1018 may retrieve information associated with each of the one or more identified objects from which the one or more feature points have been extracted. In an example, common descriptor algorithms may include SIFT, ORB, and FREAK. In a non-limiting example, the feature extractor module 1010 may match the feature points between consecutive images or between distant images using a matching algorithm such as brute force matching, FLANN, or ANN. The matches are used to estimate the camera motion and to build a global map of the real-world environment and the object information extractor module 1018 may describe them using a set of descriptors. Further, in an embodiment, the one or more objects may be categorized based on movability as fixed objects and movable objects based on the information retrieved by the object information extractor module 1018.

[0098] At block 1104, the localization module 1016 may be configured to receive the extracted one or more feature points associated with one or more identified objects from the object information extractor module 1018. The localization module 1016, in case of availability of a map of the current real-world environment corresponding to the capture images having the one or more identified objects, may further be configured to retrieve the map from the global map. The localization module 1016 may estimate the HMD device’s 1000 position and orientation relative to the extracted feature points.

[0099] At block 1106, a mapper module 1020 may receive the extracted feature points and may be configured to incorporate the extracted one or more feature points and the retrieved information associated with each of the one or more identified objects on the map of the real-world environment. The mapper module 1020 may select a few feature points from each of the movable objects and mark them as landmark points. Similarly, the mapper module 1020 may select a few feature points from each of the fixed objects and mark them as landmark points.

[0100] At block 1108, the map comprising the one or more feature points and landmark points associated with the one or more fixed and movable objects may be stored to the global map. In an example, the global map can be represented in various forms, such as a grid-based occupancy map, a point cloud, or a geometric mesh. The global map may grow and evolve as new feature points are observed and incorporated.

[0101] Fig. 12 illustrates an example implementation of the functional block diagram 1100 of localization and mapping of the HMD device 1000, in accordance with an embodiment of the present disclosure.

[0102] The camera sensor of the HMD device 1000 may capture one or more images 1202 of the current real-world environment.

[0103] At block 1102, the feature extractor module 1010 may receive the captured images 1202. The feature extractor module 1010 may extract one or more distinctive feature points, as marked in the image 1204, associated with one or more identified objects from the received images and make a list of feature points 1206. Further, again at block 1102, the object information extractor module 1018 may retrieve information 1208 associated with each of the one or more identified objects from which the feature points have been extracted. In an embodiment, a machine learning model may be utilized to retrieve information 1208 associated with each of the one or more identified objects. In an embodiment, information, as shown in table 1208 in Fig. 12, may include at least one of the size, placement, and shape of the one or more identified objects. In an example embodiment, the information 1208 includes size of the one or more identified objects and includes an estimated weight and computed volume of the one or more identified objects, along with their associated feature points.

[0104] Further, an object classifier 1210, a sub-unit of the object information extractor module 1018, may categorize the one or more identified objects according to their movability as fixed objects and movable objects based on their size, placement, and shape. In an embodiment, as shown in table 1212 in Fig. 12, the one or more objects may be categorized as one of a fixed object and a movable object based on the size of the corresponding object. The categorization based on size includes determining whether an estimated weight and computed volume of the corresponding identified object is greater than a predetermined threshold value. In an example embodiment, the threshold for weight is 30 Kg and the threshold for volume is 0.75 m3.

[0105] At block 1104, the localization module 1016 may receive the extracted one or more feature points associated with one or more identified objects from the list of feature points 1206 form the feature extractor module 1010, and may also receive information 1208 associated with each of the one or more identified objects and associated movability from the object information extractor module 1018. The localization module 1016, in case of availability of the map of the current real-world environment corresponding to the capture images having the one or more identified objects, may retrieve the map from the global map. The localization module 1016 may estimate the HMD device’s 1000 position and orientation relative to the extracted one or more feature points associated with one or more identified objects from the list of feature points 1206.

[0106] At block 1106, the mapper module 1020 may receive the extracted one or more feature points and may be configured to incorporate the extracted one or more feature points from the list of feature points 1206 and the retrieved information 1208 along with information associated with the movability of the one or more identified objects form the table 1212 into the map of the real-world environment. Further, as shown in Fig. 12 in table 1214, the mapper module 1020 may select a few feature points from each of the movable objects and mark them as landmark points. Similarly, the mapper module 1020 may select a few feature points, from each of the fixed objects and mark them as landmark points. For example, for ease of understanding, the landmark points shown in table 1214 are indicated by enclosing the feature point in a box.

[0107] At block 1108, the feature points and landmark points associated with the one or more fixed and movable objects may be stored in the global map. As shown in Fig. 12, in table 1216, the global map may store information about one or more objects, their associated feature points, and along with the associated movability of the one or more objects. The information stored in the global map, such as feature points, landmark points, position of objects based on feature points or landmark points, and movability of the objects, may be utilized by the localization module 1016 for estimating the HMD device’s 1000 position and orientation.

[0108] Fig. 13 illustrates a functional block diagram 1300 for feature and object information extraction for localization of the HMD device 1000, in accordance with an embodiment of the present disclosure. Hereafter the feature and object information extraction step will be explained in detail.

[0109] At block 1102, the feature extractor module 1010 may receive the captured image 1202 as an input. The feature extractor module 1010 may extract one or more distinctive feature points 1204 associated with one or more identified objects from the received images. In an embodiment, the object information extractor module 1018, using a machine learning model, may retrieve information 1208 associated with each of the one or more identified objects from which the feature points have been extracted. The information may include at least one of the size, placement, and shape of the one or more identified objects. As shown, in an example embodiment, the information 1208 includes size of the one or more identified objects and includes the estimate weight and compute volume of the one or more identified objects, along with their associated feature points.

[0110] In an embodiment, an AI model may be utilized for determining the estimate weight and computed volume associated with the one or more identified objects. In an embodiment, a look-up table may be utilized for estimating weight and computing volume associated with the one or more objects, wherein the look-up table may include one or more common objects generally available in a home environment along with predefined approx. weight and volume. Therefore, once an object has been identified, its weight and volume can be found using the look-up table.

[0111] Based on the information 1208 the objects may be categorized into two or more classes based on their movability. In an embodiment, the one or more objects are categorized as fixed objects and movable objects.

[0112] Based on the weight and volume, one or more objects may be categorized as the fixed objects or the movable objects. The categorization based on size includes determining whether the estimated weight and computed volume of the corresponding identified object is greater than a predetermined threshold value.

[0113] In an example embodiment, the threshold for weight is 30 Kg and the threshold for volume is 0.75 m3.

[0114] Therefore, any object with a volume less than 0.75 m³ and a weight of 30 kg is categorized as the movable object. For example, if a chair is identified, its estimated volume may be 0.5 m³ and its estimated weight may be 10 kg. Since volume < 0.75 m³ and weight < 30 kg, the chair may be classified as the movable object. Similarly, for example, if a sofa is identified, its approximate volume may be 2.5 m³ and its approximate weight may be 75 kg. Since volume > 0.75 m³ and weight > 30 kg, the sofa may be classified as the fixed object. Table 1 below shows an example of a look-up-table in accordance with the object classification of the aforementioned example.

[0115] S.NOObjectWeight (Kg)Volume (m³)1.Chair100.52.Sofa752.5

[0116] Furthermore, the output of the module is a list 1212 of feature points grouped based on the one or more identified objects. Each group is accompanied by a classification indicating the movability of the object.

[0117] Fig. 14 illustrates a functional block diagram 1400 of re-localization and mapping of the HMD device 1000, in accordance with an embodiment of the present disclosure. The rendering of an application window in a dynamic environment requires re-localization after changes in the environment.

[0118] At block 1402, the feature extractor module 1010 may receive current captured images when the user re-wears the HMD device 1000 in a real-world environment. The feature extractor module 1010 may extract one or more distinctive current feature points associated with one or more identified objects from the received images.

[0119] At block 1404, the evaluation module 1012 may receive the extracted one or more current feature points associated with one or more identified objects. Further, the evaluation module 1012 may retrieve one or more pre-stored feature points, a pre-stored position, and movability associated with each identified object based on a historic localization and mapping of the identified objects by referring to a global map of the real-world environment. The historic localization may refer to previously performed localization, for example, localization performed in association with the description of FIG. 11 to FIG. 13. In an embodiment, if one or more pre-stored landmark points are available from the historic localization in the global map, the evaluation module 1012 may be configured to obtain (e.g. derive) one or more current landmark points associated with each of the one or more objects from the current feature points corresponding to one or more pre-stored landmark points associated with each of the one or more objects of the real-world environment.

[0120] Further, at block 1404, the evaluation module 1012 may evaluate if any fixed object has moved from its pre-stored position or not. Here, the pre-stored position may indicate the position of the one or more fixed objects stored in the global map based on the historic localization. In one embodiment, the evaluation module 1012 compares the one or more current feature points with the one or more pre-stored feature points associated with each of the one or more fixed objects of the real-world environment, to determine whether any of the one or more fixed objects have moved to the current position from the pre-stored position within the real-world environment. In an embodiment, the evaluation module 1012 compares the one or more current landmark points with one or more pre-stored landmark points associated with each of the one or more fixed objects of the real-world environment, to determine whether any of the one or more fixed objects have moved to the current position from the pre-stored position within the real-world environment. The feature point filter module 1014, based on the determination by the evaluation module 1012 that one or more fixed objects have moved from their pre-stored position, may filter out the one or more current feature points and / or one or more current landmark points associated with the one or more moved fixed objects from consideration for performing re-localization.

[0121] At block 1406, the evaluation module 1012 may evaluate if any movable object has moved from its pre-stored position or not. Here, the pre-stored position may indicate the position of the one or more movable objects stored in the global map based on the historic localization. In one embodiment, the evaluation module 1012 compares the one or more current feature points with the one or more pre-stored feature points associated with each of the one or more movable objects of the real-world environment, to determine whether any of the one or more movable objects have moved to the current position from the pre-stored position within the real-world environment. In an embodiment, the evaluation module 1012 compares the one or more current landmark points with one or more pre-stored landmark points associated with each of the one or more movable objects of the real-world environment, to determine whether any of the one or more movable objects have moved to the current position from the pre-stored position within the real-world environment. The feature point filter module 1014, based on the determination by the evaluation module 1012 that one or more movable objects have moved from their pre-stored position, may filter out the one or more current feature points and / or one or more current landmark points associated with the one or more moved movable objects from consideration for performing re-localization.

[0122] At block 1408, the localization module 1016 may perform pose estimation of the HMD device 1000 using the filtered one or more current feature points or filtered one or more current landmark points associated with the one or more objects that have not moved. Further, the localization module 1016 may receive the extracted one or more current feature points from the feature extractor module 1010. The localization module 1016 may further determine unmatched one or more current feature points associated with identified one or more objects by comparing the extracted one or more current feature points with at least one of the filtered one or more current feature points received from the feature point filter module 1014. The unmatched one or more current feature points may indicate either that the at least one movable object has been moved in the real-world environment or presence of new one or more identified objects in the real-world environment.

[0123] At block 1410, the object information extractor module 1018 may retrieve information associated with the determined one or more unmatched feature points. The retrieved information may include at least one of the size, placement, and shape of the one or more identified objects, along with movability associated with the one or more objects associated with the unmatched one or more current feature points.

[0124] At block 1412, the mapper module 1020 may receive the determined one or more unmatched current feature points corresponding to the one or more objects that have moved from their pre-stored position from the localization module 1016 and the information associated with unmatched one or more current feature points from the object information extractor module 1018. The mapper module 1020 may select a few current feature points from the unmatched one or more current feature points and mark them as landmark points.

[0125] At block 1414, the one or more current feature points and / or one or more current landmark points associated with at least one of the one or more moved objects and / or new one or more identified objects may be stored in the global map.

[0126] Fig. 15 illustrates an example implementation of the functional block diagram 1500 of re-localization and mapping of the HMD device 1000, in accordance with an embodiment of the present disclosure. An example scenario depicted here may relate to re-localization after an object, for example a stool, has been moved from its pre-stored position. Therefore, when the user re-wears the HMD device 1000, the HMD device 1000 may perform re-localization. Further, in an example, during localization, the stool may be object 3, and the extracted feature points associated with the stool may be “F5, F6, F8, F14, F16, F17, F20, F21, F23”.

[0127] The camera sensor of the HMD device 1000 may again capture one or more images 1502 of the current real-world environment.

[0128] At block 1402, the feature extractor module 1010 may receive current captured images. The feature extractor module 1010 may extract one or more distinctive current feature points, as marked in the image 1504, associated with one or more identified objects from the received images and make a list of feature points 1506.

[0129] At block 1404, the evaluation module 1012 may receive the extracted one or more current feature points associated with the one or more identified objects from the feature extractor module 1010. Further, the evaluation module 1012 may retrieve information 1508 stored in the global map. The information 1508 may include at least one of the one or more pre-stored feature points, a pre-stored position, and movability associated with each identified object based on a historic localization and mapping of the identified objects. In an embodiment, if one or more pre-stored landmark points are available from the historic localization in the global map, the evaluation module 1012 may further be configured to derive one or more current landmark points associated with each of the one or more objects from the current feature points corresponding to one or more pre-stored landmark points associated with each of the one or more objects of the real-world environment.

[0130] Further, at block 1404, the evaluation module 1012 may first determine whether any one or more fixed objects have moved to the current position from their corresponding pre-stored position within the real-world environment.

[0131] Accordingly, the evaluation module 1012 may compute a distance between the one or more current feature points or current landmark points associated with the one or more fixed objects and one or more pre-stored feature points or pre-stored landmark points associated with the one or more fixed objects.

[0132] For example, consider pre-stored feature points “F1, F2, F7, F12, F13, F18” associated with a fixed object 1 and pre-stored feature points “F24, F26, F27, F28, F30” associated with a fixed object 4 from the extracted information 1508. Next, the evaluation module 1012 may compute a distance between the pre-stored feature point “F18” and pre-stored feature point “F24”. Further, the evaluation module 1012 may also compute a distance between corresponding current feature point, for example, the current extracted feature point “F18” associated with object 1 and the current extracted feature point “F24” associated with object 4 from the list of feature points 1506. Further, evaluation module 1012 may compare the computed distance, and as the distance are same, it may be determined that the fixed object 1 and fixed object 4 has not moved in the current real-world environment from their pre-stored position in the global map. Accordingly, the evaluation module 1012 may also determine whether one or more fixed objects have moved to the current position for rest of the identified one or fixed objects.

[0133] Further, in case, it is determined by the evaluation module 1012 that a fixed object has moved from its pre-stored position, the filter module 1014 may filter out the current feature point and / or current landmark points associated with the moved fixed object, from any further evaluation.

[0134] At block 1406, the evaluation module 1012 may determine whether any one or more movable objects have moved to the current position from their corresponding pre-stored position within the real-world environment.

[0135] Accordingly, the evaluation module 1012 may compute a distance between the one or more current feature points or one or more current landmark points associated with the one or more fixed objects and one or more current feature points or one or more current landmark points associated with the one or more movable objects. In an embodiment, the one or more fixed objects considered for the determination at block 1406 may be based on the determination at block 1404 that the one or more fixed objects have not moved in the current real-world environment from their pre-stored position in the global map. Further, the evaluation module 1012 may compare the computed distance with a distance between the one or more pre-stored feature points or pre-stored landmark points associated with the one or more fixed objects and the one or more pre-stored feature points or pre-stored landmark points associated with the one or more movable objects.

[0136] For example, consider pre-stored feature points “F1, F2, F7, F12, F13, F18” associated with fixed object 1 and pre-stored feature points “F3, F4, F9, F11, F15, F19, F22, F25, F29” associated with a movable object 2 from the extracted information 1508. Next, the evaluation module 1012 may compute a distance between the pre-stored feature point “F18” and pre-stored feature point “F29”. Further, the evaluation module 1012 may compute a distance between the current extracted feature point “F18” associated with object 1 and the current extracted feature point “F29” associated with object 2 from the list of feature points 1506. Further, evaluation module 1012 may compare the computed distance, and as the distance are same, it may be determined that the movable object has not moved.

[0137] In an example, consider pre-stored feature points “F1, F2, F7, F12, F13, F18” associated with fixed object 1, and pre-stored feature points “F5, F6, F8, F14, F16, F17, F20, F21, F23” associated with the object 3, which in the present example is the moved object, i.e., the stool. Next, the evaluation module 1012 may compute a distance between the pre-stored feature point “F18” and pre-stored feature point “F23”. Further, the evaluation module 1012 may compute a distance between the current extracted feature point “F18” and the current extracted feature point “F23”. However, as the stool has moved, the computed distance may not match. Accordingly, the evaluation module 1012 may determine that a movable object has been moved from its pre-stored position.

[0138] Further, the feature point filter module 1014, based on the determination by the evaluation module 1012 that the movable object 3 associated feature points “F5, F6, F8, F14, F16, F17, F20, F21, F23” has moved from its pre-stored position, may filter out the current feature points and / or current landmark points associated with the moved movable object 3 from consideration for performing re-localization, as can be seen in filtered information 1512.

[0139] At block 1408, the localization module 1016 may perform pose estimation of the HMD device 1000 based on the filtered one or more current feature points or filtered one or more current landmark points associated with the objects that have not moved. Further, the localization module 1016 may receive the extracted feature points from the feature extractor module 1010, for example, new feature points “F31, F32, F33, F8, F6, F17, F34, F35”. The unmatched feature points may indicate either that the at least one movable object has been moved in the real-world environment, or the presence of new one or more identified objects in the real-world environment. In an example, the extracted feature points “F31, F32, F33, F8, F6, F17, F34, F35” in the list of feature points 1506 may correspond to new identified object.

[0140] At block 1410, the object information extractor module 1018 may retrieve information associated with the determined one or more unmatched feature points and provide the retrieve information to the mapper module 1020.

[0141] At block 1412, the mapper module 1020 may receive the determined one or more unmatched current feature points corresponding to the one or more objects from the localization module 1016 and the information associated with unmatched one or more current feature points from the object information extractor module 1018. For example, extracted feature points “F31, F32, F33, F8, F6, F17, F34, F35” in the list of feature points 1506, corresponding to the new identified object may be classified as movable on the basis of the retrieve information from the object information extractor module 1018. Further, the mapper module may further receive the information associated with the one or more unmatched feature points and / or the unmatched one or more landmark points associated with the one or more moved objects to be updated in the global map.

[0142] At block 1414, the updated feature points associated with the moved object, i.e., the stool, and one or more feature points associated with the new identified one or more objects may be stored in the global map.

[0143] Figs. 16A and 16B illustrate a pictorial depiction 1600 of rendering of an application window, in accordance with an embodiment of the present disclosure. Referring to Fig. 16A, a user may be sitting in a dynamic environment wearing the HMD device 1000. The user may open an application window 1602 in the HMD device 1000 and may be working on it. The application window 1602 may be rendered at a location in the 3D space with respect to a plurality of feature points associated with one or more objects in the scene. Accordingly, the plurality of feature points or few landmark points selected from the plurality of feature points may be stored in a global map for future rendering of the application window 1602. Now, referring to Fig. 16B, after some time, the user leaves the room for some work, in the meantime, the one or more objects in the room may be moved for cleaning. Further, when the user re-enters the room and wears the HMD device 1000, the location of the feature points may also be different since the objects have been moved. The HMD device 1000, in accordance with the present disclosure, may identify the one or more objects that have been moved and may filter out the one or more current feature points 1603, 1604, 1605 and / or current landmark points associated with the moved objects for re-localization consideration. Accordingly, for rendering the application window 1602, only the current plurality of feature points or plurality of landmark points corresponding to the filtered plurality of current feature points or filtered plurality of current landmark points may be utilized. Hence, the application window 1602 may be rendered in the original set location without any tracking failure.

[0144] Fig. 17 illustrates a pictorial depiction 1700 of landmark points comparison, in accordance with an embodiment of the present disclosure. For example, consider two objects in a room, Object 1 may be a fixed object, and Object 2 may be a movable object. Generally, when a user uses an HMD device for the first time in the room, the HMD device may add a thousand points from the Object 1 and a thousand points from the Object 2 to the map. For example, there are 200 points in the current scene. Now, in case, and after some time, the Object 2 is moved, then when the user re-visits the room, the HMD device may not recognize that the 1000 points from the Object 2 have been moved. Here, the HMD device may perform 2000 * 200 = 400000 comparisons every time. Further, if a camera sensor of the HMD has a frame rate of 30 fps, then the HMD device would perform 30 * 2000 * 200 = 12000000 comparisons every second and would have added an extra 1000 points to the map.

[0145] Now, in accordance with the present disclosure, the mapper module 1020 may select 10 feature points from the Object 1 and mark them as landmark points. Similarly, the mapper module 1020 may select 10 feature points from each of the Object 2 and mark them as landmark points. Now, in case after some time, the Object 2 is moved, then when the user re-visits the room, the HMD device 1000 only compares the distance between landmark points (e.g., landmark points 1701, 1702) in object 1 and object 2 in the map, i.e., a total of 20 comparisons, to determine that the object 2 has been moved. Moreover, instead of comparing all the feature points in the map that belong to object 2, the HMD device 1000 may remove the thousand feature points in the map that belong to the Object 2, thereby decluttering the map. Further, in case there are 200 points in the current scene, the HMD may perform 1000*200 = 200000 comparisons every time. If the camera has a frame rate of 30 fps, then there would be 30 * 1000 * 200 = 6000000 comparisons performed every second. Along with the initial 20 comparisons, a total of 6000020 comparisons may be performed. Therefore, a total gain may be 12000000 ? 6000020 = 5999980 comparisons. Hence, by adding 20 comparisons, the present disclosure is able to reduce 60 lakh comparisons every second. This advantageously reduces the overloading of the memory 1006 and processor 1004 and further increases the accuracy of the HMD device 1000.

[0146] Figs. 18A and 18B illustrate a pictorial depiction 1800 of object identification, in accordance with an embodiment of the present disclosure. As shown in Fig. 18A, during localization, the feature extractor module 1010 may extract one or more feature points from one or more identified objects present in the real-world environment. The object information extractor module 1018 may retrieve information associated with each of the one or more identified objects from which the feature points have been extracted. All the feature points belonging to the same object may be grouped together.

[0147] Further, the movability of the one or more objects may be classified as one of a movable object or fixed object. For the purpose of explanation, Figs. 18A and 18B depict feature points for categorized one or more fixed objects with black circles, and feature points for categorized one or more movable objects with white circles. The extracted one or more feature points along with other determined information such as the position of the objects, and movability associated with each identified object may be stored in a global map of the current real-world environment, as shown in Fig, 18A.

[0148] Now, referring to Fig. 18B, after some time, the user leaves the room for some work, in the meantime, the one or more objects in the room may be moved for cleaning. Further, when the user re-enters the room and wears the HMD device 1000, the location of the one or more feature points may also be different since the one or more objects have been moved. At this time, the HMD device 1000 may perform re-localization. As shown in Fig. 18B, the feature extractor module 1010 may again extract one or more current feature points (e.g., current feature points 1803, 1804) from one or more objects present in the real-world environment. The evaluation module 1012 may compute a distance between the one or more current feature points associated with the one or more fixed objects (e.g., current feature point 1803) and one or more current feature points associated with the one or more movable objects (e.g., current feature point 1804). The evaluation module 1012 may further compare the computed distance with a distance between the one or more pre-stored feature points associated with the one or more fixed objects (e.g., current feature point 1802) and the one or more pre-stored feature points associated with the one or more movable objects (e.g., current feature point 1801) pre-stored in the global map from historic localization. If it is determined that the distance has changed, then all the current feature points belonging to the one or more movable objects that have been moved may be filtered out and removed from consideration, and also removed the global map. Accordingly, only filtered one or more current feature points associated with unmoved one or more objects may be considered for pose estimation of the HMD device 1000. Therefore, by discarding all feature points from the one or more objects that have been moved, the present disclosure ensures that the re-localization never fails in a dynamic environment, and all the previously opened application windows are always rendered in the same location.

[0149] Fig. 19 illustrates a block diagram 1900 of the object information extractor module 1018, in accordance with an embodiment of the present disclosure. The object information extractor module 1018 may categorize the objects into two or more classes based on their movability, such as " Movable" or "fixed.". In an embodiment, the object information extractor module 1018 may be configured to categorize each of the one or more objects as one of a fixed object and a movable object based on size, position, and shape of the identified object from which the one or more feature points have been extracted. The input received by the object information extractor module 1018 may consist of an image 1902 and a list 1904 of feature points detected within the image. Further, the output information 1906 may assign a category to each one or more object associated with the extracted one or more feature point, indicating the movability of the identified one or more objects.

[0150] Fig. 20 illustrates a block diagram 2000 of the feature point filter module 1014 filtering based on landmark matching, in accordance with an embodiment of the present disclosure. The feature point filter module 1014, may be configured to discard, the one or more current feature points associated with the one or more identified objects that have moved to the current position from the pre-stored position. The feature point filter module 1014, may receive input from the evaluation module 1012 with a determination of whether any one or more objects have been moved to the current position from a pre-stored position within the real-world environment.

[0151] In operation, the input includes a pre-stored feature point list 2002 from the global map and a list 2004 of the current feature points extracted by the extractor module 1018 from the one or more identified objects from the input image. Further, the one or more current feature points extracted from the one or more identified objects are compared with one or more pre-stored feature points provided by the global map, establishing one-to-one matches. For example, matching an extracted current feature point “F11” with a pre-stored feature point “F3” from the global map; matching an extracted feature current point “F12” with a pre-stored landmark “F33” from the global map; matching an extracted current feature point “F13” with a pre-stored landmark “F9” from the global map; matching an extracted current feature point “F14” with a pre-stored landmark “F18” from the global map.

[0152] Further, the distance between the one or more landmark points associated with the fixed object and one or more landmark points associated with all the movable objects in the global map is computed. For example, computing a distance between the pre-stored landmark “F18” and the pre-stored landmark “F9” from the global map. Similarly, computing a distance between the pre-stored landmark “F18” and the pre-stored landmark “F22” from the global map.

[0153] Furthermore, the distance between corresponding points in the current extracted feature point list 2004 is computed. For Example, computing a distance between the extracted current feature point “F14” and the extracted current feature point “F13” from the current feature point extracted list 2004. Similarly, computing a distance between the extracted current feature point “F14” and the extracted current feature point “F12” from the current feature point extracted list 2004.

[0154] Furthermore, if the corresponding distances don't match, it is concluded that the object has moved. Accordingly, all feature points and / or landmark points corresponding to the moved object are removed from the global map.

[0155] For example, the computed distance between a pre-stored landmark “F18” and a pre-stored landmark “F33” from the global map” may not be equal to the computed distance between the extracted current feature point “F14” and the extracted current feature point “F12” from the current feature point extracted list 2004”. In this scenario, all the feature points and / or landmark points associated with object 4 are removed from the global map. Accordingly, the one or more current feature points and / or landmark points associated with object 4 may not be considered for performing re-localization of the HMD device 1000. The output from the feature point filter module 1014 may include a list 2006 of one or more pre-stored feature points whose position has not changed since mapping was performed.

[0156] Further, the feature point filter module 1014 filters out the one or more current feature points whose position has changed from the pre-stored position since mapping.

[0157] In an aspect of the present disclosure, a method for an enhanced re-localization of a head-mounted display (HMD) device is provided. FIG. 21 illustrates a flow chart of a method 2100 for an enhanced re-localization of a head-mounted display (HMD) device, in accordance with an embodiment of the present disclosure. The order in which the method steps are described below is not intended to be construed as a limitation, and any number of the described method steps can be combined in any appropriate order to execute the method or an alternative method. Additionally, individual steps may be deleted from the method without departing from the spirit and scope of the subject matter described herein.

[0158] At operation 2102, the method 2100 may include obtaining (e.g. identifying) one or more feature points of one or more objects identifed (e.g. present) in their current position in a real-world environment. In an embodiment, the one or more objects may include fixed objects and movable objects. At operation 2104, the method 2100 may include obtaining (e.g. deriving) 2104 one or more landmark points associated with each of the identified one or more objects from the one or more feature points corresponding to one or more pre-stored landmark points associated with each of the identified one or more objects of the real-world environment. At operation 2106, the method 2100 may include determining whether any one or more objects have moved to the current position from a pre-stored position within the real-world environment by comparing the obtained one or more landmark points with the one or more pre-stored landmark points associated with each of the one or more objects of the real-world environment. In an embodiment, the method 2100 may include comparing the one or more landmark points with the one or more pre-stored landmark points associated with each of the one or more objects of the real-world environment, to determine whether any one or more objects have moved to the current position from a pre-stored position within the real-world environment. Further, at operation 2108, the method 2100 may include filtering the one or more landmark points by discarding the one or more feature points associated with the one or more objects that have moved to the current position from the pre-stored position. Further, at operation 2110, the method 2100 may include performing re-localization of the HMD device 1000 using the filtered one or more landmark points.

[0159] In an embodiment, the method 2100 may further include performing pose estimation of the HMD device 1000 using the filtered one or more landmark points.

[0160] In an embodiment, the method 2100 may further include categorizing each of the one or more objects as one of a fixed object and a movable object based on size of the corresponding object by determining whether an estimated weight and computed volume of the corresponding identified object is greater than a defined (e.g. predetermined) threshold value.

[0161] In an embodiment, the method 2100 may further include categorizing each of the one or more objects as one of a fixed object and a movable object based on a placement of the corresponding object by determining whether the corresponding identified object is placed on at least one of a horizontal surface or a vertical surface.

[0162] In an embodiment, the one or more pre-stored landmark points and pre-stored feature points associated with each of the one or more objects of the real-world environment may be stored in a global map of the real-world environment. In an embodiment, the global map may further include information related to movability of the one or more objects. In an embodiment, the movability corresponds to the one or more objects being fixed objects and movable objects.

[0163] In an embodiment, one or more pre-stored landmark points associated with the one or more fixed objects and one or more pre-stored landmark points associated with the one or more movable objects are utilized for establishing the pre-stored position of each of the one or more fixed and movable objects of the real-world environment. In an embodiment, each pre-stored landmark point of a single identified object is at least a defined (e.g. predetermined) distance apart from another pre-stored landmark point of said single identified object.

[0164] In an embodiment, for comparing the one or more current landmark points with one or more pre-stored landmark points of the real-world environment, the method may include computing a distance between the one or more current landmark points associated with the one or more fixed objects and one or more current landmark points associated with the one or more movable objects. The method may further include comparing the computed distance with a distance between the one or more pre-stored landmark points associated with the one or more fixed objects and the one or more pre-stored landmark points associated with the one or more movable objects.

[0165] FIG. 22 illustrates a flow chart of a method 2200 for an enhanced re-localization of a head-mounted display (HMD) device, in accordance with an embodiment of the present disclosure.

[0166] At operation 2202, the method 2200 may include determining one or more current feature points associated with each of one or more identified objects from a real-world environment, the one or more identified objects include fixed objects and movable objects.

[0167] At operation 2204, the method 2200 may include retrieving one or more pre-stored feature points, a pre-stored position, and movability associated with each identified object based on a historic localization and mapping of the identified objects by referring to a global map of the real-world environment.

[0168] At operation 2206, the method 2200 may include determining that at least one movable object among the one or more identified objects has moved from the associated pre-stored position by correlating the one or more current and pre-stored feature points associated with the movable objects with the one or more current and pre-stored feature points associated with the fixed objects.

[0169] At operation 2208, the method 2200 may include filtering the one or more current feature points by discarding the feature points associated with the determined movable object that has moved.

[0170] At operation 2210, the method may include performing re-localization of the HMD device 1000 using the filtered one or more current feature points.

[0171] In an embodiment, the method 2200 may include performing pose estimation of the HMD device 1000 using the filtered one or more current feature points.

[0172] In an embodiment, before initiating the re-localization, the method may include performing the historic localization and mapping of the one or more identified objects to the global map of the real-world environment. The method for performing the historic localization may include identifying the one or more objects in the real-world environment captured by the HMD device 1000. The method may further include extracting one or more feature points associated with each of the one or more identified objects. The method may further include receiving information associated with each of the one or more identified objects. The method may further include categorizing the one or more identified objects as fixed objects and movable objects based on the received information. Further, the method may include performing the localization of the HMD device 1000 based on the extracted one or more feature points, and mapping the extracted feature points and movability associated with the one or more identified objects to the global map.

[0173] In an embodiment, the method 2200 may further include categorizing each of the one or more identified objects as one of a fixed object and a movable object based on size of the corresponding object, by determining whether an estimated weight and computed volume of the corresponding identified object is greater than a predetermined threshold value.

[0174] In an embodiment, the method 2200 may further include categorizing each of the one or more identified objects as one of a fixed object and a movable object based on a placement of the corresponding object, by determining whether the corresponding identified object is placed on at least one of a horizontal surface or a vertical surface.

[0175] In an embodiment, the method 2200 may further include deriving one or more landmark points of each of the identified objects from each of the current and pre-stored feature points.

[0176] In an embodiment, one or more pre-stored landmark points of the at least one fixed object and one or more pre-stored landmark points of the at least one movable object are utilized for establishing the pre-stored position of each of the one or more fixed and movable objects on the global map of the real-world environment. In an embodiment, each landmark point of a single identified object is at least a predetermined distance apart from another landmark point of said single identified object.

[0177] In an embodiment, for correlating the determined one or more current feature points and the one or more pre-stored feature points, the method 2200 may include computing a distance between the determined one or more current feature points associated with the one or more fixed objects and the determined one or more current feature points of the one or more movable objects, in an embodiment, the method 2200 may further include comparing the computed distance with a distance between the one or more pre-stored feature points associated with the one or more fixed objects and the one or more pre-stored feature points associated with the one or more movable objects.

[0178] According to an embodiment of the disclosure, a system for an enhanced re-localization of a head-mounted display (HMD) device is provided. In an embodiment, the system may comprise a feature extractor module (1010) configured for identifying one or more current feature points of one or more objects present in their current position in a real-world environment. The system may comprise an evaluation module (1012) configured for deriving one or more of landmark points associated with each of the objects from the current feature points corresponding to one or more pre-stored landmark points associated with each of the identified one or more objects of the real-world environment. The system may comprise an evaluation module (1012) configured for comparing the one or more landmark points with the one or more pre-stored landmark points associated with each of the one or more objects of the real-world environment, to determine whether any one or more objects have moved to the current position from a pre-stored position within the real-world environment. The system may comprise a feature point filter module (1014) configured for filtering the one or more landmark points by discarding the feature points associated with the one or more objects that have moved to the current position from the pre-stored position. The system may comprise a localization module (1016) configured for performing re-localization of the HMD device (1000) using the filtered one or more landmark points.

[0179] According to an embodiment of the disclosure, the localization module (1016) is further configured to perform pose estimation of the HMD device (1000) using the filtered one or more landmark points.

[0180] According to an embodiment of the disclosure, the system (1002) may comprise an object information extractor module (1018) configured for categorizing each of the one or more objects as one of a fixed object and a movable object based on size of the corresponding object, by determining whether an estimated weight and computed volume of the corresponding object is greater than a predetermined threshold value.

[0181] According to an embodiment of the disclosure, the object information extractor module (1018) is configured to categorize each of the one or more objects as one of a fixed object and a movable object based on a placement of the corresponding object, by determining whether the corresponding identified object is placed on at least one of a horizontal surface or a vertical surface.

[0182] According to an embodiment of the disclosure, the one or more pre-stored landmark points and prestored feature points associated with each of the one or more objects of the real-world environment are stored in a global map of the real-world environment, wherein the global map further comprises information related to movability of the one or more objects, wherein the movability corresponds to the one or more objects being fixed objects and movable objects.

[0183] According to an embodiment of the disclosure, each pre-stored landmark point of a single identified object is at least a predetermined distance apart from another pre-stored landmark point of said single identified object.

[0184] According to an embodiment of the disclosure, the one or more objects include fixed objects and movable objects.

[0185] According to an embodiment of the disclosure, the one or more pre-stored landmark points associated with the one or more fixed objects and one or more pre-stored landmark points associated with the one or more movable objects are utilized to establish the pre-stored position of each of the one or more fixed and movable objects of the real-world environment.

[0186] According to an embodiment of the disclosure, wherein to compare the one or more landmark points with one or more pre-stored landmark points of the real-world environment, evaluation module (1012) is configured to compute a distance between the one or more landmark points associated with the one or more fixed objects and one or more landmark points associated with the one or more movable objects. According to an embodiment of the disclosure, evaluation module (1012) is configured to compare the computed distance with a distance between the one or more pre-stored landmark points associated with the one or more fixed objects and the one or more pre-stored landmark points associated with the one or more movable objects.

[0187] According to an embodiment of the disclosure, a system for an enhanced re-localization of a head-mounted display (HMD) device (1000) is provided. In an embodiment, the system (1002) may comprise a feature extractor module (1010) configured for determining one or more current feature points associated with each of one or more identified objects from a real-world environment, the one or more identified objects include fixed objects and movable objects. The system may comprise an evaluation module (1012) configured for retrieving one or more pre-stored feature points, a pre-stored position, and movability associated with each identified object based on a historic localization and mapping of the identified objects by referring to a global map of the real-world environment. The system may comprise an evaluation module (1012) configured for determining that at least one movable object among the one or more identified objects has moved from the associated pre-stored position by correlating the one or more current and pre-stored feature points associated with the movable objects with the one or more current and pre-stored feature points associated with the fixed objects. The system may comprise a feature point filter module (1014) configured for filtering the one or more current feature points by discarding the feature points associated with the determined movable object that has moved. The system may comprise a localization module (1016) configured for performing re-localization of the HMD device (1000) by using the filtered one or more current feature points.

[0188] According to an embodiment of the disclosure, the localization module (1016) is configured to perform pose estimation of the HMD device (1000) by using the filtered one or more current feature points.

[0189] According to an embodiment of the disclosure, prior to the initiation of the re-localization, the system (1002) may perform the historic localization and mapping of the one or more identified objects to the global map of the real-world environment by: identifying the one or more objects in the real-world environment captured by the HMD device (1000), extracting one or more feature points associated with each of the one or more identified objects; receiving information associated with each of the one or more identified objects; categorizing the one or more identified objects as fixed objects and movable objects based on the received information; and performing the localization of the HMD device (1000) based on the extracted one or more feature points, and mapping the extracted feature points and movability associated with the one or more identified objects to the global map.

[0190] According to an embodiment of the disclosure, the system may comprise an object information extractor module (1018) configured for categorizing each of the one or more objects as one of a fixed object and a movable object based on size of the corresponding object, by determining whether an estimated weight and computed volume of the corresponding identified object is greater than a predetermined threshold value.

[0191] According to an embodiment of the disclosure, the object information extractor module (1018) is configured to categorize each of the one or more objects as one of a fixed object and a movable object based on a placement of the corresponding object, by determination of whether the corresponding identified object is placed on at least one of a horizontal surface or a vertical surface.

[0192] According to an embodiment of the disclosure, one or more landmark points are derived from each of the identified objects from each of the current and pre-stored feature points.

[0193] According to an embodiment of the disclosure, the one or more pre-stored landmark points associated with the at least one fixed object and the pre-stored landmark points associated with the at least one movable object are utilized to establish the pre-stored position of each of the one or more fixed and movable objects on the global map of the real-world environment.

[0194] According to an embodiment of the disclosure, each landmark point of a single identified object is at least a predetermined distance apart from another landmark point of said single identified object.According to an embodiment of the disclosure, to correlate the determined one or more current feature points and the one or more pre-stored feature points, the evaluation module (1012) is configured to: compute a distance between the determined one or more current feature points associated with the one or more fixed objects and the determined one or more current feature points of the one or more movable objects; and compare the computed distance with a distance between the one or more pre-stored feature points associated with the one or more fixed objects and the one or more pre-stored feature points associated with the one or more movable objects. According to an embodiment of the disclosure, a method for a re-localization of a head-mounted display (HMD) device may be disclosed. The method may comprise obtaining (2102) one or more feature points of one or more objects identified in their current position in a real-world environment. The method may comprise obtaining (2104) one or more landmark points associated with each of the identified one or more objects from the one or more feature points corresponding to one or more pre-stored landmark points associated with each of the identified one or more objects of the real-world environment. The method may comprise determining (2106) whether any one or more objects have moved to the current position from a pre-stored position within the real-world environment by comparing the obtained one or more landmark points with the one or more pre-stored landmark points associated with each of the one or more objects of the real-world environment. The method may comprise filtering (2108) the one or more landmark points by discarding the one or more feature points associated with the one or more objects that have moved to the current position from the pre-stored position. The method may comprise performing (2110) re-localization of the HMD device (1000) using the filtered one or more landmark points.

[0195] According to an embodiment of the disclosure, the method may comprise performing pose estimation of the HMD device (1000) using the filtered one or more landmark points.

[0196] According to an embodiment of the disclosure, the method may comprise categorizing each of the one or more objects as one of a fixed object and a movable object based on size of the corresponding object by determining whether an estimated weight and computed volume of the corresponding identified object is greater than a defined threshold value.

[0197] According to an embodiment of the disclosure, the method may comprise categorizing each of the one or more objects as one of a fixed object and a movable object based on a placement of the corresponding object by determining whether the corresponding identified object is placed on at least one of a horizontal surface or a vertical surface.

[0198] According to an embodiment of the disclosure, the one or more pre-stored landmark points and prestored feature points associated with each of the one or more objects of the real-world environment are stored in a global map of the real-world environment. In an embodiment, the global map further comprises information related to movability of the one or more objects. In an embodiment, the movability corresponds to the one or more objects being fixed objects and movable objects.

[0199] According to an embodiment of the disclosure, each pre-stored landmark point of a single identified object is at least a defined distance apart from another pre-stored landmark point of said single identified object.

[0200] According to an embodiment of the disclosure, the one or more objects include fixed objects and movable objects. In an embodiment, one or more pre-stored landmark points associated with the one or more fixed objects and one or more pre-stored landmark points associated with the one or more movable objects are utilized for establishing the pre-stored position of each of the one or more fixed and movable objects of the real-world environment.

[0201] According to an embodiment of the disclosure, comparing the one or more landmark points with one or more pre-stored landmark points of the real-world environment may comprise computing a distance between the one or more landmark points associated with the one or more fixed objects and one or more landmark points associated with the one or more movable objects; and comparing the computed distance with a distance between the one or more pre-stored landmark points associated with the one or more fixed objects and the one or more pre-stored landmark points associated with the one or more movable objects.

[0202] According to an embodiment of the disclosure, a system (1002) for performing a re-localization of a head-mounted display (HMD) device (1000) is provided. The system (1002) may comprise at least one processor (1004) including processing circuitry. In an embodiment, the system (1002) may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system (1002) to obtain one or more feature points of one or more objects identified in their current position in a real-world environment. In an embodiment, the system (1002) may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system (1002) to obtain one or more of landmark points associated with each of the identified one or more objects from the feature points corresponding to one or more pre-stored landmark points associated with each of the identified one or more objects of the real-world environment. In an embodiment, the system (1002) may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system (1002) to determine whether any one or more objects have moved to the current position from a pre-stored position within the real-world environment by comparing the obtained one or more landmark points with the one or more pre-stored landmark points associated with each of the one or more objects of the real-world environment. In an embodiment, the system (1002) may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system (1002) to filter the one or more landmark points by discarding the one or more feature points associated with the one or more objects that have moved to the current position from the pre-stored position. In an embodiment, the system (1002) may comprise memory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system (1002) to perform re-localization of the HMD device (1000) using the filtered one or more landmark points.

[0203] According to an embodiment of the disclosure, the one or more instructions, executed by the at least one processor (1004) individually or collectively, to cause the system (1002) to perform pose estimation of the HMD device (1000) using the filtered one or more landmark points.

[0204] According to an embodiment of the disclosure, the one or more instructions, executed by the at least one processor (1004) individually or collectively, to cause the system (1002) to categorize each of the one or more objects as one of a fixed object and a movable object based on size of the corresponding object, by determining whether an estimated weight and computed volume of the corresponding object is greater than a defined threshold value.

[0205] According to an embodiment of the disclosure, wherein the one or more pre-stored landmark points and prestored feature points associated with each of the one or more objects of the real-world environment are stored in a global map of the real-world environment, wherein the global map further comprises information related to movability of the one or more objects, wherein the movability corresponds to the one or more objects being fixed objects and movable objects.

[0206] According to an embodiment of the disclosure, the one or more objects include fixed objects and movable objects. In an embodiment, the one or more pre-stored landmark points associated with the one or more fixed objects and one or more pre-stored landmark points associated with the one or more movable objects are utilized to establish the pre-stored position of each of the one or more fixed and movable objects of the real-world environment.

[0207] According to an embodiment of the disclosure, the one or more instructions, executed by the at least one processor (1004) individually or collectively, to cause the system (1002) to compute a distance between the one or more landmark points associated with the one or more fixed objects and one or more landmark points associated with the one or more movable objects. In an embodiment, the one or more instructions, executed by the at least one processor (1004) individually or collectively, to cause the system (1002) to compare the computed distance with a distance between the one or more pre-stored landmark points associated with the one or more fixed objects and the one or more pre-stored landmark points associated with the one or more movable objects.

[0208] According to an embodiment of the disclosure, a computer-readable medium containing instructions is disclosed. The instructions, when executed by at least one processor, cause the head-mounted display (HMD) device (1000) to perform the method provided.According to the present disclosure, feature points pertaining to objects that have moved are smartly removed. By discarding all the feature points from the moved objects, re-localization becomes accurate in a dynamic environment, equivalent to the accuracy in a still environment. Further, as old and obsolete feature points belonging to a previous position of the moved object are removed, and new feature points for the moved object’s current position are added, decluttering the global map. Further, with smaller map sizes, memory consumption and power consumption are also reduced, preventing device overheating and improving runtime. This further cut down on power usage and memory requirements.

[0209] Further, according to the present disclosure, re-localization never fails in the dynamic environment, ensuring that previously opened application windows are always rendered at the right spot. Further, re-localization accuracy is significantly increased in a dynamic environment, preventing application windows from appearing in the wrong location.

[0210] In this application, unless specifically stated otherwise, the use of the singular includes the plural, and the use of “or” means “and / or.” Furthermore, the use of the terms “including” or “having” is not limiting. Any range described herein will be understood to include the endpoints and all values between the endpoints. Features of the disclosed embodiments may be combined, rearranged, omitted, etc., within the scope of the disclosure to produce additional embodiments. Furthermore, certain features may sometimes be used to advantage without a corresponding use of other features.

[0211] While at least one exemplary embodiment has been presented in the foregoing detailed description, it should be appreciated that a vast number of variations exist.

Claims

1.A method (2100) for a re-localization of a head-mounted display (HMD) device (1000), the method comprising:obtaining (2102) one or more feature points of one or more objects identified in their current position in a real-world environment;obtaining (2104) one or more landmark points associated with each of the identified one or more objects from the one or more feature points corresponding to one or more pre-stored landmark points associated with each of the identified one or more objects of the real-world environment;determining (2106) whether any one or more objects have moved to the current position from a pre-stored position within the real-world environment by comparing the obtained one or more landmark points with the one or more pre-stored landmark points associated with each of the one or more objects of the real-world environment;filtering (2108) the one or more landmark points by discarding the one or more feature points associated with the one or more objects that have moved to the current position from the pre-stored position; andperforming (2110) re-localization of the HMD device (1000) using the filtered one or more landmark points.2.The method (2100) as claimed in claim 1, comprising performing pose estimation of the HMD device (1000) using the filtered one or more landmark points.3.The method (2100) as claimed in claim 1 or claim 2, comprising categorizing each of the one or more objects as one of a fixed object and a movable object based on size of the corresponding object by determining whether an estimated weight and computed volume of the corresponding identified object is greater than a defined threshold value.4.The method (2100) as claimed in any one of claims 1 to 3, comprising categorizing each of the one or more objects as one of a fixed object and a movable object based on a placement of the corresponding object by determining whether the corresponding identified object is placed on at least one of a horizontal surface or a vertical surface.5.The method (2100) as claimed in any one of claims 1 to 4, wherein the one or more pre-stored landmark points and prestored feature points associated with each of the one or more objects of the real-world environment are stored in a global map of the real-world environment, wherein the global map further comprises information related to movability of the one or more objects, wherein the movability corresponds to the one or more objects being fixed objects and movable objects.6.The method (2100) as claimed in any one of claims 1 to 5, wherein each pre-stored landmark point of a single identified object is at least a defined distance apart from another pre-stored landmark point of said single identified object.7.The method (2100) as claimed in any one of claims 1 to 6, wherein the one or more objects include fixed objects and movable objects; and wherein one or more pre-stored landmark points associated with the one or more fixed objects and one or more pre-stored landmark points associated with the one or more movable objects are utilized for establishing the pre-stored position of each of the one or more fixed and movable objects of the real-world environment.8.The method (2100) as claimed in any one of claims 1 to 7, wherein comparing the one or more landmark points with one or more pre-stored landmark points of the real-world environment comprises:computing a distance between the one or more landmark points associated with the one or more fixed objects and one or more landmark points associated with the one or more movable objects; andcomparing the computed distance with a distance between the one or more pre-stored landmark points associated with the one or more fixed objects and the one or more pre-stored landmark points associated with the one or more movable objects.9.A system (1002) for a re-localization of a head-mounted display (HMD) device (1000), the system (1002) comprising:at least one processor (1004) including processing circuitry; andmemory storing one or more instructions that, when executed by the at least one processor individually or collectively, cause the system (1002) to:obtain one or more feature points of one or more objects identified in their current position in a real-world environment;obtain one or more of landmark points associated with each of the identified one or more objects from the one or more feature points corresponding to one or more pre-stored landmark points associated with each of the identified one or more objects of the real-world environment; anddetermine whether any one or more objects have moved to the current position from a pre-stored position within the real-world environmentby comparing the obtained one or more landmark points with the one or more pre-stored landmark points associated with each of the one or more objects of the real-world environment;filter the one or more landmark points by discarding the one or more feature points associated with the one or more objects that have moved to the current position from the pre-stored position; andperform re-localization of the HMD device (1000) using the filtered one or more landmark points.10.The system (1002) as claimed in claim 9, wherein the one or more instructions, executed by the at least one processor (1004) individually or collectively, to cause the system (1002) to:perform pose estimation of the HMD device (1000) using the filtered one or more landmark points.11.The system (1002) as claimed in claim 9 or claim 10, wherein the one or more instructions, executed by the at least one processor (1004) individually or collectively, to cause the system (1002) to:categorize each of the one or more objects as one of a fixed object and a movable object based on size of the corresponding object, by determining whether an estimated weight and computed volume of the corresponding object is greater than a defined threshold value.12.The system (1002) as claimed in any one of claims 9 to 11, wherein the one or more pre-stored landmark points and prestored feature points associated with each of the one or more objects of the real-world environment are stored in a global map of the real-world environment, wherein the global map further comprises information related to movability of the one or more objects, wherein the movability corresponds to the one or more objects being fixed objects and movable objects.13.The system (1002) as claimed in any one of claims 9 to 12, wherein the one or more objects include fixed objects and movable objects; and wherein the one or more pre-stored landmark points associated with the one or more fixed objects and one or more pre-stored landmark points associated with the one or more movable objects are utilized to establish the pre-stored position of each of the one or more fixed and movable objects of the real-world environment.14.The system (1002) as claimed in any one of claims 9 to 13, wherein the one or more instructions, executed by the at least one processor (1004) individually or collectively, to cause the system (1002) to: compute a distance between the one or more landmark points associated with the one or more fixed objects and one or more landmark points associated with the one or more movable objects; andcompare the computed distance with a distance between the one or more pre-stored landmark points associated with the one or more fixed objects and the one or more pre-stored landmark points associated with the one or more movable objects.15.A computer-readable medium containing instructions, wherein the instructions, when executed by at least one processor, cause the head-mounted display (HMD) device (1000) to perform the method of any one of claims 1 to 8.

Images