Method and system for digital twins of physical environments for extended reality applications

Abstract

A computer-implemented method and system for facilitating the generation of a digital twin of a physical environment for an extended reality application. The method includes receiving data associated with a representation of the physical environment and data associated with a representation of one or more objects in the physical environment, as well as data associated with the arrangement, structure, function, operating state, and / or performance of each of the one or more objects in the physical environment at a particular moment. The method further includes generating digital twin data for the extended reality application based on at least some of the received data, and providing the digital twin data to an extended reality device on which the extended reality application is installed to facilitate the generation of a digital twin of the physical environment having the one or more objects at a particular moment.

Description

[0001] The present invention relates to digital twins of physical environments for augmented reality applications. Specifically, the present invention relates to systems and methods for facilitating the generation or for generating digital twins of physical environments for augmented reality applications. Background Art

[0002] Engineers of engineering service providers (e.g., telecommunication service providers) often need to inspect, maintain, and / or repair devices located at different locations. Sometimes, an engineer may not be familiar with the device (e.g., a telecommunication device) to be inspected, maintained, and / or repaired (and thus how to inspect, maintain, and / or repair it), or the engineer may not be familiar with the location. Therefore, the engineer may have to spend extra time to complete the work. On the other hand, after the work at the location is completed, the engineer may need to update the records of the device and / or the location, typically using paper and / or spreadsheets. This process is sometimes time-consuming and prone to human errors. Summary of the Invention

[0003] Some embodiments of the present invention are aimed at solving one or more of the above needs or overcoming or generally improving one or more of the above disadvantages. Some other embodiments of the present invention may not solve the above needs and may not overcome or generally improve the above disadvantages (but solve other needs or problems).

[0004] In a first aspect, there is provided a computer-implemented method for facilitating the generation of a digital twin of a physical environment for an augmented reality application. The computer-implemented method includes: (a) receiving data associated with a representation of the physical environment and data associated with a representation of one or more objects in the physical environment, and (b) receiving data associated with the arrangement, structure, function, operating state, and / or performance of each of the one or more objects in the physical environment at a certain moment. The computer-implemented method further includes: (c) generating digital twin data for the augmented reality application based on at least some of the data received in steps (a) and (b), and (d) providing the digital twin data to an augmented reality device installed with the augmented reality application to facilitate the generation of a digital twin of the physical environment having one or more objects at the certain moment. Steps (a) to (d) may be executed at a data processing system operably connected to the augmented reality device installed with the augmented reality application. The digital twin can be used to improve the efficiency and / or reliability of an operation process (e.g., an operation to be performed or being performed by a user wearing the augmented reality device).

[0005] Optionally, the physical environment includes a data center environment (e.g., a server room). Optionally, the one or more objects include one or more containers and one or more devices disposed in or on the one or more containers. The one or more containers may include at least one of the following: racks, cabinets, and frames. The one or more devices may include at least one of the following: servers, routers, switches, disk memories, and power supplies.

[0006] Optionally, step (a) includes: (a1) receiving data associated with a graphical representation of the physical environment and data associated with a graphical representation of one or more objects in the physical environment.

[0007] Optionally, the data associated with the graphical representation of the physical environment includes a digital map of the physical environment. The digital map may be a two-dimensional (2D) map or a three-dimensional (3D) map of the physical environment.

[0008] Optionally, the data associated with the graphical representation of one or more objects in the physical environment includes corresponding models of the one or more objects. The corresponding model may be a 2D model or a 3D model. The corresponding 3D model may be defined by the width, depth, and height of the corresponding object.

[0009] Optionally, step (a) includes: (a2) receiving data associated with predefined features of the physical environment and / or data associated with predefined features of one or more objects in the physical environment. The predefined features may be user-defined features. In one example, only the data received in steps (a2) and (b) is used in step (c) (i.e., the data received in step (a1) is not used in step (c)).

[0010] Optionally, the data associated with the predefined features of the physical environment includes data associated with one or more predefined regions in the digital map of the physical environment. The data associated with one or more predefined regions in the digital map of the physical environment may include data associated with the location and area where the one or more regions are disposed (relative to the map of the physical environment), data associated with the orientation and / or type of the containers in the region, data associated with the number of rows and columns of the containers in the region, etc.

[0011] Optionally, the data associated with the predefined features of one or more objects in the physical environment includes data associated with a predefined layout of at least one of the one or more containers. The data associated with a predefined layout of at least one of the one or more containers may include data associated with the number of rows and / or columns of the compartments in at least one of the one or more containers, and data associated with the type of at least one of the one or more containers.

[0012] Optionally, the data associated with the predefined features of one or more objects in the physical environment includes data associated with one or more predefined points of interest of at least one of the one or more devices. The data associated with one or more predefined points of interest of at least one of the one or more devices may include data associated with the type of at least one of the one or more devices, and data associated with the location, area, and description of one or more predefined points of interest in the device. One or more predefined points of interest may be arranged on the surface of at least one of the one or more devices.

[0013] Optionally, the data in step (a) is received from an object model database that stores object model data associated with a representation of the physical environment and object model data associated with a representation of one or more objects in the physical environment.

[0014] Optionally, the method further includes providing data associated with a graphical representation of the physical environment and data associated with a graphical representation of one or more objects in the physical environment to an extended reality device installed with an extended reality application to facilitate the generation of a digital twin (or an updated digital twin) of the physical environment with one or more objects at that moment (or at another moment). The data associated with the graphical representation of the physical environment may include a digital map of the physical environment. The providing may be from the object model database to the extended reality device. The data associated with the graphical representation of one or more objects in the physical environment may include corresponding models of the one or more objects (e.g., 2D models or 3D models).

[0015] Optionally, step (b) includes receiving data associated with the arrangement, structure, function, operating state, and performance of each of one or more objects in the physical environment at that moment.

[0016] Optionally, the data associated with the arrangement of each of the one or more objects includes data associated with the location and area of each of the one or more objects in the physical environment at that moment. Optionally, the data associated with the structure of each of the one or more objects includes data associated with the internal and / or external structure of each of the one or more objects in the physical environment at that moment. The internal and / or external structure may include orientation, layout, and / or models associated with each corresponding object (e.g., a container, a device in or on the container, etc.). Optionally, the data associated with the function of each of the one or more objects includes data associated with inter-object or intra-object logical connections of each of the one or more objects in the physical environment at that moment. Optionally, the data associated with the performance of each of the one or more objects includes data associated with the historical performance of each of the one or more objects in the physical environment at that moment.

[0017] Optionally, the method further includes: (e) generating a digital twin of a physical environment having one or more objects at a certain moment based at least on digital twin data, and (f) presenting to a user wearing an extended reality device with an extended reality application installed a digital twin of a physical environment having one or more objects. Optionally, step (e) is performed at the extended reality device. Optionally, step (f) is performed at the extended reality device when the user wearing the extended reality device is in the physical environment.

[0018] Optionally, the method further includes: (g) receiving updated data associated with an updated (or changed) arrangement, structure, function, operating state, and / or performance of one or more objects in the physical environment at another moment after the certain moment, (h) generating updated digital twin data for the extended reality application based at least on the updated data, and (i) providing the updated digital twin data to an extended reality device with the extended reality application installed to facilitate generating an updated digital twin of the physical environment at the another moment. Steps (g) to (i) may be performed at a data processing system operably connected to the extended reality device with the extended reality application installed. The receiving may be performed automatically based on an update (or change) in the arrangement, structure, function, operating state, and / or performance.

[0019] Optionally, the updated data corresponds to dynamic data, data associated with the arrangement, structure, function, operating state, and / or performance of one or more objects in the physical environment that is not updated (e.g., not changed) at the another moment corresponds to static data, and in step (h), the updated digital twin data is generated based on the static data, the dynamic data, and the data received in step (a).

[0020] Optionally, the method further includes (j) generating an updated digital twin of a physical environment having one or more objects at another moment based at least on the updated digital twin data, and (k) presenting to a user wearing an extended reality device with an extended reality application installed an updated digital twin of a physical environment having one or more objects at the another moment. Optionally, step (j) is performed at the extended reality device. Optionally, step (k) is performed at the extended reality device when the user wearing the extended reality device is in the physical environment.

[0021] Optionally, the method further includes repeating steps (g) to (i) at least once based on detecting a change in the arrangement, structure, function, operating state, and / or performance of one or more objects in the physical environment, or when detecting a change in the arrangement, structure, function, operating state, and / or performance of one or more objects in the physical environment. The change can be detected by an extended reality device worn by a user in the physical environment. The repetition can be automatic without user intervention.

[0022] Optionally, the method further includes repeating steps (j) and (k) at least once based on the repetition of steps (g) to (i).

[0023] Optionally, the extended reality device or extended reality application is operable to assist a user wearing the extended reality device in the physical environment to navigate in the physical environment.

[0024] Optionally, the extended reality device or extended reality application is operable to guide a user wearing the extended reality device in the physical environment to a predetermined location in the physical environment.

[0025] Optionally, the extended reality device or extended reality application is operable to instruct a user wearing the extended reality device in the physical environment to perform a predetermined task in the physical environment.

[0026] Optionally, the extended reality device or extended reality application is operable to assist a user wearing the extended reality device in the physical environment to determine the structure, function, operating state, and / or performance of one or more objects in the physical environment.

[0027] Optionally, the data in step (b) and / or step (g) is received from a backend database that stores data associated with the physical environment and one or more objects in the physical environment.

[0028] Optionally, the data in step (b) and / or step (g) is received in response to a user input received at an extended reality device on which the extended reality application is installed.

[0029] Optionally, the extended reality device or extended reality application is operable to facilitate an update of the backend database based on detected changes in the structure, function, operating state, and / or performance of one or more objects when a user wearing the extended reality device manipulates the one or more objects.

[0030] Optionally, the extended reality application is one of the following: a virtual reality application, an augmented reality application, and a mixed reality application; and the extended reality device is the corresponding one of the following: a virtual reality device, an augmented reality device, and a mixed reality device.

[0031] Optionally, the method further includes: (l) receiving update data associated with a representation of the physical environment and / or update data associated with a representation of one or more objects in the physical environment; (m) generating updated digital twin data for an extended reality application based at least on the update data; and (n) providing the updated digital twin data to an extended reality device on which the extended reality application is installed to facilitate generating an updated digital twin of the physical environment at a moment in time.

[0032] In a second aspect, a method for operating an extended reality device on which an extended reality application is installed is provided. The method includes receiving digital twin data (and updated digital twin data) generated based on the method of the first aspect, generating a digital twin (and an updated digital twin) based at least on the received digital twin data (and updated digital twin data), and presenting the digital twin (and updated digital twin) to a user wearing the extended reality device. The user is preferably located in the physical environment. These method steps are preferably all performed at the extended reality device.

[0033] Optionally, the method further includes receiving at the extended reality device data associated with a graphical representation of the physical environment and data associated with a representation of one or more objects in the physical environment to facilitate generating a digital twin (or updated digital twin) of the physical environment with one or more objects at a certain moment in time (or another moment in time). The data associated with the graphical representation of the physical environment may include a digital map of the physical environment. The providing may be from an object model database to the extended reality device. The data associated with the graphical representation of one or more objects in the physical environment may include respective models of the one or more objects (e.g., 2D models or 3D models).

[0034] Optionally, the extended reality device or the extended reality application is operable to use the digital twin to: assist a user wearing the extended reality device in navigating in the physical environment. Optionally, the method further includes presenting a map or layout of the physical environment to the user using the digital twin. The presentation may be based on user input received at the extended reality device.

[0035] Optionally, the extended reality device or the extended reality application is operable to use the digital twin to: guide a user wearing the extended reality device in the physical environment to a predetermined location in the physical environment. Optionally, the method further includes presenting a travel path to a predetermined location or object in the physical environment to the user using the digital twin. Optionally, the method further includes presenting the location of one or more objects in the physical environment to the user using the digital twin. The presentation may be based on user input received at the extended reality device.

[0036] Optionally, the extended reality device or extended reality application is operable to use the digital twin to: guide a user wearing the extended reality device in a physical environment to perform a predetermined task in the physical environment. Optionally, the method further includes presenting instructions for performing the predetermined task to the user using the digital twin. The presentation may be based on user input received at the extended reality device.

[0037] Optionally, the extended reality device or extended reality application is operable to use the digital twin to: assist a user wearing the extended reality device in a physical environment to determine the structure, function, operating state, and / or performance of one or more objects in the physical environment. Optionally, the method further includes presenting information associated with the structure, function, operating state, and / or performance of the one or more objects to the user using the digital twin. Optionally, the method further includes presenting information associated with the updated structure, function, operating state, and / or performance of the one or more objects to the user using the updated digital twin. The presentation may be based on user input received at the extended reality device.

[0038] Optionally, the method further includes detecting a change in the arrangement, structure, function, operating state, and / or performance of one or more objects in the physical environment (e.g., the change is a result of a user's manipulation of the one or more objects); and transmitting data associated with the detected change to a backend database. The detection and / or transmission may be performed by the extended reality device installed with the extended reality application. Transmitting data associated with the detected change to the backend database may trigger steps (g) to (i) (and steps (j) and (k)) in the first aspect.

[0039] Optionally, the extended reality application is one of the following: a virtual reality application, an augmented reality application, and a mixed reality application; and the extended reality device is the corresponding one of the following: a virtual reality device, an augmented reality device, and a mixed reality device.

[0040] In a third aspect, a system is provided for facilitating the generation of a digital twin of a physical environment for an extended reality application. The system includes one or more processors arranged to perform or facilitate the performance of the method of the first aspect. The one or more processors are arranged to: (a) receive data associated with a representation of the physical environment and data associated with a representation of one or more objects in the physical environment, and (b) receive data associated with the arrangement, structure, function, operating state, and / or performance of each of the one or more objects in the physical environment at a certain moment. The one or more processors are further arranged to: (c) generate digital twin data for the extended reality application based on at least some of the data received in steps (a) and (b), and (d) provide the digital twin data to an extended reality device on which the extended reality application is installed to facilitate the generation of a digital twin of the physical environment having the one or more objects at the certain moment. The system may be a data processing system operably connected to an extended reality device on which the extended reality application is installed.

[0041] Optionally, the physical environment includes a data center environment (e.g., a server room). Optionally, the one or more objects include one or more containers and one or more devices arranged in or on the one or more containers. The one or more containers may include at least one of the following: a rack, a cabinet, and a frame. The one or more devices may include at least one of the following: a server, a router, a switch, a disk storage, and a power supply.

[0042] Optionally, for (a), the one or more processors are arranged to: (a1) receive data associated with a graphical representation of the physical environment and data associated with a graphical representation of one or more objects in the physical environment.

[0043] Optionally, the data associated with the graphical representation of the physical environment includes a digital map of the physical environment. The digital map may be a two-dimensional (2D) map or a three-dimensional (3D) map of the physical environment.

[0044] Optionally, the data associated with the graphical representation of one or more objects in the physical environment includes corresponding models of the one or more objects. The corresponding models may be 2D models or 3D models. The corresponding 3D models may be defined by the width, depth, and height of the corresponding objects.

[0045] Optionally, for (a), the one or more processors are arranged to: (a2) receive data associated with predefined features of the physical environment and / or data associated with predefined features of one or more objects in the physical environment. The predefined features may be user-defined features.

[0046] Optionally, the data associated with the predefined features of the physical environment includes data associated with one or more predefined regions in the digital map of the physical environment. The data associated with one or more predefined regions in the digital map of the physical environment may include data associated with the location and area where the one or more regions are arranged (relative to the map of the physical environment), data associated with the orientation and / or type of the containers in the region, data associated with the number of rows and columns of the containers in the region, etc.

[0047] Optionally, the data associated with the predefined features of one or more objects in the physical environment includes data associated with the predefined layout of at least one of the one or more containers. The data associated with the predefined layout of at least one of the one or more containers may include data associated with the number of rows and / or columns of the compartments in at least one of the one or more containers, and data associated with the type of at least one of the one or more containers.

[0048] Optionally, the data associated with the predefined features of one or more objects in the physical environment includes data associated with one or more predefined points of interest of at least one of the one or more devices. The data associated with one or more predefined points of interest of at least one of the one or more devices may include data associated with the type of at least one of the one or more devices, and data associated with the location, area, and description of one or more predefined points of interest in the device. One or more predefined points of interest may be arranged on the surface of at least one of the one or more devices.

[0049] Optionally, the data in (a) is received from an object model database that stores object model data associated with the representation of the physical environment and object model data associated with the representation of one or more objects in the physical environment.

[0050] Optionally, for (b), the one or more processors are arranged to: receive data associated with the arrangement, structure, function, operating state, and performance of each of the one or more objects in the physical environment at a certain moment.

[0051] Optionally, the data associated with the arrangement of each of the one or more objects includes data associated with the position and area of each of the one or more objects in the physical environment at a certain moment. Optionally, the data associated with the structure of each of the one or more objects includes data associated with the internal and / or external structure of each of the one or more objects in the physical environment at a certain moment. The internal and / or external structure may include orientation, layout, and / or model associated with each corresponding object (e.g., a container, a device in or on the container, etc.). Optionally, the data associated with the function of each of the one or more objects includes data associated with the inter-object or intra-object logical connections of each of the one or more objects in the physical environment at a certain moment. Optionally, the data associated with the performance of each of the one or more objects includes data associated with the historical performance of each of the one or more objects in the physical environment at a certain moment.

[0052] Optionally, the one or more processors are further arranged to: (g) receive updated data associated with an updated (or changed) arrangement, structure, function, operating state, and / or performance of one or more objects in the physical environment at another moment after the certain moment, (h) generate updated digital twin data for the extended reality application based at least on the updated data, and (i) provide the updated digital twin data to an extended reality device on which the extended reality application is installed to facilitate generating an updated digital twin of the physical environment at the another moment. The receiving may be automatic based on an update (or change) in the arrangement, structure, function, operating state, and / or performance.

[0053] Optionally, the updated data corresponds to dynamic data, and the data associated with the arrangement, structure, function, operating state, and / or performance of one or more objects in the physical environment that is not updated (e.g., not changed) at the another moment corresponds to static data, and in (h), the updated digital twin data is generated based on the static data, the dynamic data, and the data received in (a).

[0054] Optionally, the one or more processors are further arranged to: based on detecting a change in the arrangement, structure, function, operating state, and / or performance of one or more objects in the physical environment, or when detecting a change in the arrangement, structure, function, operating state, and / or performance of one or more objects in the physical environment, repeat (g) to (i) at least once. The change may be detected by an extended reality device worn by a user in the physical environment. The repetition may be automatic without user intervention.

[0055] Optionally, the extended reality application is operable to: assist a user wearing an extended reality device in the physical environment to navigate in the physical environment.

[0056] Optionally, the extended reality application is operable to: guide a user wearing an extended reality device in a physical environment to a predetermined location in the physical environment.

[0057] Optionally, the extended reality application is operable to: instruct a user wearing an extended reality device in a physical environment to perform a predetermined task in the physical environment.

[0058] Optionally, the extended reality application is operable to: assist a user wearing an extended reality device in a physical environment in determining the structure, function, operating state, and / or performance of one or more objects in the physical environment.

[0059] Optionally, the data in (b) and / or (g) is received from a backend database that stores data associated with the physical environment and one or more objects in the physical environment.

[0060] Optionally, the data in (b) and / or (g) is received in response to user input received at an extended reality device on which the extended reality application is installed.

[0061] Optionally, the extended reality application is operable to: when a user wearing an extended reality device manipulates one or more objects, facilitate an update of the backend database based on detected changes in the structure, function, operating state, and / or performance of the one or more objects.

[0062] Optionally, the extended reality application is one of the following: a virtual reality application, an augmented reality application, and a mixed reality application; and the extended reality device is the corresponding one of the following: a virtual reality device, an augmented reality device, and a mixed reality device.

[0063] Optionally, the one or more processors are further arranged to: (l) receive update data associated with a representation of the physical environment and / or update data associated with a representation of one or more objects in the physical environment; (m) generate updated digital twin data for the extended reality application based at least on the update data; and (n) provide the updated digital twin data to an extended reality device on which the extended reality application is installed to facilitate the generation of an updated digital twin of the physical environment at a moment.

[0064] In a fourth aspect, there is provided an extended reality device on which an extended reality application is installed. The extended reality device is arranged to receive digital twin data (and updated digital twin data) generated based on the method of the first aspect, generate a digital twin (and an updated digital twin) based at least on the received digital twin data (and updated digital twin data), and present the digital twin (and the updated digital twin) to a user wearing the extended reality device. The user is preferably located in a physical environment.

[0065] Optionally, the extended reality device is further arranged to receive data associated with a graphical representation of the physical environment and data associated with a representation of one or more objects in the physical environment to facilitate the generation (or update) of a digital twin of the physical environment with one or more objects at a certain moment (or another moment). The data associated with the graphical representation of the physical environment may include a digital map of the physical environment. The provision may be from an object model database to the extended reality device. The data associated with the graphical representation of one or more objects in the physical environment may include corresponding models of the one or more objects (e.g., 2D models or 3D models).

[0066] Optionally, the extended reality device or the extended reality application is operable to use the digital twin to: assist a user wearing the extended reality device in the physical environment to navigate in the physical environment. Optionally, the extended reality device or the extended reality application is operable to use the digital twin to present a map or layout of the physical environment to the user. The presentation may be based on user input received at the extended reality device.

[0067] Optionally, the extended reality device or the extended reality application is operable to use the digital twin to: guide a user wearing the extended reality device in the physical environment to a predetermined location in the physical environment. Optionally, the extended reality device or the extended reality application is operable to use the digital twin to present a travel path to a predetermined location or object in the physical environment to the user. Optionally, the extended reality device or the extended reality application is operable to use the digital twin to present the location of one or more objects in the physical environment to the user. The presentation may be based on user input received at the extended reality device.

[0068] Optionally, the extended reality device or the extended reality application is operable to use the digital twin to: guide a user wearing the extended reality device in the physical environment to perform a predetermined task in the physical environment. Optionally, the extended reality device or the extended reality application is operable to use the digital twin to present instructions for performing the predetermined task to the user. The presentation may be based on user input received at the extended reality device.

[0069] Optionally, the extended reality device or the extended reality application is operable to use the digital twin to: assist a user wearing the extended reality device in a physical environment to determine the structure, function, operating state, and / or performance of one or more objects in the physical environment. Optionally, the extended reality device or the extended reality application is operable to use the digital twin to present information associated with the structure, function, operating state, and / or performance of one or more objects to the user. Optionally, the extended reality device or the extended reality application is operable to use the updated digital twin to present information associated with the updated structure, function, operating state, and / or performance of one or more objects to the user. The presentation may be based on user input received at the extended reality device.

[0070] Optionally, the extended reality device is further arranged to detect changes in the arrangement, structure, function, operating state, and / or performance of one or more objects in the physical environment (e.g., the change is the result of a user's manipulation of one or more objects); and transmit data associated with the detected change to a backend database. Transmitting the data associated with the detected change to the backend database may trigger an update of the digital twin.

[0071] Optionally, the extended reality device is further arranged to: (e) generate a digital twin of the physical environment having one or more objects at a certain moment, at least based on digital twin data, and (f) present the digital twin of the physical environment having one or more objects to a user wearing the extended reality device installed with the extended reality application.

[0072] Optionally, the extended reality device is further arranged to: (j) generate an updated digital twin of the physical environment having one or more objects at another moment, at least based on the updated digital twin data, and (k) present the updated digital twin of the physical environment having one or more objects at the other moment to a user wearing the extended reality device installed with the extended reality application.

[0073] Optionally, the extended reality application is one of the following: a virtual reality application, an augmented reality application, and a mixed reality application; and the extended reality device is the corresponding one of the following: a virtual reality device, an augmented reality device, and a mixed reality device.

[0074] In a fifth aspect, there is provided a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors, the one or more programs including instructions for performing the method of the first aspect.

[0075] In a sixth aspect, there is provided a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors, the one or more programs including instructions for performing the method of the second aspect.

[0076] In a seventh aspect, there is provided a computer program product comprising one or more programs configured to be executed by one or more processors, the one or more programs comprising instructions for performing the method of the first aspect. The computer program product may be implemented using hardware and / or software.

[0077] In an eighth aspect, there is provided a computer program product comprising one or more programs configured to be executed by one or more processors, the one or more programs comprising instructions for performing the method of the second aspect. The computer program product may be implemented using hardware and / or software.

[0078] In a ninth aspect, there is provided a system comprising the system of the third aspect and an extended reality device of the fourth aspect. The system of the third aspect and the extended reality device of the fourth aspect are arranged to communicate data with each other.

[0079] In a tenth aspect, there is provided a method comprising the method of the first aspect and the method of the second aspect.

[0080] By considering the detailed description and the drawings, other features and aspects of the present invention will become apparent. Where appropriate and applicable, any one or more features described herein with respect to one aspect or embodiment may be combined with any one or more other features described herein with respect to any one or more other aspects or embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Embodiments of the present invention will now be described by way of example with reference to the drawings, in which:

[0082] Figure 1 is a schematic diagram of a system according to an embodiment of the present invention;

[0083] Figure 2 is a flowchart of a method for facilitating the generation of a digital twin according to an embodiment of the present invention;

[0084] Figure 3 is a flowchart of a method for operating an extended reality device installed with an extended reality application according to an embodiment of the present invention;

[0085] Figure 4 is a schematic diagram for facilitating the generation of a digital twin according to an embodiment of the present invention;

[0086] Figure 5A is a schematic diagram of a 3D map of an example physical environment;

[0087] Figure 5B is a schematic diagram of a 2D map of an example physical environment;

[0088] Figure 5C Schematic diagram of a 3D model of an example container;

[0089] Figure 5D Schematic diagram of a 3D model of an example router;

[0090] Figure 5E Schematic diagram of a 3D model of an example switch;

[0091] Figure 6A Schematic diagram showing the definition of regions in a physical environment map in an example;

[0092] Figure 6B Schematic diagram showing placing a container in a region in an example;

[0093] Figure 6C Schematic diagram showing the definition of the layout of a region in an example;

[0094] Figure 6D Schematic diagram showing the allocation of containers in a region in an example;

[0095] Figure 7A Schematic diagram of a 3D model of a container in an example;

[0096] Figure 7B Shows Figure 7A Schematic diagram of the definition of the layout of the container;

[0097] Figure 8A Schematic diagram showing a 3D model of an example switch;

[0098] Figure 8B Shows Figure 8A Schematic diagram of the definition of the layout of points of interest on the face of the switch;

[0099] Figure 9 Schematic diagram showing the order of containers in an example;

[0100] Figure 10A Diagram showing the digital twin representation for object / path finding applications;

[0101] Figure 10B Another diagram showing the digital twin representation for object / path finding applications;

[0102] Figure 11A Diagram showing the digital twin representation for data visualization applications;

[0103] Figure 11B Another diagram showing the digital twin representation for data visualization applications;

[0104] Figure 11C is another figure showing a digital twin representation for a data visualization application;

[0105] Figure 12 is a flowchart of a method of using a digital twin representation for a mixed reality human - machine interaction application;

[0106] Figure 13 is a schematic diagram of a system according to an embodiment of the present invention; and

[0107] Figure 14 is a block diagram of an information processing system according to an embodiment of the present invention, which is arranged to implement at least a part of one or more methods of the present invention. Detailed Embodiments

[0108] Figure 1 FIG. shows a system 100 according to an embodiment of the present invention. The system 100 includes an extended reality device 102 and a data processing system 104. The extended reality device 102 is installed with an extended reality application. The data processing system 104 is used to facilitate the generation of a digital twin of a physical environment for the extended reality application. In one example, the physical environment is a data center environment having one or more objects. The objects include containers (e.g., racks, cabinets, frames, etc.) and devices (e.g., servers, routers, switches, disk memories, power supplies, etc.) arranged in or on the containers. The extended reality device 102 can be a virtual reality device (e.g., a headset), an augmented reality device (e.g., a headset), or a mixed reality device (e.g., a headset). The extended reality application can be a virtual reality application, an augmented reality application, or a mixed reality application. The extended reality device 102 can be worn or carried by a user who may be in the physical environment. The system 104 can include a processor, a computer, or a server, or multiple processors, computers, and / or servers operably connected to each other. The extended reality device 102 and the system 104 are arranged to communicate data via one or more wired communication links or wireless communication links.

[0109] Figure 2 FIG. shows a method 200 arranged to be executed by the system 104 according to an embodiment of the present invention. Unless otherwise specified, the method steps in the shown method 200 are executed at the system 104.

[0110] Method 200 includes step 202, in which data associated with a representation of a physical environment and data associated with a representation of one or more objects in the physical environment are received. In one example, step 202 involves receiving data associated with a graphical representation of the physical environment and data associated with a graphical representation of an object in the physical environment. The data associated with the graphical representation of the physical environment may include a digital map (2D or 3D) of the physical environment. The data associated with the graphical representation of an object in the physical environment may include a corresponding model of the object (e.g., a 3D model defined by width, depth, and / or height). In one example, step 202 involves receiving data associated with predefined (e.g., user-defined) features of the physical environment and / or data associated with predefined features of an object in the physical environment. The data associated with the predefined features of the physical environment includes data associated with predefined regions in the digital map of the physical environment, which may include, for example: data associated with the location of the region in the environment and the region, data associated with the orientation and / or type of a container in the region, data associated with the number of rows and columns of containers in the region, etc. The data associated with the predefined features of an object in the physical environment includes data associated with a predefined layout of at least one container and / or data associated with one or more predefined points of interest of at least one of the devices. The data associated with the predefined layout of at least one container may include data associated with the number of rows and / or columns of compartments in at least one container, and data associated with the type of at least one container. The data associated with one or more predefined points of interest of at least one of the devices may include data associated with the type of at least one of the devices, and data associated with the location, region, and description of the point of interest (e.g., the point of interest may be arranged on a face of at least one of the devices). The data in step 202 may be received from a user interface (user input), from the local memory of system 104 (e.g., an object model database), and / or from an external memory connected to system 104.

[0111] Method 200 also includes step 204, in which data associated with the arrangement, structure, function, operating state, and / or performance of each object in the physical environment is received. In one example, step 204 involves receiving data associated with the arrangement, structure, function, operating state, and performance (all five items) of each object in the physical environment. The data may represent the arrangement, structure, function, operating state, and / or performance of each object in the physical environment at a particular moment (it should be understood that the arrangement, structure, function, operating state, and / or performance of each object in the physical environment may change over time). The data associated with the arrangement of each object may include data associated with the position and area of each object in the physical environment. The data associated with the structure of each object may include data associated with the internal and / or external structure of each object in the physical environment. The data associated with the function of each object may include data associated with the inter-object or intra-object logical connections of each object in the physical environment. Optionally, the data associated with the performance of each object may include data associated with the historical performance of each object in the physical environment. According to an embodiment, step 204 may be performed before step 202, after step 202, or simultaneously with step 202. The data in step 204 may be received from a user interface (user input), from the local memory 4 of system 104, and / or from an external memory connected to system 104 (e.g., a backend database storing data associated with the physical environment and the objects in the physical environment).

[0112] After steps 202 and 204, method 200 proceeds to step 206, in which digital twin data for the extended reality application is generated based on the data received in steps 202 and 204. The digital twin data can be used to facilitate the generation of a digital twin of the physical environment with one or more objects.

[0113] In step 208, the generated digital twin data is provided to the extended reality device 102 installed with the extended reality application to facilitate the generation of a digital twin of the physical environment with one or more objects at a certain moment (corresponding to the moment of the data in step 204), which may be substantially real-time to reflect the real-time physical environment in the real world.

[0114] If the data of step 204 and / or the data of step 202 are changed or updated, method 200 may return to step 202 and step 204 and repeat with the updated data to generate updated digital twin data and provide the updated digital twin data to the extended reality device 102 installed with the extended reality application. In one example, method 200 includes detecting a change or update of the data in step 202 and step 204, and automatically repeating method 200 based on the changed or updated data when the change or update is detected. This allows the digital twin to stay in sync or up-to-date with the real-time physical environment in the real world. In one example, system 104 will be informed of the change or update of the data. In one example, system 104 will periodically or cyclically determine whether a change or update of the data has occurred.

[0115] Figure 3 Method 300, arranged to be executed by an extended reality device 102 installed with an extended reality application, of one embodiment of the present invention is shown. Unless otherwise stated, the method steps in method 300 are executed at the extended reality device 102.

[0116] Method 300 begins at step 302, where digital twin data is received at the extended reality device 102. The digital twin data may be the digital twin data provided from system 104 in step 208.

[0117] In step 304, a digital twin of the physical environment with one or more objects is generated at the extended reality device 102. The generation of the digital twin may be automatic or in response to a user input (e.g., a request) at the device 102. The generation of the digital twin is at least based on the digital twin data and the data associated with the graphical representation of the physical environment and its objects. The data associated with the graphical representation of the physical environment and its objects may be part of the digital twin data or a separate data stream (e.g., from the same data source or different data sources).

[0118] In step 306, the digital twin of the physical environment with one or more objects is presented to the user wearing the extended reality device 102. The presentation of the digital twin may be automatic or in response to a user input (e.g., a request) at the device 102. The user wearing the extended reality device 102 may be in the physical environment and optionally interact with the physical environment.

[0119] Presenting a digital twin to a user can increase the efficiency and / or reliability of an operation process that the user is about to perform or is performing. In one application, the digital twin can assist a user wearing an extended reality device 102 in a physical environment to navigate in the physical environment. For example, the digital twin can be used to present a map or layout of the physical environment to the user. In one application, the digital twin can guide a user wearing an extended reality device 102 in a physical environment to a predetermined location in the physical environment. For example, the digital twin can be used to present a travel path to a predetermined location or object in the physical environment or the location of an object in the physical environment. In one application, the digital twin can guide a user wearing an extended reality device 102 in a physical environment to perform a predetermined task. For example, the digital twin can be used to present instructions for performing a predetermined task to the user. In one application, the digital twin can assist a user wearing an extended reality device 102 in a physical environment to determine the structure, function, operating state, and / or performance of one or more objects in the physical environment. For example, the digital twin can present the structure, function, operating state, and / or performance of an object or updated structure, function, operating state, and / or performance to the user.

[0120] Method 300 may further include step 308, where the extended reality device 102 detects a change in the arrangement, structure, function, operating state, and / or performance of an object in the physical environment. The detection can be performed using one or more sensors of the device 102 (e.g., an image sensor, a motion sensor, a depth sensor) or one or more sensors operatively coupled to the device 102 in other ways. The detection can be performed as a result of an object in the environment being manipulated (e.g., by the user).

[0121] Method 300 may further include step 310, where data associated with the detected change is transmitted to an external device (e.g., a backend database that stores data associated with the physical environment and the objects in the physical environment). In one example, this may in turn trigger an update in step 204 of method 200 and a repetition of method 200, resulting in receiving updated digital twin data in step 302 and at least repeating steps 302 to 306.

[0122] Figure 4Illustrated is a diagram 400 for facilitating the generation of a digital twin according to an embodiment of the present invention. Diagram 400 may be considered a specific implementation of method 200. In diagram 400, a user U (not necessarily a user wearing an extended reality device) may input data associated with a physical environment (e.g., a data center environment) and a representation of an object in the physical environment through a user interface. The data may include data associated with: a digital map of the physical environment, a 3D model of an object in the physical environment, and user-defined inputs or features associated with the physical environment and / or the object in the physical environment (e.g., user-defined map regions, container and point-of-interest layouts). The user interface may be a user interface provided by a data processing system. The data provided by user U may be used to form or may be stored in an object model database (OMD). The object model database may be part of the data processing system or may be separate from but operably connected to the data processing system.

[0123] The backend system database may be a database storing data associated with a physical environment and an object in the physical environment. The data stored in the database may include data associated with the arrangement, structure, function, operating state, and / or performance of an object in the physical environment. The data stored in the backend system database may include static data (i.e., data that generally remains relatively stable or unchanged over time) and dynamic data (i.e., data that may change over time). The static data may be related to the structural characteristics of the physical environment and / or the object. The dynamic data may be related to the functional characteristics of an object in the physical environment. The data in the backend system database may be updated, optionally substantially in real time, to reflect the real-time conditions of the physical environment and the object.

[0124] User-defined inputs or features associated with the physical environment and / or the object (e.g., user-defined map regions, container and point-of-interest layouts), as well as static data and dynamic data from the backend system database, are combined to create a set of digital twin data for creating a digital twin of the physical environment. The created digital twin data may be referred to as real-time digital twin data. The real-time digital twin data and data associated with the digital map of the physical environment and the 3D model of an object in the physical environment may be provided to an augmented reality (AR) application installed in an augmented reality device that may be worn by a user in the physical environment. The augmented reality device with the augmented reality application may generate a digital twin based on the received data.

[0125] In some examples, an augmented reality application of an augmented reality device can detect changes in the arrangement, structure, function, operating state, and / or performance of objects in a physical environment. The detected changes (i.e., data associated with the detected changes ("partial dynamic data")) can be provided to a backend system database. The data associated with the detected changes can be stored in the backend system database and can overwrite or may not overwrite existing old data.

[0126] Figure 5A A digital map (3D) of an example physical environment is shown, which can be used as an example of data associated with the representation of the physical environment in illustration 400.

[0127] Figure 5B A digital map (2D) of an example physical environment is shown, which can be used as an example of data associated with the representation of the physical environment in illustration 400.

[0128] Figure 5C A model (3D) of an example container (e.g., a server rack) is shown, which can be used as an example of data associated with the representation of an object in the physical environment in illustration 400.

[0129] Figure 5D A model (3D) of an example router is shown, which can be used as an example of data associated with the representation of an object in the physical environment in illustration 400.

[0130] Figure 5E A model (3D) of an example switch is shown, which can be used as an example of data associated with the representation of an object in the physical environment in illustration 400.

[0131] Figures 6A to 6D The creation of the layout of a user-defined map area in one example is shown. The layout of the user-defined map area can be used as an example of user-defined input or features in illustration 400. In Figure 6A which, the area can be defined (e.g., drawn) by the user in the map of the physical environment. In Figure 6B which, the user can select the objects to be placed in the defined area, such as containers. As part of this selection, the user can input the 3D model name of the container and the orientation of the container (e.g., vertical, horizontal, etc.). In Figure 6C which, the layout of the area can be defined by the user. The layout of the area can include the number of containers in the area (e.g., the number of rows of containers and the number of columns of containers). As part of the definition of the layout of the area, the user can input the desired number of rows and columns of containers. In Figure 6D which, once Figures 6A to 6COnce the operations in [the system] are completed, the system can automatically allocate containers to regions based on the received user input. In one example, the layout of the user-defined map regions can include information associated with the location and regions of the regions in the map, the 3D model name / identifier of the containers in the regions, the orientation of the containers in the regions, the location of the containers, and the container layout identifier, etc.

[0132] Figure 7A and Figure 7B illustrates the creation of the layout of a user-defined container in one example. The layout of the user-defined container can include information associated with the number of compartments in the container (e.g., the number of rows of compartments and the number of columns of compartments) and the 3D model name / identifier of the corresponding container.

[0133] Figure 8A and Figure 8B illustrates the creation of the layout of user-defined points of interest in one example. The layout of the user-defined points of interest can include information associated with the 3D model name / identifier of the device and the points of interest on the front of the device (e.g., location, region, description, etc.). The points of interest can be associated with tasks that the user wants to perform with the assistance of an extended reality application / device.

[0134] In one example, the static data in FIG. 400 can include information associated with a set of regions and a set of container sub-regions in the map of the environment. The information associated with a set of regions in the map of the environment can include the start position of the region (e.g., upper left, upper right, lower left, lower right, etc.), the sorting of the containers in the region (column / row, discrete / continuous), and a set of container reality identifiers in the container sorting. Figure 9 illustrates different examples of container sorting (column or row, discrete or continuous). The information associated with a set of container sub-regions can include the start position of the container sub-region, the sorting of the sub-regions in the container (column / row, discrete / continuous), a set of container sub-region reality identifiers in the container sub-region sorting, and the 3D model name or identifier of the container. As used herein, "reality identifier" or more generally "identifier" is an identifier used to map a virtual / digital object or object part to the corresponding real-world object or object part.

[0135] In one example, the static data is combined with the layout of the user-defined map regions and the layout of the user-defined containers to provide a container list. The container list can include information associated with a set of containers, which can include the container reality identifier of the container, the location and orientation of the container, the object model identifier of the container, the identifier of the region where the container is arranged, and a set of container sub-regions (container sub-region reality identifiers).

[0136] In one example, the dynamic data in Diagram 400 can include information associated with a set of objects (e.g., devices) and a set of connections (e.g., logical and / or functional connections). The information associated with a set of objects (devices) can include a container reality identifier, a container sub-region reality identifier, the 3D model name of the object, the group name of the objects (e.g., console, server, router, switch, etc.), and the object name (e.g., name, prefix). The information associated with a set of connections (of the objects) can include the starting point of the connection, the ending point of the connection, and the group name associated with the connection (e.g., circuit name). In one example, {object identifier, point of interest identifier} is used to define a point, where the point of interest reality identifier is optional and can be a logical identifier or a physical location.

[0137] In one example, a container list is combined with a user-defined point of interest layout and dynamic data to provide a set of real-time digital twin data (DT data). The real-time digital twin data can include information associated with a set of containers and a set of connections. The information associated with a set of containers can include the container reality identifier of the container, the location and orientation of the container, the object model identifier of the container, the region identifier associated with the region where the container is located, and a set of objects or devices in the container (which may further include connections in the objects). The information associated with a set of connections can include the starting point of the connection, the ending point of the connection, and the group name associated with the connection. In one example, {container identifier - sub-region (row, column), point of interest identifier} is used to define a point.

[0138] In one example, the real-time digital twin data and an object model list are combined to form a digital twin. The object model list can include information associated with a single digital map of the environment and a set of models (including the name of the model, the 3D data of the model, and a set of user-defined points of interest associated with the model). In one example, for the formed digital twin, the 3D digital map of the object model list is placed in a virtual space. For each object of the real-time digital twin data, a model with the same name as its 3D model name is searched in the object model list, and its 3D data is placed in the space - sub-region (row, column) occupied by the container identifier. For each connection of the real-time digital twin data, the location of the point of interest is determined by the space occupied by the container identifier - sub-region (row, column) and the point of interest identifier, and the logical correlation between the points of interest is displayed or shown in a 2D or 3D diagram.

[0139] A digital twin can be presented to a user wearing an extended reality device with an extended reality application installed thereon. In one example, the extended reality application can "extract" some digital twin data based on an object name, a container reality identifier, and / or a region identifier. In one example, if dynamic data, static data, and / or user input change, an updated digital twin is created and "pushed" to the extended reality device on which the extended reality application is installed. The digital twin can be used to visualize data associated with objects in the environment.

[0140] Figure 10A and Figure 10B illustrates a digital twin representation for an object / pathfinding application in one example. A user of an extended reality device with an extended reality application installed thereon can interact with the device to use the digital twin to navigate the environment. The digital twin can assist the user in visualizing a physical travel path in the environment to reach a target destination (e.g., a target server in a data center environment).

[0141] Figures 11A to 11C illustrates a digital twin representation for a data visualization application in one example. A user of an extended reality device with an extended reality application installed thereon can interact with the device in the environment to use the digital twin to read the performance data of the device. The digital twin can assist the user in visualizing the logical connections between objects in the environment.

[0142] Figure 12 illustrates a method 1200 for using a digital twin representation in a mixed reality human-computer interaction application in one example. The method steps in method 1200 can be executed at an extended reality device with an extended reality application installed thereon.

[0143] Method 1200 begins at step 1202, where a 3D model of a container and devices in / on the container are combined into M (where M refers to a digital model of an environment with the container and the devices). Then, in step 1204, a function P is determined such that P(M) corresponds to reality. This helps in coordinate transformation between the digital twin of the physical environment and the physical environment. In step 1206, the hand / finger position of a user wearing the extended reality device in the environment is detected (e.g., by the extended reality device). In step 1208, it is determined whether the detected hand / finger position of the user corresponds to an interest point on the device. If it is determined in step 1208 that the detected hand / finger position of the user corresponds to an interest point on the device, this indicates that the device is being manipulated, and thus the method proceeds to step 1210 to update the backend system database. If it is determined in step 1208 that the detected hand / finger position of the user does not correspond to an interest point on the device, then method 1200 returns to step 1206.

[0144] Figure 13Schematic diagram of a system 1300 according to an embodiment of the present invention. The system 1300 includes three main groups of processors 1302, 1304, and 1306. The system 1300 can be considered a more detailed representation of the illustration 400.

[0145] The processor 1302 is part of an augmented reality headset installed with an augmented reality application. The processor 1302 includes a data visualization module for implementing data visualization, a rendering module for rendering a digital twin, a spatial mapping model for mapping between virtual and real environments, a calibration model for calibrating the virtual space with reference to the real space, a tracking module for determining the position of the augmented reality headset relative to the environment, and a human-machine interaction (HMI) module for detecting human-machine interaction or processing data associated with the interaction between the user and the augmented reality headset in the environment. The processor 1302 can be operably connected to a memory, a display, a power source (e.g., a battery), sensors (e.g., a depth sensor, an image sensor, a motion sensor, etc.).

[0146] The processor 1304 is part of a data processing system arranged to prepare digital twin data. The processor 1304 includes a user interface module arranged to receive data associated with a physical environment (e.g., a data center environment) and a representation of an object in the physical environment, an object model database generation module for generating an object model database, the generated object model database, and a digital twin data creation module arranged to create digital twin data (which is used for transmission to the processor 1302 for further processing).

[0147] The processor 1306 is part of a backend system database. The backend system database can include a database storing data associated with the physical environment and objects in the physical environment and a data conversion module for converting data received from the processor 1302.

[0148] Figure 13 The operation of the system 1300 is substantially the same as that of the illustration 400 in Figure 4 and will not be elaborated herein.

[0149] Figure 14FIG. 0 shows an example information processing system 1400 of an embodiment of the present invention, which can be used as a server, a data processing system, a database (e.g., a backend system database), or other types of information processing systems. The information processing system 1400 can be used to execute at least a part of one or more methods of the present invention. The information processing system 1400 generally includes appropriate components required to receive, store, and execute appropriate computer instructions, commands, and / or code. The main components of the information processing system 1400 are a processor 1402 and a memory 1404. The processor 1402 may include one or more of the following: one or more CPUs, one or more MCUs, one or more logic circuits, one or more Raspberry Pi chips, one or more digital signal processors (DSPs), one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or any one or more digital or analog circuits / circuits configured to interpret and / or execute program instructions and / or process signals and / or information and / or data. The memory 1404 may include: one or more volatile memories (e.g., RAM, DRAM, SRAM), one or more non-volatile memories (e.g., ROM, PROM, EPROM, EEPROM, FRAM, MRAM, FLASH, SSD, NAND, NVDIMM), or any combination thereof. Appropriate computer instructions, commands, code, information, and / or data may be stored in the memory 1404. Computer instructions for executing or facilitating the execution of the method embodiments of the present invention may be stored in the memory 1404. The processor 1402 and the memory 1404 may be integrated or separated (and operably connected). Optionally, the information processing system 1400 further includes one or more input devices 1406. Examples of the input device 1406 include: a keyboard, a mouse, a stylus, an image scanner, a microphone, a tactile / touch input device (e.g., a touch-sensitive screen), an image / video input device (e.g., a camera), etc. Optionally, the information processing system 1400 further includes one or more output devices 1408. Examples of the output device 1408 include: a display (e.g., a monitor, a screen, a projector, etc.), a speaker, headphones, a headset, a printer, an additive manufacturing machine (e.g., a 3D printer), etc. The display may include an LCD display, an LED / OLED display, or other suitable displays, which may or may not be touch-sensitive. The information processing system 1400 may further include one or more disk drives 1412, which may include one or more of the following: a solid-state drive, a hard disk drive, an optical drive, a flash drive, a tape drive, etc. A suitable operating system may be installed in the data processing system 1400, e.g., in the disk drive 1412 or the memory 1404. The memory 1404 and the disk drive 1412 may be operated by the processor 1402.Optionally, the information processing system 1400 further includes a communication device 1410 for communicating with devices such as servers, personal computers, terminals, tablets, telephones, watches, Internet of Things (IoT) devices, or other wireless computing devices. The communication device 1410 may include one or more of the following: a modem, a network interface card (NIC), an integrated network interface, an NFC transceiver, a ZigBee transceiver, a Wi-Fi transceiver, a Bluetooth transceiver, a radio frequency transceiver, a cellular (2G, 3G, 4G, 5G, above 5G, etc.) transceiver, an optical port, an infrared port, a USB connection, or other wired or wireless communication interfaces. The transceiver may be implemented by one or more devices (integrated transmitter and receiver, separate transmitter and receiver, etc.). The communication link may be wired or wireless and is used to transmit commands, instructions, information, and / or data. In one example, the processor 1402, the memory 1404 (and optionally the input device 1406, the output device 1408, the communication device 1410, and the disk drive 1412, if present) are directly or indirectly interconnected with each other via a bus, a peripheral component interconnect (PCI) such as PCI Express, a universal serial bus (USB), an optical bus, or other similar bus structures. In one embodiment, at least some of these components may be wirelessly connected, for example, via a network such as the Internet or a cloud computing network. Those skilled in the art will understand that... Figure 14 The information processing system 1400 shown in [reference] is merely exemplary and in other embodiments, the information processing system 1400 may have different configurations (e.g., including additional components, having fewer components, etc.).

[0150] Although not necessary, the embodiments described with reference to the accompanying drawings may be implemented as an application programming interface (API) or a series of libraries for developers or may be included in another software application (the other software application operates in an operating system such as a terminal or a computer operating system or a portable computing device operating system). Generally, since program modules include routines, programs, objects, components, and data files that help execute specific functions, those skilled in the art will understand that the functions of a software application may be distributed among multiple routines, objects, and / or components to achieve the same functions described herein.

[0151] It should also be understood that in cases where the methods and systems of the embodiments of the present invention are fully or partially implemented by a computing system, any suitable computing system architecture may be utilized, including stand-alone computers, network computers, dedicated or non-dedicated hardware devices, etc. Herein, the terms "computing system" and "computing device" (or similar terms) are intended to include, but are not limited to, a computer or information processing hardware (any suitable arrangement) capable of implementing the described functions.

[0152] In some embodiments, the systems and methods of the present invention may help accurately identify the location of an object in a dynamic and complex environment through digital twins. In one example, the location of a server to be maintained can be easily found. In some embodiments, the systems and methods of the present invention can achieve the visualization of digital correlations between different objects in an environment through digital twins. In one example, a network diagram associated with different objects / devices in the environment can be visualized. In some embodiments, the systems and methods of the present invention can achieve the automatic update of a backend system based on a mixed reality human-machine interface through digital twins. In one example, the network diagram associated with a device in the environment can be easily updated without user intervention after a manual operation on the device.

[0153] Those skilled in the art will understand that different changes and / or modifications can be made to the present invention as shown in specific embodiments to provide other embodiments of the present invention. Therefore, the described embodiments of the present invention should be considered illustrative rather than restrictive in all aspects. Examples of optional features of some aspects of the present invention are set forth in the Summary of the Invention section above. Some embodiments of the present invention may include one or more of these optional features (some of which are not specifically shown in the drawings). Some embodiments of the present invention may lack one or more of these optional features (some of which are not specifically shown in the drawings). One or more features in one embodiment and one or more features in another embodiment can be combined to provide further embodiments of the present invention. The data processing method of the present invention can be generally executed online in real time or offline. Some embodiments of the present invention do not limit the physical environment to a data center environment.

Claims

1. A computer-implemented method for facilitating the generation of a digital twin of a physical environment for an extended reality application, the computer-implemented method comprises: (a) receiving first object model data associated with a representation of a physical environment and second object model data associated with representations of one or more objects in the physical environment; (b) receiving real-time data associated with the arrangement, structure, function, operating state, and / or performance of each of the one or more objects in the physical environment at a certain moment; the real-time data includes static data stored in a backend system database and dynamic data, wherein the static data includes data associated with the physical environment and one or more objects in the physical environment that generally remains relatively stable or unchanged over time; the dynamic data includes data associated with the physical environment and one or more objects in the physical environment that may change over time; (b1) receiving predefined data associated with predefined features of the physical environment and / or the one or more objects; (c) generating digital twin data for the extended reality application based on the predefined data, the static data, and the dynamic data stored in the backend system database; (d) providing the digital twin data to an extended reality device installed with the extended reality application; and (d1) combining the digital twin data with a list of object models to generate a digital twin of the physical environment having the one or more objects at the certain moment; the list of object models is associated with the first object model data and the second object model data.

2. The computer-implemented method according to claim 1, wherein: the physical environment includes a data center environment; and the one or more objects include: one or more containers; and one or more devices disposed in or on the one or more containers.

3. The computer-implemented method according to claim 2, wherein: the one or more containers include at least one of the following: racks, cabinets, and frames; and / or the one or more devices include at least one of the following: servers, routers, switches, disk memories, and power supplies.

4. The computer-implemented method according to claim 1, wherein: the real-time data associated with the arrangement of each of the one or more objects includes data associated with the position and area of each of the one or more objects in the physical environment at the certain moment; the real-time data associated with the structure of each of the one or more objects includes data associated with the internal and / or external structure of each of the one or more objects in the physical environment at the certain moment; the real-time data associated with the function of each of the one or more objects includes data associated with the inter-object or intra-object logical connections of each of the one or more objects in the physical environment at the certain moment, and / or The real-time data associated with the performance of each of the one or more objects includes data associated with the historical performance of each of the one or more objects in the physical environment at the certain moment.

5. The computer-implemented method according to claim 1, wherein, the step (a) includes: (a1) Receiving the first object model data associated with the graphical representation of the physical environment and the second object model data associated with the graphical representation of the one or more objects in the physical environment.

6. The computer-implemented method according to claim 5, wherein: The first object model data associated with the graphical representation of the physical environment includes a digital map of the physical environment; and The second object model data associated with the graphical representation of the one or more objects in the physical environment includes the corresponding models of the one or more objects.

7. The computer-implemented method according to claim 1, wherein, the step (a) includes: (a2) Receiving the first object model data associated with the predefined features of the physical environment and / or the second object model data associated with the predefined features of the one or more objects in the physical environment.

8. The computer-implemented method according to claim 2, wherein: The step (a) includes: (a2) Receiving the first object model data associated with the predefined features of the physical environment and / or the second object model data associated with the predefined features of the one or more objects in the physical environment; The first object model data associated with the predefined features of the physical environment includes data associated with one or more predefined regions in the digital map of the physical environment; The second object model data associated with the predefined features of one or more objects in the physical environment includes data associated with the predefined layout of at least one of the one or more containers; and The second object model data associated with the predefined features of one or more objects in the physical environment includes data associated with one or more predefined points of interest of at least one of the one or more devices.

9. The computer-implemented method according to claim 1, wherein, the first object model data and the second object model data in the step (a) are received from an object model database.

10. The computer-implemented method according to claim 1, wherein, the object model list includes a digital map of the physical environment and the corresponding models of the one or more objects.

11. The computer-implemented method according to claim 1, wherein, the computer-implemented method further includes: (f) Presenting to the user of the extended reality device wearing and installed with the extended reality application a digital twin of the physical environment having the one or more objects at the certain moment. Wherein, the rendering of step (f) is performed when the user wearing the extended reality device is in the physical environment.

12. The computer-implemented method according to any one of claims 1 to 11, wherein, the computer-implemented method further comprises: (g) receiving updated data associated with an updated arrangement, structure, function, operating state, and / or performance of the one or more objects in the physical environment at another moment after the certain moment; (h) generating updated digital twin data for the extended reality application program based at least on the updated data; and (i) providing the updated digital twin data to the extended reality device installed with the extended reality application program to facilitate generating an updated digital twin of the physical environment at the another moment.

13. The computer-implemented method according to claim 12, wherein: the updated data corresponds to the dynamic data; data associated with the arrangement, structure, function, operating state, and / or performance of the one or more objects in the physical environment that is not updated at the another moment corresponds to the static data; and in step (h), the updated digital twin data is generated based on the static data, the dynamic data, and the first object model data and the second object model data received in step (a).

14. The computer-implemented method according to claim 12, wherein, the computer-implemented method further comprises: (j) generating the updated digital twin of the physical environment having the one or more objects at the another moment based at least on the updated digital twin data; and (k) presenting the updated digital twin of the physical environment having the one or more objects at the another moment to the user wearing the extended reality device installed with the extended reality application program; wherein, the presentation of step (k) is performed when the user wearing the extended reality device is in the physical environment.

15. The computer-implemented method according to claim 12, wherein, the computer-implemented method further comprises: repeating steps (g) to (i) based on detecting a change in the arrangement, structure, function, operating state, and / or performance of the one or more objects in the physical environment, or when detecting a change in the arrangement, structure, function, operating state, and / or performance of the one or more objects in the physical environment.

16. The computer-implemented method according to any one of claims 1 to 11, wherein, the extended reality device or the extended reality application program is operable to: assist a user wearing the extended reality device in the physical environment to navigate in the physical environment; guide a user wearing the extended reality device in the physical environment to a predetermined position in the physical environment; instruct a user wearing the extended reality device in the physical environment to perform a predetermined task; and / or Assist a user wearing the extended reality device in the physical environment to determine the structure, function, operating state, and / or performance of one or more objects in the physical environment.

17. The computer-implemented method according to any one of claims 1 to 11, wherein, the real-time data is received from a backend database that stores data associated with the physical environment and the one or more objects in the physical environment.

18. The computer-implemented method according to claim 17, wherein, the real-time data in step (b) is received in response to a user input received at the extended reality device on which the extended reality application is installed.

19. The computer-implemented method according to claim 17, wherein, the extended reality device or the extended reality application is operable to: when a user wearing the extended reality device manipulates one or more objects, facilitate an update of the backend database based on detected changes in the structure, function, operating state, and / or performance of the one or more objects.

20. The computer-implemented method according to any one of claims 1 to 11, wherein: the extended reality application is one of the following: a virtual reality application, an augmented reality application, and a mixed reality application; and the extended reality device is the corresponding one of the following: a virtual reality device, an augmented reality device, and a mixed reality device.

21. A system for facilitating the generation of a digital twin of a physical environment for an extended reality application, the system comprising: one or more processors arranged to execute or facilitate the execution of the computer-implemented method according to any one of claims 1 to 20.

Images