Extended Reality (XR) Collaborative Environments

Abstract

An exemplary embodiment of the present disclosure provides an extended reality (XR) system including an autonomous robotic device and a user interface. The autonomous robotic device may be disposed in a physical environment. The user interface may be configured to display an XR environment corresponding to at least a portion of the physical environment and to receive input from a user based on the user's perception of the XR environment. The autonomous robotic device may be configured to perform an autonomous operation based at least in part on the input received from the user.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Application No. 63 / 234,452, filed August 18, 2021, which is incorporated by reference in its entirety as if fully set forth below.

[0002] (Technical field) Various embodiments of the present disclosure relate generally to extended reality (XR) collaborative systems. [Background technology]

[0003] Manufacturing operations in natural spaces are challenging and require significant human capital to perform the necessary tasks. The value of human capital in manufacturing operations is the ability to accommodate the natural variability of the raw materials involved, especially within these natural spaces. Traditionally, workers performing manufacturing roles are not able to work remotely and must be physically present at the manufacturing facility to perform work-related tasks. Previous attempts to enable remote work for manufacturing workers have been unsuccessful as systems have proven challenging to program and implement. Thus, there is a need to provide a collaborative environment between humans and autonomous robotic devices to address the aforementioned issues present when performing manufacturing operations in natural spaces. Summary of the Invention

[0004] An exemplary embodiment of the present disclosure provides an extended reality (XR) system including an autonomous robotic device and a user interface. The autonomous robotic device may be disposed in a physical environment. The user interface may be configured to display an XR environment corresponding to at least a portion of the physical environment and to receive input from a user based on the user's perception of the XR environment. The autonomous robotic device may be configured to perform an autonomous operation based at least in part on the input received from the user.

[0005] In any of the embodiments disclosed herein, the autonomous robotic device may be further configured to perform autonomous operations using machine learning algorithms.

[0006] In any of the embodiments disclosed herein, the machine learning algorithm may be trained using data points representing the physical environment and inputs based on the user's perception in the XR environment.

[0007] In any of the embodiments disclosed herein, the machine learning algorithm may be further trained with data points indicative of a success score of the autonomous operation.

[0008] In any of the embodiments disclosed herein, the autonomous robotic device may be configured to request that a user of the XR system provide input.

[0009] In any of the embodiments disclosed herein, the autonomous robotic device may be configured to request that a user of the extended reality system provide input if the robotic device is unable to perform an autonomous operation using a machine learning algorithm without user input.

[0010] In any of the embodiments disclosed herein, the user interface may be configured to receive input from a user via a network interface.

[0011] In any of the embodiments disclosed herein, the XR system may further include one or more sensors configured to monitor at least one discrete data value in the physical environment, and the user interface may be further configured to display the XR environment based at least in part on the at least one discrete data value.

[0012] In any of the embodiments disclosed herein, the XR system may further include user equipment that may be configured to allow a user to interact with the user interface.

[0013] In any of the embodiments disclosed herein, the user equipment may include a head mounted display (HMD) that may be configured to display the XR environment to the user.

[0014] In any of the embodiments disclosed herein, the user equipment may include a controller that may be configured to enable a user to provide input based on the user's perception in the XR environment.

[0015] In any of the embodiments disclosed herein, the user interface may be further configured to monitor manipulation of the controller by a user and alter the display of the XR environment based on the manipulation.

[0016] Another embodiment of the present disclosure provides a method for operating an autonomous robotic device disposed in a physical environment using an extended reality (XR) system. The method may include displaying an XR environment in a user interface corresponding to at least a portion of the environment, receiving input from a user based on a perception of the user in the XR environment, and causing the robotic device to perform an autonomous action based at least in part on the input received from the user.

[0017] In any of the embodiments disclosed herein, the method may further include causing the autonomous robotic device to perform the autonomous operation using a machine learning algorithm.

[0018] In any of the embodiments disclosed herein, the method may further include training a machine learning algorithm with the data points representative of the physical environment and the input received from the user based on the user's perception of the XR environment.

[0019] In any of the embodiments disclosed herein, the method may further include further training the machine learning algorithm with the points indicative of a success score of the autonomous operation performed by the autonomous robotic device.

[0020] In any of the embodiments disclosed herein, the method may further include requesting a user of the XR system to provide input.

[0021] In any of the embodiments disclosed herein, the method may further include requesting a user of the XR system to provide input if the autonomous robotic device is unable to perform the autonomous operation using the machine learning algorithm without user input.

[0022] In any of the embodiments disclosed herein, input may be received from a user via a network interface.

[0023] In any of the embodiments disclosed herein, the method may further include monitoring input provided by the user by one or more other users interacting with the XR environment.

[0024] In any of the embodiments disclosed herein, the method may further include the user interacting with the user interface using a user device.

[0025] In any of the embodiments disclosed herein, the method may further include the user equipment comprising a head mounted display (HMD), and displaying the XR environment to the user on the HMD.

[0026] In any of the embodiments disclosed herein, the method may further include a user generating an input with the controller based on the user's perception of the XR environment.

[0027] In any of the embodiments disclosed herein, the method may further include monitoring manipulation of the controller by a user and altering a display of the XR environment based on the manipulation of the controller.

[0028] These and other aspects of the present disclosure are described below in the Detailed Description of the Invention and in the accompanying drawings. Other aspects and features of the embodiments will become apparent to those skilled in the art upon review of the following description of certain exemplary embodiments in conjunction with the drawings. Although features of the present disclosure may be described in conjunction with certain embodiments and drawings, all embodiments of the present disclosure may include one or more of the features described herein. Furthermore, although one or more embodiments may be described as having certain advantageous features, one or more of such features may also be used with various embodiments described herein. Similarly, although exemplary embodiments may be described below as device, system, or method embodiments, it should be understood that such exemplary embodiments may be implemented in various devices, systems, and methods of the present disclosure.

[0029] The following detailed description of certain embodiments of the present disclosure will be better understood when read in conjunction with the accompanying drawings. For the purpose of illustrating the present disclosure, certain embodiments are shown in the drawings. It will be understood, however, that the disclosure is not limited to the precise arrangements and instrumentalities of the embodiments shown in the drawings. [Brief description of the drawings]

[0030] [Figure 1] FIG. 1 is an illustrative diagram showing how a user provides input to a user interface via a user device in an extended reality (XR) environment, resulting in an autonomous action being performed by an autonomous robotic device, according to an exemplary embodiment of the present disclosure.

[0031] [Diagram 2]FIG. 2 is an illustration showing how sensors monitor at least one discrete data value in a physical environment to at least partially assist in constructing an XR environment that is displayed to a user via a user interface, according to an example embodiment of the present disclosure.

[0032] [Diagram 3] FIG. 3 is a diagram illustrating how a user interacts with a user interface via a user device according to an exemplary embodiment of the present disclosure.

[0033] [Figure 4] FIG. 4 is a flowchart of an example process for causing an autonomous robotic device to perform an autonomous operation using an XR environment, according to an exemplary embodiment of the present disclosure. [Diagram 5] FIG. 5 is a flowchart of an example process for causing an autonomous robotic device to perform an autonomous operation using an XR environment, according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] In order to facilitate the understanding of the principles and features of the present disclosure, various exemplary embodiments are described below. The components, steps, and materials described below as constituting various elements of the embodiments disclosed herein are intended to be illustrative and not limiting. Many suitable components, steps, and materials that will perform the same or similar functions as the components, steps, and materials described herein are intended to be encompassed within the scope of the present disclosure. Such other components, steps, and materials not described herein include, but are not limited to, similar components or steps developed after the development of the embodiments disclosed herein.

[0035] Also, it should be noted that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. For example, a reference to a component is intended to include compositions of multiple components. A reference to a composition containing "a" component is intended to include the other components in addition to the specified component.

[0036] Additionally, in describing the exemplary embodiments, terminology is employed for the sake of clarity. Each term is intended to have its broadest meaning as understood by one of ordinary skill in the art and is intended to encompass all technical equivalents that operate in a similar manner to accomplish a similar purpose.

[0037] "Comprising," "containing," or "including" means that at least the specified compounds, elements, particles, or method steps are present in a composition, article, or method, but does not exclude the presence of other compounds, substances, particles, or method steps if such other compounds, substances, particles, or method steps have the same function as the one specified.

[0038] It should also be understood that the recitation of one or more method steps does not exclude the presence of additional or intervening method steps between the steps expressly identified.Similarly, it should also be understood that the recitation of one or more components in a composition does not exclude the presence of additional components other than those expressly identified.

[0039] The materials described as constituting various elements of the present invention are intended to be exemplary and not limiting. Many suitable materials that would perform the same or similar functions as the materials described herein are intended to be encompassed within the scope of the present invention. Such other materials not described herein include, for example, but are not limited to, materials developed after the development of the present invention.

[0040] In particular, in the biological world, autonomous robots do not exist. Thus, there is a need for tools that can provide easy and seamless collaboration between humans and robotic devices to assist in manufacturing operations, especially within natural spaces. The tools described herein allow for benefits such as remotely operating machines to perform tasks and increase productivity within manufacturing facilities, as opposed to the significant costs of increasing human capital.

[0041] A collaborative extended reality (XR) system (100) may include an autonomous robotic device (300) configured to perform autonomous operations, a user interface (600) configured to display an XR environment, a user equipment (400) configured to display the user interface (600) and enable a user (200) to interact with the user interface (600), and one or more sensors (500) configured to monitor at least one discrete data value in a physical environment.

[0042] For ease of explanation, the XR system (100) is described in the context of its application in the poultry production industry. However, the present disclosure is not so limited. Rather, as will be appreciated by those skilled in the art, the XR system (100) disclosed herein may find many uses in a variety of applications where it may be desirable to provide user input to assist in task completion. In the poultry production industry, secondary and further processing operations require significant human worker participation. Generally, tasks are classified as either high-motion tasks, including moving the entire product or portions thereof from machine to machine, or fine tasks, including cutting and properly stacking ingredients for packaging, which require more anatomical knowledge and manual dexterity. Using the XR system (100) described herein, a user (200) can provide inputs via a user interface (600) to an autonomous robotic device (300) to perform autonomous actions corresponding to high-motion or fine tasks in a poultry production facility.

[0043] As one skilled in the art will appreciate, an autonomous robotic device (300) is a different class of device than a telerobotic device. Specifically, an autonomous robotic device (300) differs from a telerobotic device in that an autonomous robotic device (300) does not require user input to control each aspect of the operations it performs, whereas a telerobotic device is directly controlled by a user. Similarly, an autonomous operation performed by an autonomous robotic device (300) is an operation that takes into account, but is not identical to, instructions / input received from a user (200). For example, in a poultry production application, an autonomous robotic device (300) performing an autonomous operation may load natural raw materials onto a cone that moves through an assembly line. User input may specify where the autonomous robotic device (300) should grab the natural raw materials, but the autonomous robotic device (300) may determine a path to move the natural raw materials from its current location to the cone independently of user input. In other words, the user (200) provides input that the autonomous robotic device (300) uses to determine where to grab the natural ingredients, but the robot makes additional decisions autonomously to move the natural ingredients to the cone.

[0044] A user (200) of the XR system (100) may use a user equipment (400) to provide input that enables the autonomous robotic device (300) to perform autonomous actions. The user equipment (400) may include many different components known in the art. For example, in some embodiments, the user equipment (400) may include a controller (420) and / or a head mounted display (HMD) (410) to enable the user (200) to interact with the user interface (600). In some embodiments, for example, the HMD (410) may include, but is not limited to, a brain implant for visualizing a see-through display, an immersive display helmet, or the like. FIG. 1 illustrates how a user (200) uses a controller (420) of a user equipment (400) in an XR system (100) to specify where to grab a natural raw material (e.g., where / how to grab a poultry). This input can be provided to an autonomous robotic device (300) to perform an autonomous action with the natural raw material (e.g., grab a poultry and move it to a desired location). Natural variability exists in the conditions within the task performed by the autonomous robotic device (300), but collaboration with the user (200) viewing the XR environment using the user equipment (400) allows the autonomous robotic device (300) to appropriately respond to real-time novel situations. This result can occur by the user (200) interacting with the XR environment and providing input to a user interface (600) via the user equipment (400) to guide the autonomous action performed by the autonomous robotic device (300) using the user's prior experience and situational awareness. As one skilled in the art will appreciate, examples of user equipment (400) that a user (200) can use to interact with a user interface (600) include, but are not limited to, an Oculus Quest II or a Meta Quest II.

[0045] Within the XR system (100), a user interface (600) aggregates discrete data sets from one or more sensors (500). As one skilled in the art can appreciate, there are many different types of sensors that can be configured to monitor discrete data values ​​in a physical environment. Examples of different types of sensors can include, but are not limited to, temperature sensors, light sensors, vibration sensors, motion sensors, color sensors, etc. Within the XR system (100), one or more sensors (500) can monitor discrete data sets in the physical environment, and the discrete data sets can be aggregated by a user interface (600). The user interface can construct an XR environment for a user (200) that is based at least in part on the discrete data sets and that at least in part corresponds to the physical environment.

[0046] 2 illustrates an XR system (100) including an autonomous robotic device (300) located within a physical environment, one or more sensors (500) that monitor at least one discrete dataset in the physical environment, and a user interface (600) that displays an XR environment constructed at least in part by the discrete dataset monitored by the one or more sensors (500). In addition to contributing to the construction of the XR environment, the discrete dataset monitored by the one or more sensors (500) of the XR system (100) can also assist a user (200) in making decisions in the XR environment and interacting with the user interface (600).

[0047] A user (200) can use a user equipment (400) to interact with a user interface (600) that displays an XR environment built in part based on a discrete data set monitored by one or more sensors (500). The user interface (600) can be displayed to the user (200) via an HMD (410) of the user equipment (400). As can be appreciated by those skilled in the art, using the HMD (410) to display the user interface (600) and the XR environment thereon can aid the user's (200) perception when interacting with the user interface (600). Additionally, by using the HMD (410), the user (200) can determine input points that can be provided to the user interface (600) via the controller (420). Inputs provided by the user (200) can be received by the user interface (600) and transmitted to the autonomous robotic device (300) via a network interface. As can be appreciated by those skilled in the art, a network interface can be a medium of interconnection between two devices separated by a large physical distance. Examples of interconnection media relevant to the preferred application can include, but are not limited to, cloud-based networks, wired networks, wireless (Wi-Fi) networks, Bluetooth networks, etc.

[0048] FIG. 3 illustrates how a user (200) with a HMD (410) and controller (420) of a user equipment (400) navigates through an XR environment and interacts with a user interface (600). In some embodiments, the user (200) can provide a place to hold natural raw materials in a user interface (600) that can send inputs via a network interface to an autonomous robotic device (300) located at a large physical distance from the user (200), causing the autonomous robotic device (300) to perform an autonomous action, such as placing the natural raw materials on a cone device. In some embodiments, the user (200) can use the user equipment (400) to provide multiple inputs to the autonomous robotic device (300) through the user interface (600) and provide guidance for longer processes. For example, if the user (200) is monitoring a process that requires multiple inputs to the user interface (600) over time to cause the autonomous robotic device (300) to perform an autonomous action, the claimed invention can assist in allowing multiple inputs. An example of a user (200) providing multiple inputs during a process may be applied in situations including, but not limited to, commercial baking ovens, agricultural production operations, and the like. For example, a commercial baking oven is loaded with kneaded dough that exhibits several characteristics over time, such as leavening. The dough is then baked as it passes through the oven. The color of the dough is monitored to meet specifications. Due to natural variability in wheat-yeast mixtures, oven parameters may need to be manipulated throughout the process to achieve a desired result. These parameters include, but are not limited to, temperature, residence time, humidity, and the like. The ability to manipulate oven parameters throughout the baking process may be achieved using the XR system (100) described herein. The XR system (100) may enable a user (200) to provide multiple inputs to an autonomous robotic device (300) that can perform multiple autonomous actions during the process.Additionally, the XR system (100) may enable a user (200) to be “on the line” while a process is being performed, allowing the user (200) to provide multiple required inputs in real time, and may enable the user (200) to monitor parts of a process in a physical environment that may be impractical or dangerous for humans.

[0049] 4 shows a method flowchart (700) illustrating a method for a user (200) to provide input to an XR system (100) to enable an autonomous robotic device (300) to perform an autonomous operation. The method may include initializing the XR system (100) and displaying (710) an XR environment constructed at least in part based on a discrete data set monitored by one or more sensors (500), which may correspond to at least a portion of a physical environment. The method may further include receiving (720) an input from a user via a network interface based on the user's (200) perception of a user interface (600) with a user equipment (400), the user equipment (400) including an HMD (410) for displaying the user interface (600) to the user (200) and a controller (420) for generating input in the XR environment. The method may further include causing (730) the autonomous robotic device (300) to perform an autonomous action based at least in part on the input received from the user (200).

[0050] The autonomous robotic device (300) of the XR system (100) may also be configured to perform autonomous operations using machine learning algorithms without input from the user (200). This configuration is desirable because it can increase productivity within manufacturing operations, particularly because it can allow the autonomous robotic device (300) to perform repetitive tasks with high efficiency while accounting for the natural variability of natural raw materials. In a preferred application, the natural variability may include, but is not limited to, determining the gripping position of the natural raw material for large-motion tasks, or modifying the anatomical representation of the raw material for fine tasks. As will be appreciated by those skilled in the art, machine learning algorithms are a subfield within artificial intelligence (AI) that enable computer systems and other related devices to learn how to perform tasks and improve their performance in performing the tasks over time. Examples of types of machine learning that may be used include, but are not limited to, supervised learning algorithms, unsupervised learning algorithms, semi-supervised learning algorithms, reinforcement learning algorithms, and the like. In addition to the aforementioned examples, other algorithms, such as deterministic and statistical algorithms that are not solely based on machine learning or AI, may also be used to cause the autonomous robotic device (300) to perform autonomous operations. In some embodiments, if the machine learning algorithm cannot complete an autonomous operation due to natural variability of the raw materials, the XR system (100) can request the user (200) to provide input to the user interface (600), which is sent to the autonomous robotic device (300) via the network interface to enable the autonomous operation. This collaboration by requesting the user (200) to provide input to the autonomous robotic device (300) to complete the autonomous operation can be advantageous because it allows the user (200) to further train the autonomous robotic device (300) in addition to immediately performing the intended autonomous operation. The input provided by the user (200) to the autonomous robotic device (300) can also be used to aid in the development of the specific autonomous operation application used by the autonomous robotic device (300).

[0051] FIG. 5 shows a method flowchart (800) illustrating how an autonomous robotic device (300) can perform an autonomous operation using a machine learning algorithm. The method (810) can include initializing an XR system (100) and displaying an XR environment constructed at least in part based on a discrete data set monitored by one or more sensors (500), which can correspond to at least a portion of a physical environment. The method can further include having the autonomous robotic device (300) perform an autonomous operation using a machine learning algorithm (820), which can be trained based on data points representative of the physical environment, historical input data from a user (200) based on the user's perception of the XR environment, and data points indicating a success score of the autonomous operation performed by the autonomous robotic device (300). The method can further include the XR system (100) requesting (830) the user (200) to provide input to a user interface (600) if the autonomous robotic device (300) cannot perform the autonomous operation using the machine learning algorithm. For example, if the autonomous robotic device (300) determines a predicted success score for an autonomous operation, the XR system (100) may request that the user (200) provide input, thereby increasing the likelihood that the autonomous operation will be completed successfully.

[0052] It is to be understood that the embodiments and claims disclosed herein are not limited in their application to the details of construction and arrangement of the components described herein and illustrated in the drawings. Rather, the specification and drawings provide examples of possible embodiments. The embodiments and claims disclosed herein are capable of further embodiments and can be practiced and carried out in various ways. It is also to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be construed as limiting the scope of the claims.

[0053] As such, those skilled in the art will appreciate that the conception underlying the present application and claims may be readily utilized as a basis for the designing of other structures, methods and systems for carrying out the purposes of the embodiments and claims presented herein, and it is important that the claims be regarded as including such equivalent constructions.

[0054] Furthermore, the purpose of the Abstract is to enable the U.S. Patent and Trademark Office and the general public, including those not familiar with patent and legal terminology and language, to quickly grasp the content and gist of the technical disclosure of the application upon a single reading. The Abstract does not define the scope of the claims of the application, nor does it limit the scope of the claims in any way.

Claims

1. an autonomous robotic device positioned at a device location in the physical environment; a user interface configured to construct an extended reality (XR) environment corresponding to at least a portion of the physical environment, receive input based on a user's perception of the XR environment from a user at a user location interacting with the XR environment, and transmit the input to the autonomous robotic device; the user location is remote from the device location; the user interface is further configured to transmit the input with a delay within a range; The autonomous robotic device is configured to perform an autonomous action based at least in part on input transmitted by the user interface.

2. The XR system of claim 1, wherein the user location is physically separated from the device location.

3. The input is transmitted to the autonomous robotic device via a network interface; 3. The XR system of claim 2, wherein the device location and the user location are physically separate such that the user's perception of the physical environment is solely through the XR environment.

4. 10. The XR system of claim 1, wherein the autonomous robotic device is further configured to perform the autonomous movement using a machine learning algorithm.

5. 5. The XR system of claim 4, wherein the machine learning algorithm is trained using data points representing the physical environment, data points indicative of a success score of the autonomous operation, and input received from the user based on the user's perception of the XR environment.

6. 10. The XR system of claim 1, wherein the autonomous robotic device is further configured to request the user of the XR system to provide input if the autonomous robotic device is unable to perform the autonomous operation using a machine learning algorithm without user input.

7. 10. The XR system of claim 1, wherein the user interface is further configured to receive input from the user via the network interface to enable one or more additional users to interact with the XR environment.

8. further comprising one or more sensors at the device location configured to monitor at least one discrete data value in the physical environment; The XR system of claim 1 , wherein the user interface is further configured to display the XR environment based at least in part on the at least one discrete data value.

9. The XR system of claim 1 , further comprising a user device configured to enable the user to interact with the XR environment.

10. The user interface is further configured to display the constructed XR environment; 10. The XR system of claim 9, wherein the user equipment comprises a head-mounted display (HMD) configured to display the XR environment to the user.

11. The XR system of claim 10 , wherein the user equipment further comprises a controller configured to enable the user to provide the input.

12. The XR system of claim 11 , wherein the user interface is further configured to monitor manipulations of the controller by the user and to alter a display of the XR environment based on one or more monitored manipulations.

13. 13. A method of operating an autonomous robotic device located in a physical environment using an XR system according to any one of claims 1 to 12, comprising: constructing an XR environment corresponding to at least a portion of the physical environment of a deployed autonomous robotic device; receiving input from a user at a user location interacting with the XR environment based on the user's perception of the physical environment solely through the XR environment; transmitting the input to the autonomous robotic device via a network interface that provides a medium for interconnection between the user location and the autonomous robotic device separated by a large physical distance; causing the autonomous robotic device to perform an autonomous action based at least in part on the transmitted input.

14. The performing step further comprises, at least in part, using a machine learning algorithm; 14. The method of claim 13, wherein the machine learning algorithm was trained using data points representing the physical environment, data points indicative of a success score for the autonomous operation, and input received from the user based on the user's perception of the XR environment.

15. requesting the user of the XR system to provide input if the autonomous robotic device is unable to perform the autonomous action using a machine learning algorithm without user input; Displaying the constructed XR environment to the user on a head-mounted display (HMD); monitoring at least one discrete data value in the physical environment with one or more sensors at the device location; the displayed XR environment is based at least in part on the at least one discrete data value; the user interacting with the XR environment using a user device; generating said input by said user on a controller; The method of claim 13 , further comprising: altering a display of the XR environment based on the monitored manipulation.

Images