Determining the extent of a virtual reality (VR) environment displayed on a VR device

Abstract

According to one approach, a computer-implemented method includes identifying machines involved in performing a manufacturing process at a manufacturing location and identifying a workflow sequence for the machines to perform. Conditions associated with remote operators remotely controlling the machines to perform the workflow sequence for execution at the manufacturing location using virtual reality (VR) devices are received. The method further includes determining, for each of the VR devices, a range of a VR collaborative environment to display. The range is determined based on the condition, thereby reducing latency in performing the workflow sequence for execution at the manufacturing location. The method further includes outputting the range to the VR devices.

Description

[Technical Field]

[0001] The present invention relates to virtual reality (VR), and more particularly, the present invention relates to determining the extent of a VR environment to display on a VR device configured to control a machine that performs operations in a workflow sequence.

[0002] VR is a three-dimensional computer-generated environment that a person can explore and interact with. The person becomes part of and / or immerses themselves in the virtual world, and while there, can manipulate objects and / or perform a series of actions. VR benefits the manufacturing industry by improving worker safety, improving product production, saving manufacturers money, enabling remote control of machine operation, enabling monitoring of various machines on the industrial floor, and the like. More specifically, in some cases, VR allows operators to remotely control and monitor machine operation, e.g., start, stop, change operating parameters, etc., by using VR devices from one or more remote locations.

[0003] On any industrial floor, there may be different machines located and operating in different portions of the floor, where the machines may cooperate with one another to accomplish a common task or to implement any manufacturing workflow. In some contexts, a "workflow" may be defined as one or more operations performed to accomplish a predetermined task. More specifically, such a task may be realized as a result of one or more machines performing machine operations according to a "workflow sequence," in which one or more machines perform machine operations in parallel and / or sequentially in an interconnected manner. In some cases, each machine may be assigned a specific work task to perform, while in some other cases, multiple machines may additionally and / or alternatively work together to perform a single machine operation. On any industrial floor, the relative positions and orientations of these machines may be monitored and controlled to align with the machine's workflow sequence.

[0004] A human, hereinafter referred to as a “remote operator,” may remotely control a machine to perform machine operations using a VR device. For example, the remote operator may wear known types of VR glasses to view the machine's robotic arm and control the robotic arm using a handheld control. While remotely controlling, monitoring, or operating any machine on any industrial floor, the remote operator may visualize the target machine in a respective “VR environment” and perceive himself or herself as actually being on the industrial floor where the machine resides. At the same time, various other remote operators may also remotely control, monitor, or operate other machines in the surrounding area. In this case, different operators may have different operating speeds, levels of Internet bandwidth, experience in controlling machines using a VR device, etc. Thus, in at least some implementations in which instructions for machines to execute in a workflow sequence are remotely submitted by an operator using a VR device, the timing of execution of the instructions on each machine is important, for example, to prevent the machines from performing the operations of the workflow sequence in a different order. During VR interactions between multiple operators, a machine may encounter problems properly executing commands in a workflow sequence due to one or more types of latency associated with the condition of the remote operators. At the same time, latency in submitting or executing commands also impacts productivity in the context of a workflow sequence. Therefore, techniques are needed to mitigate latency in submitted commands. Furthermore, techniques are needed to enable execution of machine commands in a synchronized manner even when latency exists in multiple received machine commands, for example, as a result of one or more VR devices issuing machine commands with relatively low Internet bandwidth. Summary of the Invention

[0005] According to one approach, a computer-implemented method includes identifying machines involved in performing a manufacturing process at a manufacturing location and identifying a workflow sequence for the machines to perform. The relative scope of a task performed by a particular one of the machines in the manufacturing process is recognized based on their identification and how this scope relates to operations performed by one or more of the other machines in the workflow sequence for execution. Conditions related to a remote operator remotely controlling the machines to perform the workflow sequence for execution at the manufacturing location using a virtual reality (VR) device are received. The relative internet bandwidth of the VR device is one such condition, and by taking such conditions into account, the scope of the VR collaborative environment displayed by the VR device can be adjusted so that the condition does not introduce latency into the workflow sequence for execution. To prevent such latency, the scope of the VR collaborative environment to display is determined for each VR device. The scope is determined based on the condition, thereby reducing latency in performing the workflow sequence for execution at the manufacturing location. The scope is output to the VR device. This allows the remote operator to operate and control the machine remotely as if the remote operator were physically working at the manufacturing site, without the remote operation having to incur the potential dangers of being physically present at the manufacturing site.

[0006] Determining the extent of the VR collaborative environment includes determining a first machine remotely controlled by a first VR device having a relatively low internet bandwidth and determining a second machine remotely controlled by a second VR device having a relatively high internet bandwidth. As a result of determining such relative differences in internet bandwidth, the extent of the VR collaborative environment determined and output for display by the first VR device includes relatively less clarity and / or content than the extent of the VR collaborative environment determined and output for display by the second VR device. Thus, the extent of the VR collaborative environment determined and output for display by the first VR device reduces the likelihood that the first VR device will introduce latency into the workflow sequence of execution based on the relatively low internet bandwidth. Reducing latency in this manner improves the performance of computing devices used in carrying out the workflow sequence of execution. This also results in reduced waste in the production process at the manufacturing location because errors that would otherwise result in the production process as a result of such latency are avoided.

[0007] Another approach is to provide a computer program product that includes a computer-readable storage medium having program instructions embodied therein, the program instructions being readable and / or executable by a computer to cause the computer to perform the method described above.

[0008] Another approach is to provide a system that includes a processor and logic integrated with, executable by, or integrated with and executable by the processor, the logic configured to perform the method described above.

[0009] Other aspects and techniques of the present invention will become apparent from the following detailed description, which, when taken in conjunction with the drawings, illustrate by way of example the principles of the invention. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram of a computing environment in accordance with one approach of the present invention.

[0011] [Figure 2A-1] 1 is a flowchart of a method according to one approach of the present invention. [Figure 2A-2] 1 is a flowchart of a method according to one approach of the present invention. [Figure 2A-3] 1 is a flowchart of a method according to one approach of the present invention.

[0012] [Figure 2B] 2A is a flowchart of a portion of the operations of FIGS. 2A-1 to 2A-3 according to one method of the present invention.

[0013] [Figure 2C] 2A is a flowchart of a portion of the operations of FIGS. 2A-1 to 2A-3 according to one method of the present invention.

[0014] [Figure 3] 1 is a flowchart of a method according to one approach of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] The following description is made for the purpose of illustrating the general principles of this invention and is not meant to limit the inventive concepts claimed herein. Moreover, particular features described herein can be used in combination with other described features in each of the various possible combinations and permutations.

[0016] Unless otherwise specifically defined herein, all terms are to be given their broadest possible interpretation, including the meanings implied by this specification and the meanings understood by a person skilled in the art and / or defined in dictionaries, treatises, etc.

[0017] It should also be noted that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless otherwise specified. It will be further understood that the terms "comprises" and / or "comprising," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0018] The following description discloses several preferred approaches to systems, methods and computer program products for determining the extent of a virtual reality (VR) environment for display on a VR device configured to control a machine that performs operations in a workflow sequence.

[0019] In one general approach, a computer-implemented method includes identifying machines involved in performing a manufacturing process at a manufacturing location and identifying a workflow sequence for the machines to perform. Conditions associated with remotely controlling the machines to perform the workflow sequence for execution at the manufacturing location by remote operators using virtual reality (VR) devices are received. The method further includes determining, for each of the VR devices, a range of a VR collaborative environment to display. The range is determined based on the condition, thereby reducing latency in performing the workflow sequence for execution at the manufacturing location. The method further includes outputting the range to the VR devices.

[0020] In another general approach, a computer program product comprises a computer-readable storage medium having program instructions embodied therein, the program instructions being readable and / or executable by a computer to cause the computer to perform the method described above.

[0021] In another general approach, a system includes a processor and logic integrated with, executable by, or integrated with and executable by the processor, the logic configured to perform the method described above.

[0022] Various aspects of the present disclosure are described by text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in computer program product (CPP) techniques. For any flowchart, depending on the technology involved, operations may be performed in an order different from that shown in a given flowchart. For example, again depending on the technology involved, two operations shown in successive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in an at least partially overlapping manner.

[0023] A computer program product method ("CPP method" or "CPP") is a term used in this disclosure to describe any set of one or more storage media (also referred to as "media") that are collectively contained in one or more storage devices that collectively contain machine-readable code corresponding to instructions and / or data for performing the computer operations specified in a given CPP claim. A "storage device" is any tangible device that can hold and store instructions for use by a computer processor. The computer-readable storage medium may be, but is not limited to, an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded device (such as pits / lands formed on the major surface of a punch card or disk), or any suitable combination of the foregoing. Computer-readable storage media, as the term is used in this disclosure, is not to be construed as storage in the form of a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, light pulses passing through fiber optic cables, electrical signals communicated through wires, and / or other transmission media. As will be appreciated by those skilled in the art, data is typically moved at some infrequent time during the normal operation of a storage device, such as during access, defragmentation, or garbage collection, but the foregoing does not qualify a storage device as transitory because the data is not transitory while it is stored.

[0024] The computing environment 100 includes an example environment for execution of at least some of the computer code involved in performing the inventive method, such as the VR environment extent determination module of block 200 for determining the extent of a VR environment to display on a VR device configured to control a machine that performs the operations of a workflow sequence. In addition to block 200, the computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this approach, the computer 101 includes a set of processors 110 (including processing circuitry 120 and cache 121), a communications fabric 111, volatile memory 112, persistent storage 113 (including an operating system 122 and the above-identified block 200), a set of peripheral devices 114 (including a set of user interface (UI) devices 123, storage 124, and a set of Internet of Things (IoT) sensors 125), and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140, a cloud orchestration module 141, a set of host physical machines 142, a set of virtual machines 143, and a set of containers 144.

[0025] Computer 101 may take the form of a desktop computer, a laptop computer, a tablet computer, a smartphone, a smartwatch or other wearable computer, a mainframe computer, a quantum computer, or any other form of computer or mobile device now known or later developed that is capable of executing programs, accessing a network, or querying a database, such as remote database 130. As is well understood in the field of computer technology, and depending on the technology, execution of a computer-implemented method may be distributed among multiple computers and / or among multiple locations. However, in this description of computing environment 100, for purposes of brevity, the detailed discussion focuses on a single computer, specifically computer 101. Although computer 101 is not shown in FIG. 1 within a cloud, it may be located within a cloud. On the other hand, computer 101 is not required to reside within a cloud except to any extent that may be expressly indicated.

[0026] Processor set 110 includes one or more computer processors of any type now known or later developed. Processing circuitry 120 may be distributed across multiple packages, e.g., multiple linked integrated circuit chips. Processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. Cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for fast access by threads or cores executing on processor set 110. Cache memory is typically organized into multiple levels depending on relative proximity to the processing circuitry. Alternatively, some or all caches for a processor set may be located “off-chip.” In some computing environments, processor set 110 may be designed to operate with qubits and perform quantum computing.

[0027] Computer-readable program instructions are typically loaded onto computer 101 and cause processor set 110 of computer 101 to perform a series of operational steps, thereby implementing a computer-implemented method. As a result, the instructions so executed instantiate the method set forth in the flowcharts and / or descriptions of the computer-implemented method (collectively, the "methods of the present invention") contained herein. These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 121 and other storage media discussed below. The program instructions and associated data are accessed by processor set 110 to control and direct the execution of the methods of the present invention. In computing environment 100, at least some of the instructions for executing the methods of the present invention may be stored in block 200 within persistent storage 113.

[0028] Communications fabric 111 is the signal-conducting pathway that allows various components of computer 101 to communicate with one another. Typically, this fabric is made up of switches and conductive pathways, such as the switches and conductive pathways that make up buses, bridges, physical input / output ports, etc. Other types of signal communication pathways may be used, such as fiber optic and / or wireless communication pathways.

[0029] Volatile memory 112 may be any type of volatile memory now known or later developed. Examples include dynamic random access memory (RAM) or static RAM. Typically, volatile memory 112 is characterized by random access, although this is not required unless expressly stated. In computer 101, volatile memory 112 is located in a single package and is internal to computer 101; however, alternatively or additionally, volatile memory may be distributed across multiple packages and / or located external to computer 101.

[0030] Persistent storage 113 is any form of non-volatile storage for a computer, now known or later developed. The non-volatility of this storage means that stored data is maintained regardless of whether power is supplied to computer 101 and / or directly to persistent storage 113. While persistent storage 113 may be read-only memory (ROM), typically at least a portion of persistent storage allows data to be written, data to be deleted, and data to be rewritten. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. Operating system 122 may take several forms, such as various known proprietary operating systems employing a kernel or open-source Portable Operating System Interface-type operating systems. The code contained in block 200 typically includes at least some of the computer code involved in performing the methods of the present invention.

[0031] The peripheral device set 114 includes the set of peripheral devices of the computer 101. Data communication connections between the peripheral devices and other components of the computer 101 may be implemented in various ways, such as Bluetooth connections, near field communication (NFC) connections, connections made by cables (such as universal serial bus (USB)-type cables), insertion-type connections (e.g., Secure Digital (SD) cards), connections made through local area communication networks, and even connections made through wide area networks such as the Internet. In various approaches, the UI device set 123 may include components such as display screens, speakers, microphones, wearable devices (such as goggles and smartwatches), keyboards, mice, printers, touchpads, game controllers, and haptic devices. The storage 124 may be external storage, such as an external hard drive, or insertable storage, such as an SD card. The storage 124 may be persistent and / or volatile. In some approaches, the storage 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In approaches where computer 101 is required to have a large amount of storage (e.g., computer 101 stores and manages a large database locally), in which case this storage may be provided by a peripheral storage device designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. IoT sensor set 125 consists of sensors that can be used in Internet of Things applications. For example, one sensor can be a thermometer and another sensor can be a motion detector.

[0032] Network module 115 is a collection of computer software, hardware, and firmware that enables computer 101 to communicate with other computers over WAN 102. Network module 115 may include hardware such as a modem or Wi-Fi signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for communicating data over the Internet. In some approaches, the network control and network forwarding functions of network module 115 are implemented on the same physical hardware device. In other approaches (e.g., approaches utilizing software-defined networking (SDN)), the control and forwarding functions of network module 115 are implemented on physically separate devices, such that the control function manages several different network hardware devices. Computer-readable program instructions for carrying out the methods of the present invention may be downloaded to computer 101 from an external computer or external storage device, typically through a network adapter card or network interface included in network module 115.

[0033] WAN 102 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances by any technology for transmitting computer data now known or later developed. In some approaches, WAN 102 may be replaced and / or supplemented by a local area network (LAN) designed to transmit data between devices located in a local area, such as a Wi-Fi network. WANs and / or LANs typically include copper transmission cables, optical transmission fiber, wireless transmission, and computer hardware such as routers, firewalls, switches, gateway computers, and edge servers.

[0034] End-user device (EUD) 103 is any computer system used and controlled by an end user (e.g., a customer of the enterprise operating computer 101) and may take any of the forms described above with respect to computer 101. EUD 103 typically receives useful and useful data from the operation of computer 101. For example, in the hypothetical case where computer 101 is designed to provide recommendations to the end user, the recommendations would typically be communicated from computer 101's network module 115 over WAN 102 to EUD 103. In this manner, EUD 103 can display or otherwise present the recommendations to the end user. In some approaches, EUD 103 may be a client device such as a thin client, a heavy client, a mainframe computer, a desktop computer, etc.

[0035] Remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. Remote server 104 may be controlled and used by the same entity that operates computer 101. Remote server 104 represents a machine that collects and stores useful and useful data for use by other computers, such as computer 101. For example, in the hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, this historical data may be provided to computer 101 from remote database 130 of remote server 104.

[0036] A public cloud 105 is any computer system available for use by multiple entities that provides on-demand availability of computer system resources and / or other computer capabilities, particularly data storage (cloud storage) and computing capacity, without requiring direct, active management by users. Cloud computing typically leverages resource sharing to achieve coherence and economies of scale. Direct, active management of public cloud 105 computing resources is performed by computer hardware and / or software in cloud orchestration module 141. The computing resources provided by public cloud 105 are typically implemented by virtual computing environments running on various computers comprising host physical machine set 142, a universe of physical computers within and / or available in public cloud 105. Virtual computing environments (VCEs) typically take the form of virtual machines from virtual machine set 143 and / or containers from container set 144. It is understood that these VCEs can be stored as images and can be transferred among and between various physical machine hosts, either as images or after instantiation of the VCEs. Cloud orchestration module 141 manages the transfer and storage of images, deploys new instantiations of VCE, and manages active instantiations of VCE deployments. Gateway 140 is a collection of computer software, hardware, and firmware that enables public cloud 105 to communicate over WAN 102.

[0037] Here, we provide some further explanation of virtual computing environments (VCEs). A VCE can be stored as an "image." A new, active instance of a VCE can be instantiated from the image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to a feature of an operating system in which the kernel allows the existence of multiple isolated user space instances called containers. These isolated user space instances typically behave as actual computers from the perspective of the programs running within them. A computer program running on a typical operating system can utilize all of the computer's resources, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, a program running inside a container can only use the contents of the container and the devices assigned to the container; this feature is known as containerization.

[0038] A private cloud 106 is similar to a public cloud 105, except that its computing resources are available only for use by a single enterprise. While the private cloud 106 is illustrated as being in communication with the WAN 102, in other approaches, the private cloud may be completely disconnected from the Internet and accessible only through a local / private network. A hybrid cloud is a composite of multiple clouds of different types (e.g., private, community, or public cloud types), often implemented by different vendors. While each of the multiple clouds remains a separate, discrete entity, the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the constituent clouds. In this approach, both the public cloud 105 and the private cloud 106 are part of the larger hybrid cloud.

[0039] In some aspects, systems according to various approaches may include a processor and logic integrated into and / or executable by the processor, the logic configured to perform one or more of the processing steps enumerated herein. The processor may be of any configuration described herein, such as a discrete processor or processing circuitry including numerous components such as processing hardware, memory, I / O interfaces, etc. "Integrated with" means that the processor has logic embedded as hardware logic, such as an application-specific integrated circuit (ASIC), FPGA, etc. "Executable by a processor" means that the logic is hardware logic, firmware, software logic such as part of an operating system, part of an application program, or any combination of hardware and software logic that is accessible by the processor and that, when executed by the processor, is configured to cause the processor to perform some function. Software logic may be stored in any memory type, local and / or remote memory, as known in the art. Any processor known in the art may also be used, such as a software processor module and / or hardware processors, such as an ASIC, FPGA, central processing unit (CPU), integrated circuit (IC), graphics processing unit (GPU), etc.

[0040] Of course, this logic may be implemented according to various approaches as a method on any device and / or system or as a computer program product.

[0041] As mentioned elsewhere above, virtual reality (VR) is a three-dimensional computer-generated environment that a person can explore and interact with. The person becomes part of and / or immerses themselves in the virtual world, and while there, can manipulate objects and / or perform a series of actions. VR benefits the manufacturing industry by improving worker safety, improving product production, saving manufacturers money, enabling remote control of machine operation, enabling monitoring of various machines on the industrial floor, and the like. More specifically, in some cases, VR allows operators to remotely control and monitor machine operation, e.g., start, stop, change operating parameters, etc., by using VR devices from one or more remote locations.

[0042] On any industrial floor, there may be different machines located and operating in different portions of the floor, where the machines may cooperate with one another to accomplish a common task or to implement any manufacturing workflow. In some contexts, a "workflow" may be defined as one or more operations performed to accomplish a predetermined task. More specifically, such a task may be realized as a result of one or more machines performing machine operations according to a "workflow sequence," in which one or more machines perform machine operations in parallel and / or sequentially in an interconnected manner. In some cases, each machine may be assigned a specific work task to perform, while in some other cases, multiple machines may additionally and / or alternatively work together to perform a single machine operation. On any industrial floor, the relative positions and orientations of these machines may be monitored and controlled to align with the machine's workflow sequence.

[0043] A human, hereinafter referred to as a “remote operator,” may remotely control a machine to perform machine operations using a VR device. For example, the remote operator may wear known types of VR glasses to view the machine's robotic arm and control the robotic arm using a handheld control. While remotely controlling, monitoring, or operating a machine on an industrial floor, the remote operator may visualize the machine in a respective “VR environment,” where the remote operator may remotely view the machine's actual perspective on the industrial floor. At the same time, various other remote operators may also remotely control, monitor, or operate other machines in the surrounding area. In this case, different operators may have different operating speeds, levels of Internet bandwidth, experience in controlling machines using a VR device, etc. Thus, in at least some implementations in which instructions for machines to execute in a workflow sequence are remotely submitted by an operator using a VR device, the timing of the execution of the instructions on each machine is important, for example, to prevent the machines from performing the workflow sequence operations in a different order. During VR interactions between multiple operators, a machine may encounter problems properly executing commands in a workflow sequence due to one or more types of latency associated with the condition of the remote operators. At the same time, latency in submitting or executing commands also impacts productivity in the context of a workflow sequence. Therefore, techniques are needed to mitigate latency in submitted commands. Furthermore, techniques are needed to enable execution of machine commands in a synchronized manner even when latency exists in multiple received machine commands, for example, as a result of one or more VR devices issuing machine commands with relatively low Internet bandwidth.

[0044] In stark contrast to the various drawbacks described above, the techniques of the various approaches described herein involve identifying a workflow sequence of execution for machines participating in a manufacturing process on an industrial floor. Next, a range of VR collaborative environments to be displayed on the VR devices is determined and output based on conditions related to the remote operators remotely controlling the machines using the VR devices, such as the internet bandwidth available to different remote operators performing activities from remote locations using the VR devices. In this manner, latency in the performance of the workflow sequence of execution at the manufacturing location can be minimized. For example, a first operator may have access to a relatively slow internet bandwidth, while a second operator may have access to a relatively fast internet bandwidth. Thus, for a first operator, only the machines that the first operator is actively controlling may be shown in the range of the VR environment displayed on the first operator's VR device. Meanwhile, based on the second operator's access to a relatively fast internet bandwidth, all machines involved in the manufacturing process may be shown in the range of the VR environment displayed on the second operator's VR device.

[0045] 2A-1-2A-3, there is shown a flowchart of one approach to method 201. In accordance with the present invention, method 201 may be performed in a variety of ways, particularly in any of the environments shown in Figures 1-3. Of course, as one of ordinary skill in the art will understand upon reading this specification, method 201 may include more or fewer operations than those specifically set forth in Figures 2A-1-2A-3.

[0046] Each of the steps of method 201 may be performed by any suitable component of an operating environment. For example, in various approaches, method 201 may be performed, in part or in whole, by a computer or some other device having one or more processors therein. A processor (e.g., a processing circuit, chip, and / or module) implemented in hardware and / or software, preferably having at least one hardware component, may be utilized in any device to perform one or more steps of method 201. Exemplary processors include, but are not limited to, a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like, combinations thereof, or any other suitable computing device known in the art.

[0047] Act 202 includes identifying machines, such as machine types, machine identities, collections of machines, etc., involved in performing the manufacturing process at the manufacturing location. In some approaches, the identification is performed by issuing queries to multiple machines requesting the operational responsibility of each of the machines. In another approach, the machines may be identified from network information. In yet another approach, the machines may be identified by observing the manufacturing location and determining which machines present at the manufacturing location are actively involved in performing machine operations related to the manufacturing process of the product. The machines may be of known types, such as robots, welding machines, sewing machines, hammering machines, molding machines, etc., configured to receive commands issued by, for example, a VR device controlled by a remote operator and, in response, perform one or more machine operations.

[0048] For context, a VR device may include known types of VR viewing devices, such as augmented reality (AR) glasses, a device display, VR glasses, a forward-facing camera device with a display component, etc. In some preferred approaches, the VR device is configured to display a perspective of a manufacturing location as seen by one or more cameras, e.g., mounted on, mounted to, or integrated into a machine. As described in more detail elsewhere herein, the manufacturing location may be modeled as a VR collaboration environment for display on one or more of the VR devices. More specifically, each VR collaboration environment may be a representation of the manufacturing location, which may include all or some of the content, e.g., machines, obstacles, clarity, lighting, etc., that actually exists at the manufacturing location. For example, the VR collaboration environment displayed on a particular one of the VR devices may model the perspective seen at any particular time by a camera of the associated machine controlled by the VR device. In this manner, a remote operator wearing a VR device can operate and control machines as if the remote operator were actually working at the manufacturing location, while avoiding any potential hazards of being physically present at the manufacturing location.

[0049] A workflow sequence of machine execution may be identified (e.g., see operation 204). As referred to elsewhere herein, a "workflow" may be defined as one or more operations performed to accomplish a predetermined task, where the performed operations establish a "workflow sequence of execution." More specifically, such a task may be realized as a result of one or more machines performing machine operations according to a "workflow sequence." In this workflow sequence, one or more machines perform machine operations in an interconnected manner, in parallel and / or serially. In some cases, each machine may be assigned a specific work task to perform, while in some other cases, multiple machines may additionally and / or alternatively work together to perform a single machine operation. On any industrial floor, the relative positions and orientations of these machines may be monitored and controlled to align with the machine's workflow sequence. Thus, in some approaches, a workflow sequence may be identified by monitoring the operation of a machine, for example, for a predetermined amount of time, until it is determined that product has been produced a predetermined amount of time, until at least a predetermined number of machine operational responsibilities have been complied with and determined, etc. In some other approaches, a workflow sequence may be identified from instructions issued to a machine from a VR device controlled by a remote operator. For example, a workflow sequence may include each of the machine instructions issued to the machine by a remote operator via a VR device.

[0050] Conditions related to a remote operator using the VR device to remotely control a machine to perform a workflow sequence of execution at a manufacturing location are received (e.g., see operation 206). By way of context, in some preferred approaches, these conditions related to the remote operator's use of the VR device are conditions that may introduce latency in the transmission of instructions through the VR device to perform machine operations. For example, in one preferred approach, the conditions are the relative internet bandwidth of the VR device, e.g., the internet bandwidth available to the VR device, the internet bandwidth to which one or more of the VR devices are connected, the internet bandwidth at which the VR device outputs instructions for the machine to perform machine operations, etc. By way of context, different VR devices with different internet bandwidths may introduce latency in the transmission of instructions through the VR device based on the order in which instructions are received, which may not match the order in which the instructions were previously sent. For purposes of example, a first VR device may be assumed to have a relatively smaller internet bandwidth than a second VR device. Based on this difference in bandwidth, a first command may be issued by a first VR device before a second command is issued by a second VR device, but the second command may be received before the first command is received. In some other approaches, the conditions may additionally and / or alternatively include, for example, the operator's safety record, the amount of time since the operator took their most recent break from work, the operator's relative skill level and / or proficiency level, etc.

[0051] In some approaches, the extent of the VR collaborative environment to display on the VR device can be determined based on the conditions to take these conditions into account, thereby reducing latency in the performance of the workflow sequence of execution at the manufacturing location. For example, operation 208 includes determining, for each of the VR devices, the extent of the VR collaborative environment to display. For context, as described below, taking these conditions into account can include adjusting the specific extent of the VR collaborative environment for each of the VR devices to ensure that the conditions do not introduce latency into the workflow sequence of execution at the manufacturing location, thereby reducing latency in the performance of the workflow sequence of execution at the manufacturing location that would exist without the use of the techniques described herein. Reducing latency in this manner improves the performance of the computing devices used in the performance of the workflow sequence of execution. This also results in reduced waste in the production process at the manufacturing location because errors that would otherwise result in the production process as a result of such latency are avoided.

[0052] Various exemplary techniques for determining the extent of a VR collaborative environment are now described below. For example, referring to Figure 2B, exemplary partial operations for determining the extent of a VR collaborative environment according to one approach are shown, one or more of which may be used to perform operation 208 of Figures 2A-1 through 2A-3. However, it should be noted that the partial operations of Figure 2B are shown according to one approach and are in no way intended to limit the invention.

[0053] 2B , a subprocess is shown in which bandwidth requirements are considered to determine the scope of a VR collaborative environment to optimize the effectiveness of VR collaboration and the management of latency in the execution of a workflow sequence. Suboperation 240 includes determining a first machine remotely controlled by a first VR device having a relatively low Internet bandwidth, e.g., a relatively high latency. Such determination may be based on comparing the Internet bandwidth with the Internet bandwidth of one or more other VR devices, e.g., a second VR device. Suboperation 242 determines a second machine remotely controlled by a second VR device having a relatively high Internet bandwidth, e.g., a relatively low latency. In some approaches, the scope of the VR collaborative environment may be reduced to include only one or more relevant portions of the VR collaborative environment, for example, to mitigate latency that would otherwise result from the first VR device outputting machine commands using a relatively low Internet bandwidth. For example, a machine that the first VR device remotely controls, e.g., a target device, may preferably be considered relevant, while a machine that the first VR device does not remotely control, e.g., a non-target device, may not be considered relevant. For example, in suboperation 244, the scope of the VR collaborative environment for a first VR device with a relatively low Internet bandwidth includes the first machine and excludes the second machine. This inclusion and exclusion (respectively) may be performed in response to determining that the first VR device has a relatively low Internet bandwidth. As a result of not including the second machine in the scope of the VR collaborative environment for the first VR device with a relatively low Internet bandwidth, bandwidth availability is preserved. This results in a relative reduction in latency in the performance of workflow sequences of execution at the manufacturing location because the first VR device receives a scope of the VR collaborative environment with a relatively low bandwidth usage, thereby allowing the output of machine instructions to be relatively more efficient, e.g., not delayed.In contrast, sub-operation 246 includes including the first machine and the second machine in a scope of the VR collaborative environment for the second VR device in response to determining that the second VR device has a relatively high internet bandwidth. Note that the second VR device has access to a relatively large internet bandwidth and therefore may be able to afford the costs associated with downloading additional scope of the VR collaborative environment without adding latency to the performance of the workflow sequence of execution at the manufacturing location.

[0054] It should be noted that while the various approaches herein are described from the perspective of a first VR device and a second VR device, in some other approaches, method 201 may optionally include a step of layering the scope of the VR collaborative environment for more than two VR devices according to multiple relative bandwidth speeds, e.g., a relative bandwidth speed of 5 VR devices, a relative bandwidth speed of 50 VR devices, a relative bandwidth speed of 100 VR devices, etc.

[0055] 2B , some approaches include including some machines in the scope of the VR collaborative environment while excluding others, and alternatively, increasing the relative clarity of some machines, e.g., the target machine, and decreasing the relative clarity of some machines, e.g., the target machine, in the VR collaborative environment. For example, an optional approach for determining the scope of the VR collaborative environment may include, in response to determining that a first VR device has a relatively low internet bandwidth, assigning the first machine to have a relatively high degree of clarity in the scope of the VR collaborative environment for the first VR device compared to a degree of clarity of the second machine. Further, in response to determining that a second VR device has a relatively high internet bandwidth, assigning the first machine to have approximately the same degree of clarity as the second machine in the scope of the VR collaborative environment for the second VR device.

[0056] Referring now to Figure 2C, exemplary partial operations for determining the scope of a VR collaborative environment according to one approach are shown, one or more of which may be used to perform operation 208 of Figures 2A-1-3. However, it should be noted that the partial operations of Figure 2C are shown according to one approach and are in no way intended to limit the invention.

[0057] Referring now to FIG. 2C , in some approaches, based on the level of available bandwidth, the proposed VR collaboration technique may identify which remote activities may be constrained for one or more remote operators with relatively low levels of available bandwidth. As a result, the bandwidth of at least some of these VR devices of these different remote operators may be preserved. In response, work may be dynamically assigned to different workers, e.g., other workers with relatively high levels of available bandwidth. Based on the Internet bandwidth available to the different VR devices, as different workers begin to perform activities remotely in a collaborative manner, the proposed technique may dynamically assign different work to different remote operators and may also identify which work assignments to disable and / or reassign. For example, a first machine remotely controlled by a first VR device with relatively low Internet bandwidth may be determined (e.g., see sub-act 240). Furthermore, a second machine remotely controlled by a second VR device with relatively high Internet bandwidth may additionally and / or alternatively be determined (e.g., see sub-act 242). In some approaches, based on one or more of these determinations, at least some work tasks currently assigned to a first teleoperator using a first VR device may be reassigned to a second teleoperator using a second VR device (see, e.g., suboperation 248). In alternative approaches, based on one or more of these determinations, at least some work tasks currently assigned to a first teleoperator may additionally and / or alternatively be canceled.

[0058] In some optional approaches, method 201 may include training an artificial intelligence (AI) model to determine and / or update the scope of the VR collaborative environment. More specifically, in one or more such approaches, the inputs of the trained model may include information, such as conditions, current assignments of work tasks, any of the other information described elsewhere herein used to determine scope, etc., and the outputs of the trained model may include scope. In some approaches, a subject matter expert (SME) may review the model's inputs and outputs and apply rewards and / or corrective feedback as part of the training process.

[0059] As described elsewhere herein, determining and / or updating the scope of the VR collaborative environment may additionally and / or alternatively be based on predetermined safety criteria (see, e.g., decision 228 - action 232).

[0060] Referring again to FIGS. 2A-1-2A-3, the determined range is output to an associated VR device (see, for example, operation 210).

[0061] The VR remote collaboration techniques described herein may additionally and / or alternatively evaluate the time delay of receiving commands from VR devices with different communication latencies. In some approaches, based on this latency, one or more operations may be performed to identify operations with such latencies in a workflow sequence of machine execution. In this manner, commands performed on different machines are ensured to be corrected for appropriate execution times, e.g., to not differ in order from the workflow sequence of execution as a result of latency.

[0062] In some approaches, timestamps associated with machine commands received from different VR devices for different machines may be analyzed. Based on this analysis, corrections may be made to the sequence in response to a determination that the timestamps deviate from the actual execution sequence. Furthermore, in response thereto, additional buffer latency may be incorporated into the workflow sequence of execution, e.g., allowing operations to be executed relatively smoothly without latency. For purposes of illustrative example, it may be assumed that a first command is received from a first VR device to perform a first machine operation (e.g., see act 212). A second command may be received from a second VR device to perform a second machine operation (e.g., see act 214). In the current approach, it may be assumed that the first command is received before the second command is received. The command timestamp information may be analyzed to determine whether the second command is output by the second VR device before the first command is output by the first VR device (e.g., see act 216). For example, using techniques that will be understood by those skilled in the art upon reading the present specification, timestamp information may be compared to determine which command was output first by the VR device. For context, this analysis may be performed to determine whether internet bandwidth latency is at least one of the reasons why a first command is received before a second command is received. More specifically, analysis may be performed to determine whether a second VR device begins outputting a second command before the first VR device begins outputting a first command, but based on the relatively high internet bandwidth of the first VR device, the first command is received before the second command is received. In other words, the second machine operation may actually be intended to be performed before the first machine operation in a workflow sequence of execution.

[0063] Act 218 includes causing the machine operations to be performed according to an order determined based on analysis of the timestamp information. In one approach, in response to determining, based on analysis of the timestamp information, that a second instruction was output by a second VR device before a first instruction was output by a first VR device, the second machine operation may be performed before the first machine operation. This action mitigates any latency present in the order in which instructions are received from being incorporated into the order in which the machine operations are performed by the machines. In some approaches, the analysis may additionally and / or alternatively compare timestamp information for the same VR device and / or machine. For purposes of an illustrative example, it may be assumed that a first operator submits a machine instruction for a first machine at time T1, and then a machine instruction is received from the first operator's VR device for a second machine at time T2. In some approaches, ideally, the second machine instruction should be executed first. Accordingly, method 201 may include causing this correction to be made, e.g., issuing an instruction for the second machine instruction to be issued before the first machine instruction.

[0064] In some approaches, one or more of the remote operators may have a scope of work responsibility that causes the remote operator to operate more than one machine at a manufacturing location using a VR device. Accordingly, in some approaches, determining and / or updating the scope of the VR collaborative environment may additionally and / or alternatively be based on which machine the VR device is currently remotely controlling. For example, a remote operator may occasionally need to perform activities using multiple machines, and thus, based on the worker's virtual movement in the VR collaborative environment, the level of clarity of different portions of the VR collaborative environment may change from time to time, for example, based on the sequence of the worker's activities. Accordingly, it may be determined whether a VR device, e.g., a first VR device currently controlling a first machine, will gain control of a second machine (see, e.g., decision 220). In one example, it may subsequently be assumed that the first machine is remotely controlled by a VR device with a relatively low Internet bandwidth and that the second machine is remotely controlled by a VR device with a relatively high Internet bandwidth. Furthermore, it may be assumed that the first VR device initially controls the first machine and not the second machine. In some approaches, in response to a determination that the first VR device will gain control of a second machine, e.g., a portion of a task sequence in which a remote operator of the first VR device is granted authorization to control the second machine, a relative degree of clarity of the second machine may be increased in an updated version of the scope of the VR collaborative environment for the first VR device (e.g., see act 222). Further, in response to a determination that the first VR device will gain control of the second machine, a relative degree of clarity of the first machine may be decreased in an updated version of the scope of the VR collaborative environment for the first VR device (e.g., see act 224). The updated version of the scope of the VR collaborative environment for the first machine may be output to the first VR device (e.g., see act 226).For purposes of an example involving more than two machines, a first operator may initially operate machine A via a first VR device, although it may be assumed that other machines, such as machines B, C, D, and E, are also present in the surrounding area of ​​the manufacturing location, e.g., within a predetermined proximity. Thus, in response to determining that the first VR device experiences the relatively lowest bandwidth, the scope of the VR collaborative environment for the first VR device may include machine A with the highest degree of clarity, while the remaining machines are included with relatively lower degrees of clarity. Thereafter, in response to determining that the first operator has a next activity scheduled on machine B and / or that the first VR device has gained control of machine B, machine B may be included with the relatively highest degree of clarity in the scope of the VR collaborative environment for the first VR device.

[0065] In some approaches, one or more areas of a manufacturing site may be subject to one or more predetermined safety standards. These predetermined safety standards may be established, for example, by a manufacturing site supervisor, applicable laws, a manufacturing site safety officer, etc., and may be enforced to ensure that machines operate relatively safely while collaboratively completing workflow sequences of execution. For example, while building a collaborative VR environment for different VR devices with varying Internet bandwidth, some approaches may include identifying whether information about the surrounding industrial floor, along with the target machines, is also included within the VR collaborative environment. Thus, the level of clarity of different portions of the VR collaborative environment may be determined based on the degree of safety required in different areas of the manufacturing site and included within the VR collaborative environment. For example, a first area of ​​a manufacturing site may contain hazardous waste and therefore be subject to a safety protocol that instructs machines within the first area to maintain a predetermined proximity from each other. This may help ensure that collisions between two or more machines do not occur, otherwise a hazardous waste spill event may occur. Thus, in such an example, it may be important for a remote operator controlling a machine in a first area to view the scope of the VR collaborative environment, including the machine the remote operator is controlling and other machines in the first area that may come within a predetermined proximity of the first machine. By way of another example, it may be assumed that predetermined safety standards apply to a first area of ​​a manufacturing location. Determining 228 may include determining whether the first machine has entered an area to which the predetermined safety standards apply.In response to a determination that a first machine, e.g., a target device of the first VR device remotely controlled by the first VR device, has entered and / or is scheduled to enter the first area, for example, as indicated by the “Yes” logic path of decision 228, a relative degree of clarity of a predetermined range of the area surrounding the first machine may be increased in an updated version of the range of the VR collaborative environment for the first VR device (e.g., see operation 230). One or more other areas of the manufacturing location in the machine's view to which the predetermined safety standard does not apply may have an associated degree of reduced clarity in the updated version of the range of the VR collaborative environment for the first VR device. The updated version of the range of the VR collaborative environment for the first machine may be output to the first VR device (e.g., see operation 232). In contrast, in response to a determination that the first machine has not entered an area to which the predetermined safety standard applies, for example, as indicated by the “No” logic path of decision 228, the method optionally ends (e.g., see “End”). It should be noted that although the method is described as optionally terminating, additional monitoring may be performed in some other manner, for example, monitoring of areas of the manufacturing location that are subject to one or more predetermined safety rules throughout the workflow sequence of machine execution.

[0066] In some other approaches, one or more of the machines may be specifically subject to predetermined safety standards in addition to and / or instead of one or more predetermined areas of the manufacturing location. For example, assuming a remote operator needs to see the entire surroundings of a target machine while initiating commands to be performed on the target machine, the relevant area of ​​the VR collaborative environment may be configured to have a relatively high degree of clarity. Then, based on the required level of safety at the manufacturing location, the level of clarity may vary throughout the VR collaborative environment. For example, in some approaches, the target machine remotely controlled by the remote operator's VR device may be configured to have a relatively high degree of clarity, while surrounding machines and / or the remote operator, being subject to relatively limited safety parameters, may result in other machines having a relatively low degree of clarity within the VR collaborative environment.

[0067] In some additional approaches, the presence of a worker at a manufacturing location may result in additional safety standards being applied. For example, a safety threat may arise for a worker when the worker is within a predetermined proximity of a machine that is performing machine operations according to a workflow sequence of execution. Thus, in some approaches, in response to a determination that the worker has come within a predetermined distance of a machine, the worker's relative intelligibility may be increased within a VR collaborative environment output to a VR device having remote control for the machine. In some other approaches, in response to a determination that the worker has entered a predetermined area of ​​the manufacturing location that is subject to one or more predetermined safety standards, the worker's relative intelligibility may be increased within a VR collaborative environment output to a VR device having remote control for one or more machines also in the predetermined area.

[0068] It should be noted that while the various described operations take into account relative Internet bandwidth conditions, such operations may additionally and / or alternatively be performed with respect to one or more other conditions. For example, the conditions may be assumed to include relative operator experience. In response to determining that a first remote operator is relatively less experienced, method 201 may include reallocating at least some work tasks currently assigned from the first remote operator using the first VR device to a second remote operator using the second VR device. In yet another approach, the conditions may additionally and / or alternatively be assumed to include whether a particular machine is performing a machine operation using a predetermined fragile product component. In response to determining that the first machine is performing a machine operation using a predetermined fragile product component, method 201 may include increasing a relative degree of clarity of a range of the VR collaborative environment output to the VR device remotely controlling the first machine. In some other approaches, the conditions may additionally and / or alternatively be assumed to include the relative safety record of the remote operators. For example, the relative safety record of a first remote operator using a first VR device controlling a first machine may be monitored, and in response to determining that the safety record is below a predetermined threshold, method 201 may include increasing the relative degree of clarity of the area of ​​the VR collaborative environment output to the first VR device and / or reallocating at least some work tasks currently assigned to the first remote operator.

[0069] Implementing the techniques described herein at a manufacturing location enables many benefits. For example, conditions such as different relative internet bandwidths of VR devices are prevented from introducing latency into the execution of a workflow sequence at the manufacturing location because these conditions are accounted for by adjusting the specific scope of the VR collaborative environment for each of the VR devices to ensure that the conditions do not introduce latency into the execution of a workflow sequence at the manufacturing location. Thus, the operations described herein enable machine operations to be performed efficiently, and in some cases synchronously, according to a workflow sequence, where operations would not otherwise be performed synchronously as a result of unaccounted latency, thereby reducing latency in the execution of a workflow sequence at the manufacturing location that would exist without the use of the techniques described herein. Reducing latency improves the performance of computing devices used in the execution of a workflow sequence. This also results in reduced waste within the production process at the manufacturing location because errors that would otherwise result in the production process as a result of such latency are avoided. It should also be noted that determining the extent of a VR collaborative environment to display on a VR device based on a condition has not previously been considered in conventional applications. In stark contrast, the latency that causes the condition remains unaccounted for, compromising the efficiency of the manufacturing process and the quality of the product. Thus, the inventive findings disclosed herein regarding determining the extent of a VR collaborative environment to display on a VR device based on a condition proceed contrary to conventional wisdom.

[0070] Referring now to Figure 3, there is shown a flow chart of one approach to method 300. In accordance with the present invention, method 300 may be performed in a variety of ways, particularly in any of the environments shown in Figures 1-3. Of course, more or fewer operations may be included in method 300 than those specifically illustrated in Figure 3, as will be understood by those skilled in the art upon reading this specification.

[0071] Each of the steps of method 300 may be performed by any suitable component of an operating environment. For example, in various approaches, method 300 may be performed, in part or in whole, by a computer or some other device having one or more processors therein. A processor (e.g., a processing circuit, chip, and / or module) implemented in hardware and / or software, preferably having at least one hardware component, may be utilized in any device to perform one or more steps of method 300. Exemplary processors include, but are not limited to, a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or the like, combinations thereof, or any other suitable computing device known in the art.

[0072] 3 may be prefaced by describing different remote operators that can cooperate with each other and operate machines on a manufacturing site, such as an industrial floor. In some approaches, the operation of the machines on the industrial floor may be based on a configuration change, an operating parameter change, a start command, a stop command, or any other command, and / or may be based on the available internet bandwidth of different VR devices used by different remote operators. Various operations of method 300 may be used to identify regions of the VR environment to be output for display on different VR devices.

[0073] Method 300 can be implemented on any industrial floor where one or more different types of machines are performing activities according to a workflow sequence of execution, and where the different machines have different specifications. In some approaches, while a machine is performing an activity, the output of one machine can serve as the input of another machine performing an activity according to the workflow. In this industrial context, a workflow sequence of a manufacturing process can be identified, and machine roles can be identified (e.g., see operations 302 and 304 of the initialization series of operation 310). Contextually, machine roles are determined by the activities performed and their importance to the workflow. In some approaches, these roles can be determined from work tasks currently assigned to remote operators who control the machines using VR devices. For example, multiple remote operators, each wearing a VR glasses device, are shown in FIG. 3 (see, e.g., remote operator 314, remote operator 316, remote operator 318, and remote operator 320). Using the VR devices, workers can remotely configure, control, and modify the operating parameters of different associated machines. For example, the relative positions and orientations of different machines on the industrial floor may additionally and / or alternatively be identified, such as the directions and footprints that the machines operate in in the course of performing machine operations.

[0074] While the machine is performing the activity, the machine's performance, capabilities, operating parameters, etc. may be considered and associated with the VR content configuration and activity assignment (e.g., see operation 306). The VR content may be based on identified areas of the periphery that are output and displayed on different VR devices along with a threshold level of clarity (e.g., see operation 308). In this manner, machine operations may be synchronized and executed according to a workflow sequence. However, it should be noted that different conditions associated with the VR devices controlling the machine may introduce latency into the performance of the workflow sequence. As described herein below, method 300 may include performing operations to reduce this latency incorporated into the performance of the workflow sequence.

[0075] Which machines are being controlled remotely by the VR devices may be identified. Additionally, the available internet bandwidth of different remote VR devices may be identified (e.g., see act 322). These conditions and output of the initialization series of act 310 may be input to a VR collaboration system 312, which may be a trained AI model (described elsewhere herein). A range of the VR collaboration environment to display may be determined for each of the VR devices, where the range is determined based on the conditions (e.g., see act 324). Thereby, latency in the performance of workflow sequences of execution at the manufacturing location is reduced by determining these ranges based on these conditions. In some approaches, determining the range of the VR collaboration environment includes identifying the manufacturing process, which machines should be processed in the sequence, and the timestamp of the execution. In other words, in cases where conditions cause latency and cause operations to be performed in a different order, an order of operations in the workflow sequence of execution may be determined to establish as a reference. The amount of bandwidth required for different types of VR collaboration may additionally and / or alternatively be determined to determine the range of the VR collaboration environment. For example, a VR device that does not have at least a predetermined minimum threshold of Internet bandwidth may not be issued work tasks to perform for the workflow sequence of execution. In some approaches, the operations may also include estimating bandwidth requirements and using historical learning about the bandwidth needs of the workflow sequence of execution.

[0076] In some approaches, the extent of the VR collaborative environment to display can be determined by analyzing the workflow sequence of the manufacturing process and identifying which machines are actively being remotely controlled by VR devices. In this manner, work task reassignments can be output to VR devices that are available and active at the manufacturing location.

[0077] In some other approaches, determining the scope of the VR collaborative environment may take into account one or more safety factors to which one or more machines and / or areas of the manufacturing location are subject. For example, based on the level of safety to which one or more machines and / or areas of the manufacturing location are subject, one or more machines may be included with a relatively high degree of clarity in the scope of the VR collaborative environment. In contrast, machines that are not in areas of the manufacturing location that are subject to safety factors may not be included in the scope of the VR collaborative environment or may be included with a relatively low degree of clarity.

[0078] In some approaches, the types of activities assigned to different remote operators may be based on available internet bandwidth (see, e.g., act 326). For example, in response to determining that a remote operator uses a VR device with access to relatively low internet bandwidth, the remote operator may be assigned relatively fewer work tasks and / or work tasks that are relatively less resource-intensive. Furthermore, these work task activity assignments may be changed at any time, such as in response to determining that a VR device that previously had relatively low internet bandwidth now has access to relatively higher internet bandwidth.

[0079] Operation 328 includes reordering the machine instructions in response to determining that the instructions received from the different VR devices include latency. For example, one or more operations of method 300 may include receiving timestamps associated with the instructions received from the different remote operators. The timestamps may be evaluated with respect to the manufacturing process sequence to determine whether the machine instructions are received in a different order based on the latency. In response to determining that the machine instructions are received in a different order, one or more reordering instructions may be issued to correct the instruction execution timeline. In this manner, latency is reduced and the machine execution sequence is aligned with the manufacturing process.

[0080] It will be apparent from the description given above that the various features of the systems and / or methods described above may be combined in any manner, thereby creating multiple combinations.

[0081] It will be further appreciated that the techniques of the present invention may be provided in the form of a service deployed on behalf of a customer to provide the service on demand.

[0082] The description of various techniques of the present invention has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the techniques disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the techniques described. The terminology used herein has been selected to best explain the principles of the techniques, their practical applications, or technical improvements over techniques found in the market, or to enable others skilled in the art to understand the techniques disclosed herein.

Claims

1. identifying machines involved in carrying out the manufacturing process at the manufacturing location; identifying a workflow sequence for execution of said machine; receiving associated requirements from a remote operator using a virtual reality (VR) device to remotely control the machine to perform the workflow sequence of execution at the manufacturing location; and determining, for each of the VR devices, a range of a VR collaboration environment to display, wherein the range is determined based on the condition, thereby reducing latency in performing the workflow sequence of execution at the manufacturing location; and outputting the range to the VR device.

1. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the conditions include relative internet bandwidth of the VR devices.

3. 3. The computer-implemented method of claim 2, wherein determining the extent of the VR collaboration environment includes: determining a first machine remotely controlled by a first VR device having a relatively low Internet bandwidth; and determining a second machine remotely controlled by a second VR device having a relatively high Internet bandwidth.

4. 4. The computer-implemented method of claim 3, wherein determining the extent of the VR collaboration environment includes including the first machine but not the second machine in the extent of the VR collaboration environment for the first VR device having the relatively low Internet bandwidth, and including the first machine and the second machine in the extent of the VR collaboration environment for the second VR device having the relatively high Internet bandwidth.

5. 4. The computer-implemented method of claim 3, wherein determining the extent of the VR collaboration environment includes: allocating the first machine to have a degree of clarity in the extent of the VR collaboration environment for the first VR device that is relatively higher than a degree of clarity of the second machine; and allocating the first machine to have a degree of clarity in the extent of the VR collaboration environment for the second VR device that is the same as a degree of clarity of the second machine.

6. 6. The computer-implemented method of claim 5, further comprising: in response to a determination that the first VR device will gain control of the second machine, increasing a relative degree of clarity of the second machine in an updated version of the scope of the VR collaborative environment for the first VR device; and outputting the updated version to the first VR device.

7. 7. The computer-implemented method of claim 6, further comprising: in response to the determination that the first VR device has gained control of the second machine, reducing a relative degree of clarity of the first machine in the updated version of the scope of the VR collaborative environment for the first VR device.

8. 3. The computer-implemented method of claim 2, wherein determining the scope of the VR collaboration environment includes: determining a first machine remotely controlled by a first VR device having a relatively low Internet bandwidth; and determining a second machine remotely controlled by a second VR device having a relatively high Internet bandwidth; and reassigning at least some work tasks currently assigned to a first remote operator using the first VR device to a second remote operator using the second VR device.

9. 2. The computer-implemented method of claim 1, wherein predetermined safety standards apply to a first area of ​​the manufacturing location, and in response to determining that a first machine remotely controlled by a first VR device has entered and / or is scheduled to enter the first area, the method comprises: increasing the relative degree of clarity of the area surrounding the first machine in an updated version of the scope of the VR collaborative environment for the first VR device; and outputting the updated version to the first VR device.

10. 2. The computer-implemented method of claim 1, comprising: receiving a first command from a first VR device to perform a first machine operation; receiving a second command from a second VR device to perform a second machine operation, wherein the first command is received before the second command is received; analyzing timestamp information of the commands to determine whether the second command was output by the second VR device before the first command was output by the first VR device; and performing the second machine operation before the first machine operation in response to a determination based on the analysis that the second command was output by the second VR device before the first command was output by the first VR device.

11. 1. A computer program product including a computer-readable storage medium having program instructions embodied thereon, the program instructions being readable and / or executable by a computer to cause the computer to: identifying, by said computer, machines involved in carrying out a manufacturing process at a manufacturing location; identifying, by said computer, a workflow sequence for execution by said machine; receiving, by the computer, conditions associated with a remote operator using a virtual reality (VR) device to remotely control the machine to perform the workflow sequence of the execution at the manufacturing location; and determining, by the computer, for each of the VR devices, a range of a VR collaborative environment to display, wherein the range is determined based on the condition, thereby reducing latency in the performance of the workflow sequence of execution at the manufacturing location; and outputting the range to the VR device by the computer; A computer program product that causes

12. The computer program product of claim 11 , wherein the conditions include relative internet bandwidth of the VR devices.

13. 13. The computer program product of claim 12, wherein the step of determining the extent of the VR collaboration environment includes the steps of: determining a first machine remotely controlled by a first VR device having a relatively low Internet bandwidth; and determining a second machine remotely controlled by a second VR device having a relatively high Internet bandwidth.

14. 14. The computer program product of claim 13, wherein the step of determining the extent of the VR collaboration environment includes including the first machine but not the second machine in the extent of the VR collaboration environment for the first VR device having the relatively low Internet bandwidth, and including the first machine and the second machine in the extent of the VR collaboration environment for the second VR device having the relatively high Internet bandwidth.

15. 14. The computer program product of claim 13, wherein the step of determining the extent of the VR collaboration environment includes: assigning the first machine to have a degree of clarity in the extent of the VR collaboration environment for the first VR device that is relatively higher than a degree of clarity of the second machine; and assigning the first machine to have a degree of clarity in the extent of the VR collaboration environment for the second VR device that is the same as a degree of clarity of the second machine.

16. 16. The computer program product of claim 15, wherein the program instructions readable and / or executable by the computer cause the computer to: in response to a determination that the first VR device has gained control of the second machine, increase the relative degree of clarity of the second machine in an updated version of the scope of the VR collaborative environment for the first VR device; and output the updated version to the first VR device.

17. 13. The computer program product of claim 12, wherein the step of determining the scope of the VR collaboration environment includes: determining a first machine remotely controlled by a first VR device having a relatively low Internet bandwidth; and determining a second machine remotely controlled by a second VR device having a relatively high Internet bandwidth, and the computer-readable and / or executable program instructions cause the computer to: reallocate at least some work tasks currently assigned to a first remote operator using the first VR device to a second remote operator using the second VR device.

18. 12. The computer program product of claim 11, wherein predetermined safety standards apply to a first area of ​​the manufacturing location, and the program instructions readable and / or executable by the computer cause the computer to: in response to determining that a first machine remotely controlled by a first of the VR devices has entered and / or is scheduled to enter the first area, increase the relative degree of clarity of the area surrounding the first machine in an updated version of the range of the VR collaboration environment for the first VR device; and output the updated version to the first VR device.

19. 12. The computer program product of claim 11, wherein the program instructions readable and / or executable by the computer cause the computer to perform: a procedure for receiving, by the computer, a first instruction from a first VR device to perform a first machine operation; a procedure for receiving, by the computer, a second instruction from a second VR device to perform a second machine operation, wherein the first instruction is received before the second instruction is received; a procedure for analyzing, by the computer, timestamp information of the instruction to determine whether the second instruction was output by the second VR device before the first instruction was output by the first VR device; and a procedure for causing the computer to perform the second machine operation before the first machine operation in response to a determination based on the analysis that the second instruction was output by the second VR device before the first instruction was output by the first VR device.

20. a processor; and Logic integrated with, executable by, or integrated with and executable by the processor the logic comprising: Procedures for identifying machines involved in carrying out the manufacturing process at the manufacturing location; identifying a workflow sequence for execution of said machine; receiving associated requirements from a remote operator using a virtual reality (VR) device to remotely control the machine to perform the workflow sequence of execution at the manufacturing location; and determining, for each of the VR devices, a range of a VR collaborative environment to display, wherein the range is determined based on the condition, thereby reducing latency in the performance of the workflow sequence of execution at the manufacturing location; and outputting the range to the VR device; configured to: system.

Images