Determining range of virtual reality (VR) environment for display on VR device

Abstract

According to one method, a computer-implemented method includes identifying a machine involved in performing a manufacturing process at a manufacturing location, and identifying a workflow sequence of execution of the machine. A condition associated with a sequence of workflows for a remote operator to remotely control the machine to perform the execution at the manufacturing location using a virtual reality (VR) device is received. The method further includes determining, for each of the VR devices, a range of VR collaboration environments to be displayed, where the range is determined based on the condition, thereby reducing a delay in execution of the executed workflow sequence at the manufacturing location. The method further includes outputting the range to the VR device.

Description

Background Art

[0001] The present invention relates to virtual reality (VR), and more particularly, to determining a range of a VR environment to be displayed on a VR device configured to control a machine that performs operations of a workflow sequence.

[0002] VR is a three-dimensional computer-generated environment that can be explored and interacted with by a person. This person becomes part of and / or immersed in the virtual world, and is able to manipulate objects and / or perform a series of actions therein at the same time. VR benefits the manufacturing sector by enhancing worker safety, improving product production, saving manufacturers' money, enabling remote control of machine operations, allowing monitoring of various machines on an industrial site, etc. More specifically, in some cases, VR enables an operator using a VR device from one or more remote locations to remotely control and monitor machines, such as starting, stopping, changing the operating parameters of the machines, etc.

[0003] In any industrial site, there may be different machines located in and operating in different parts of the industrial site. Here, the machines are able to cooperate with each other to perform a common task or execute any manufacturing workflow. For context, a "workflow" can be defined as one or more operations performed to complete a predetermined task. More specifically, a task such as can be completed as a result of one or more machines performing machine operations according to a "workflow sequence". Within this workflow sequence, one or more machines perform machine operations in an interconnected manner in parallel and / or sequentially. In some cases, each machine can be assigned a specific work task to perform, while in some other cases, multiple machines can additionally and / or alternatively operate together to perform a single machine operation. At any industrial site, the relative positions and orientations of these machines can be monitored and controlled to align with the workflow sequence of the machines.

[0004] For example, a person, hereinafter referred to as a "remote operator", may use a VR device to remotely control a machine to perform machine operations. For example, the remote operator may wear a VR headset of a known type to view the robotic arm of the machine and use a (one or more) hand-held control to control the robotic arm. When remotely controlling, monitoring, or operating any machine on any industrial site, the remote operator can visualize the target machine in the corresponding "VR environment" and perceive themselves as if they were actually on the industrial site where the machine is located. At the same time, various other remote operators can also remotely control, monitor, or operate other machines in the surrounding area. In this case, different operators may have different operating speeds, internet bandwidth levels, experience in controlling machines using VR devices, etc. Therefore, for at least some implementations in which commands issued by an operator are remotely submitted using a VR device for a machine to execute in a workflow sequence, the execution timing of the commands on the corresponding machine is important, e.g., to prevent the machine from executing the operations of the workflow sequence out of order. During VR interactions among multiple operators, and due to one or more types of delays associated with the conditions of the remote operator, the machine may experience problems in correctly executing the commands in the workflow sequence. At the same time, delays in submitting or executing commands also affect productivity in the context of the workflow sequence. Therefore, techniques are needed to mitigate the delays in the submitted commands. In addition, even when there are delays in multiple received machine commands, e.g., due to one or more VR devices with relatively low internet bandwidth issuing machine commands, techniques are needed to be able to execute the machine commands in a synchronous manner. Summary of the Invention

[0005] According to one method, a computer-implemented method includes: identifying machines involved in performing a manufacturing process at a manufacturing location, and a workflow sequence of the execution of the machines. Based on these identifications and how the scope relates to operations performed by one or more other machines in the executed workflow sequence, the relative scope of the tasks that a given machine has during the manufacturing process is known. Receiving conditions associated with a remote operator using a virtual reality (VR) device to remotely control the machine to execute the executed workflow sequence at the manufacturing location. The relative internet bandwidth of the VR device is one of these conditions, and by considering these conditions, the scope of the VR collaboration environment displayed by the VR device can be adjusted such that these conditions do not introduce delays into the executed workflow sequence. To prevent such delays, a scope of the VR collaboration environment to be displayed is determined for each of the VR devices. The scope is determined based on the conditions, thereby reducing the delay in the execution of the workflow sequence executed at the manufacturing location. The scope is output to the VR device. This allows the remote operator to remotely operate and control the machine as if the remote operator were actually working at the manufacturing location, without the remote work having to incur the potential hazards of being physically present at the manufacturing location.

[0006] Determining the scope of the VR collaboration environment includes determining a first machine in the machine that is remotely controlled by a first VR device in the VR device having a relatively low Internet bandwidth, and determining a second machine in the machine that is remotely controlled by a second VR device in the VR device having a relatively high Internet bandwidth. As a result of determining this relative difference in Internet bandwidth, the scope of the VR collaboration environment determined and output for display by the first VR device includes relatively less clear content and / or less content than the scope of the VR collaboration environment determined and output for display by the second VR device. Therefore, the scope of the VR collaboration environment determined and output for display by the first VR device reduces the possibility that the first VR device will introduce delays into the executed workflow sequence based on the relatively low Internet bandwidth. This reduction in delay improves the performance of computer devices used in the execution of the executed workflow sequence. This also results in less waste in the production process at the manufacturing site because errors in the production process that would otherwise be caused by such delays are avoided.

[0007] According to another method, a computer program product includes a computer-readable storage medium having program instructions embodied therewith. The program instructions can be read and / or executed by a computer to cause the computer to perform the aforementioned method.

[0008] According to another method, a system includes a processor and logic integrated with the processor, executable by the processor, or integrated and executed by the processor. The logic is configured to perform the aforementioned method.

[0009] Other aspects and methods of the present invention will become apparent from the following detailed description which, when taken in conjunction with the accompanying drawings, illustrates, by way of example, the principles of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a diagram of a computing environment according to a method of the present invention.

[0011] Figure 2A is a flow chart of a method according to the present invention.

[0012] Figure 2B According to a method of the present invention Figure 2A A flowchart of the sub-operations of an operation.

[0013] Figure 2C According to a method of the present invention Figure 2A A flowchart of the sub-operations of an operation.

[0014] Figure 3 is a flow chart of a method according to the present invention. DETAILED DESCRIPTION

[0015] The following description is intended to illustrate the general principles of the invention, rather than to limit the inventive concepts claimed herein. In addition, specific features described herein may be used in combination with other described features in one of various possible combinations and permutations.

[0016] Unless expressly defined otherwise herein, all terms are to be given their broadest possible interpretations, including the meanings implied in the specification and as understood by those skilled in the art and / or defined in dictionaries, treatises, and the like.

[0017] It must also be noted that, as used in the specification and the appended claims, the singular forms "a", "an", and "the" include plural referents unless otherwise indicated. It will also be understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0018] The following description discloses several preferred methods of systems, methods, and computer program products for determining an extent of a virtual reality (VR) environment displayed on a VR device configured to control a machine that performs operations of a workflow sequence.

[0019] In one general method, a computer-implemented method includes identifying machines involved in executing a manufacturing process at a manufacturing location, and identifying a workflow sequence for execution of the machines. Conditions associated with a remote operator are received, wherein the remote operator uses a virtual reality (VR) device to remotely control the machine to perform the workflow sequence for execution at the manufacturing location. The method also includes determining, for each VR device, a range of a VR collaborative environment to be displayed. The range is determined based on the condition, thereby reducing a delay in the execution of the workflow sequence for execution at the manufacturing location. The method also includes outputting the range to the VR device.

[0020] In another general method, a computer program product includes a computer-readable storage medium having program instructions embodied therewith. The program instructions can be read and / or executed by a computer to cause the computer to perform the aforementioned method.

[0021] In another general method, a system includes a processor and logic integrated with the processor, executable by the processor, or integrated with the processor and executable by the processor. The logic is configured to perform the aforementioned method.

[0022] Aspects of the present disclosure are described by narrative text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in computer program product (CPP) embodiments. With respect to any flowchart, depending on the technology involved, operations may be performed in an order different from that shown in a given flowchart. For example, again depending on the technology involved, two operations shown in consecutive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in a manner that at least partially overlaps in time.

[0023] The computer program product method (“CPP method” or “CPP”) is a term used in the present disclosure to describe any collection of one or more storage media (also referred to as “media”) collectively included in a set of one or more storage devices, the set of one or more storage devices collectively including machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. A “storage device” is any tangible device that can hold and store instructions used by a computer processor. By way of non-limitation, computer-readable storage media can be electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, mechanical storage media, or any suitable combination of the foregoing. Some known types of storage devices that include these media include: magnetic disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded devices such as punched cards or pits / lands formed in the main surface of a disk, or any suitable combination of the foregoing. Computer-readable storage media, as the term is used in the present disclosure, should not be construed to store in the form of a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, optical pulses passing through an optical fiber cable, or electrical signals transmitted through wires and / or other transmission media. As will be understood by those skilled in the art, data is typically moved at certain incidental points in time during the normal operation of a storage device, such as during access, defragmentation, or garbage collection, but this does not render the storage device transitory because the data is not transitory when it is stored.

[0024] Computing environment 100 includes an example of an environment for executing at least some of the computer code involved in performing the methods of the present invention, such as the VR environment range determination module of block 200, for determining the range in which a VR environment is displayed on a VR device, which is configured to control a machine that performs operations of a workflow sequence. In addition to block 200, computing environment 100 includes, for example, computer 101, wide area network (WAN) 102, end-user device (EUD) 103, remote server 104, public cloud 105, and private cloud 106. In this method, computer 101 includes a set of processors 110 (including processing circuitry 120 and cache 121), communication structure 111, volatile memory 112, persistent storage 113 (including operating system 122 and block 200, as described above), a set of peripheral devices 114 (including a set of user interface (UI) devices 123, storage device 124, and a set of Internet of Things (IoT) sensors 125), and network module 115. Remote server 104 includes remote database 130. Public cloud 105 includes gateway 140, cloud coordination module 141, a set of host physical machines 142, a set of virtual machines 143, and a set of containers 144.

[0025] Computer 101 may take the form of a desktop computer, laptop computer, tablet computer, smart phone, smart watch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device now known or developed in the future that is capable of running programs, accessing a network, or querying a database such as remote database 130. As is well known in the computer art and depending on the technology, the performance of computer-implemented methods may be distributed among multiple computers and / or among multiple locations. On the other hand, in this presentation of computing environment 100, the discussion focuses in detail on a single computer, specifically computer 101, to keep the presentation as simple as possible. Computer 101 may be located in the cloud, even though Figure 1 it is not shown in the cloud. On the other hand, computer 101 does not need to be in the cloud, unless to any extent that can be definitely indicated.

[0026] The set of processors 110 includes one or more computer processors of any type now known or later developed. The processing circuitry 120 may be distributed across multiple packages, such as multiple cooperating integrated circuit chips. The processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. The cache 121 is a memory located within the processor chip package and is generally used for data or code that should be made available for rapid access by threads or cores running on the set of processors 110. Caches are generally organized into multiple levels based on their relative proximity to the processing circuitry. Alternatively, some or all of the caches of the set of processors may be located "off-chip". In some computing environments, the set of processors 110 may be designed to work with qubits and perform quantum computing.

[0027] Computer-readable program instructions are typically loaded onto the computer 101 so that the set of processors 110 of the computer 101 executes a series of operational steps to implement a computer-implemented method such that the instructions so executed will instantiate the method specified in the flowcharts and / or narrative descriptions of the computer-implemented methods included in this document (collectively referred to as "the method of the present invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as the cache 121 and other storage media discussed below. The program instructions and associated data are accessed by the set of processors 110 to control and direct the execution of the computer-implemented method. In the computing environment 100, at least some of the instructions for performing the computer-implemented method may be stored in block 200 of the persistent storage device 113.

[0028] The communication structure 111 is a signal conduction path that allows the various components of the computer 101 to communicate with each other. Typically, this structure consists of switches and conductive paths, such as switches and conductive paths that make up a bus, a bridge, a physical input / output port, etc. Other types of signal communication paths may be used, such as fiber optic communication paths and / or wireless communication paths.

[0029] The volatile memory 112 is any type of volatile memory now known or later developed. Examples include dynamic random access memory (RAM) or static RAM. Generally, volatile memory 112 is characterized by random access, but this is not required unless affirmatively indicated. In the computer 101, the volatile memory 112 is located within a single package and inside the computer 101, however, alternatively or additionally, the volatile memory may be distributed across multiple packages and / or be located external to the computer 101.

[0030] The persistent storage device 113 is any form of non-volatile storage for a computer, now known or developed in the future. The non-volatility of this memory means that the stored data is retained regardless of whether power is supplied to the computer 101 and / or directly to the persistent storage device 113. The persistent storage device 113 can be read-only memory (ROM), but typically at least a portion of the permanent memory allows for the writing of data, the deletion of data, and the re-writing of data. Some common forms of persistent storage include magnetic disks and solid-state storage devices. The operating system 122 can take several forms, such as various known proprietary operating systems or operating systems of the open-source portable operating system interface type that employ a kernel. The code included in block 200 generally includes at least some of the computer code involved in performing the method of the present invention.

[0031] The set of peripheral devices 114 includes the set of peripheral devices of the computer 101. Data communication connections between the peripheral devices and other components of the computer 101 can be implemented in various ways, such as Bluetooth connections, near-field communication (NFC) connections, connections made by cables (such as Universal Serial Bus (USB)-type cables), plug-in connections (e.g., Secure Digital (SD) cards), connections made through a local communication network, and even connections made through a wide-area network such as the Internet. In various embodiments, the set of UI devices 123 can include components such as a display screen, speakers, microphones, wearable devices (such as goggles and smartwatches), keyboards, mice, printers, touchpads, game controllers, and haptic devices. The storage device 124 is an external storage device, such as an external hard disk drive, or a plug-in storage device, such as an SD card. The storage device 124 can be permanent and / or volatile. In some embodiments, the storage device 124 can take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where the computer 101 needs to have a large amount of storage (e.g., in the case where the computer 101 locally stores and manages a large database), the storage can be provided by a peripheral storage device (such as a storage area network (SAN) shared by multiple geographically distributed computers) designed to store a very large amount of data. The set of IoT sensors 125 consists of sensors that can be used in Internet of Things applications. For example, one sensor can be a thermometer, while another sensor can be a motion detector.

[0032] The network module 115 is a collection of computer software, hardware, and firmware that allows the computer 101 to communicate with other computers via the WAN 102. The network module 115 can include hardware such as a modem or a Wi-Fi signal transceiver, software for packetizing and / or depacketizing data transmitted over a communication network, and / or web browser software for transmitting data over the Internet. In some embodiments, the network control function and the network forwarding function of the network module 115 are executed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software-defined networking (SDN)), the control function and the forwarding function of the network module 115 are executed on physically separate devices such that the control function manages several different network hardware devices. Computer-readable program instructions for performing computer-implemented methods can generally be downloaded to the computer 101 from an external computer or an external storage device via a network adapter or network interface included in the network module 115.

[0033] The WAN 102 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances via any technology known now or developed in the future for transmitting computer data. In some embodiments, the WAN 102 can be replaced and / or supplemented by a local area network (LAN) designed to transmit data between devices located in a local area (e.g., a Wi-Fi network). The WAN and / or LAN generally includes computer hardware such as copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers.

[0034] The end-user device (EUD) 103 is any computer system used and controlled by an end user (e.g., a customer of an enterprise operating the computer 101) and can take any form discussed above in connection with the computer 101. The EUD 103 typically receives useful and available data from the operation of the computer 101. For example, in the hypothetical case where the computer 101 is designed to provide recommendations to an end user, the recommendation will typically be transmitted from the network module 115 of the computer 101 to the EUD 103 via the WAN 102. In this way, the EUD 103 can display or otherwise present the recommendation to the end user. In some embodiments, the EUD 103 can be a client device such as a thin client, a thick client, a mainframe computer, a desktop computer, etc.

[0035] The remote server 104 is any computer system that provides at least some data and / or functionality to the computer 101. The remote server 104 may be controlled and used by the same entity that operates the computer 101. The remote server 104 represents a machine that collects and stores useful and available data used by other computers such as the computer 101. For example, in the hypothetical case where the computer 101 is designed and programmed to provide recommendations based on historical data, that historical data may be provided to the computer 101 from the remote database 130 of the remote server 104.

[0036] The public cloud 105 is any computer system that is available for use by multiple entities, which provides on-demand availability of computer system resources and / or other computer capabilities (notably data storage (cloud storage) and computing power) without direct active management by the user. Cloud computing typically exploits the sharing of resources to achieve consistency and economy of scale. The direct and active management of the computing resources of the public cloud 105 is performed by the computer hardware and / or software of the cloud coordination module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers of a set of host physical machines 142, which is the universe of physical computers within and / or available for the public cloud 105. The virtual computing environment (VCE) typically takes the form of virtual machines from a set of virtual machines 143 and / or containers from a set of containers 144. It should be understood that these VCEs may be stored as images and may be transferred between various physical machine hosts as images or after instantiation of the VCE. The cloud coordination module 141 manages the transfer and storage of images, deploys new instantiations of the VCE, and manages the active instantiations of the VCE deployment. The gateway 140 is a collection of computer software, hardware, and firmware that allows the public cloud 105 to communicate via the WAN 102.

[0037] Some further explanations of the virtualized computing environment (VCE) will now be provided. The VCE may be stored as an "image". New active instances of the VCE may be instantiated from this image. Two common types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature where the kernel allows the existence of multiple isolated user space instances, called containers. From the perspective of the programs running within them, these isolated user space instances typically appear as actual computers. A computer program running on a normal operating system may utilize all the resources of that computer, such as connected devices, files and folders, network shares, CPU capabilities, and quantifiable hardware capabilities. However, a program running within a container can only use the contents of the container and the devices assigned to the container, which is a feature known as containerization.

[0038] The private cloud 106 is similar to the public cloud 105, except that the computing resources are only available for use by a single enterprise. Although the private cloud 106 is depicted as communicating with the WAN 102, in other embodiments, the private cloud can be completely disconnected from the Internet and only accessible through a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types) that are typically implemented by different vendors. Each of the multiple clouds remains an independent and discrete entity, but the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable coordination, management, and / or data / application portability between the multiple constituent clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of the larger hybrid cloud.

[0039] In some aspects, a system according to various embodiments can include a processor and logic integrated with and / or executable by the processor, the logic being configured to perform one or more of the process steps described herein. The processor can be any configuration as described herein, such as a discrete processor or a processing circuit including many components such as processing hardware, memory, I / O interfaces, etc. By being integrated with it, it means that the processor has logic embedded with it as hardware logic, such as an application-specific integrated circuit (ASIC), an FPGA, etc. By being executable by the processor, it means that the logic is hardware logic; software logic, such as firmware, a part of the operating system, a part of an application; etc., or some combination of hardware and software logic that can be accessed by the processor and is configured to cause the processor to perform a certain function when executed by the processor. The software logic can be stored on local and / or remote memory of any memory type, as is well known in the art. Any processor known in the art can be used, such as a software processor module and / or a hardware processor, such as an ASIC, an FPGA, a central processing unit (CPU), an integrated circuit (IC), a graphics processing unit (GPU), etc.

[0040] Of course, according to various methods, the logic can be implemented as a method or a computer program product on any device and / or system.

[0041] As described elsewhere above, virtual reality (VR) is a three-dimensional computer-generated environment that can be explored and interacted with by a person. This person becomes part of the virtual world and / or immersed in this environment, and at the same time is able to manipulate objects and / or perform a series of actions here. VR benefits the manufacturing sector by enhancing worker safety, improving product production, saving money for manufacturers, enabling remote control of machine operations, allowing monitoring of various machines on industrial sites, etc. More specifically, in some cases, VR enables the operation and monitoring of machines to be remotely controlled by an operator using a VR device from one or more remote locations, e.g., starting, stopping, changing the operating parameters of the machine, etc.

[0042] In any industrial site, there may be different machines located in and operating in different parts of the industrial site. Here, the machines can cooperate with each other to perform a common task or execute any manufacturing workflow. For context, a "workflow" can be defined as one or more operations performed to complete a predetermined task. More specifically, such a task can be completed as a result of one or more machines performing machine operations according to a "workflow sequence". Within this workflow sequence, one or more machines perform machine operations in an interconnected manner in parallel and / or sequentially. In some cases, each machine can be assigned a specific work task to perform, while in some other cases, multiple machines can additionally and / or alternatively operate together to perform a single machine operation. In any industrial site, the relative positions and orientations of these machines can be monitored and controlled to align with the machine's workflow program sequence.

[0043] For example, a person hereinafter referred to as a "remote operator" can use a VR device to remotely control a machine to perform a machine operation. For example, the remote operator can wear VR glasses of a known type to view the robotic arm of the machine and use a (one or more) handheld control to control the robotic arm. When remotely controlling, monitoring, or operating any machine on any industrial site, the remote operator can visualize the target machine in the corresponding "VR environment", where the remote operator can remotely view the perspective of the machine actually located on the industrial site. At the same time, various other remote operators can also remotely control, monitor, or operate other machines in the surrounding area. In this case, different operators can have different operation speeds, Internet bandwidth levels, experience in using VR devices to control machines, etc. Therefore, for at least some implementations in which commands issued by an operator are remotely submitted using a VR device for a machine to execute in a workflow sequence, the execution timing of the commands on the corresponding machine is important, for example, to prevent the machine from executing the operations of the workflow sequence out of order. During VR interactions between multiple operators, and due to one or more types of delays associated with the conditions of the remote operator, the machine may experience problems in correctly executing the commands in the workflow sequence. At the same time, the delay in submitting or executing commands also affects productivity in the context of the workflow sequence. Therefore, techniques are needed to mitigate the delay in the submitted commands. In addition, even if there are delays in multiple received machine commands, for example, due to one or more VR devices with relatively low Internet bandwidth issuing machine commands, techniques are needed to be able to execute the machine commands in a synchronous manner.

[0044] In sharp contrast to the various deficiencies described above, the techniques of the various methods described herein include identifying the workflow sequence performed by machines involved in the manufacturing process of an industrial site. Then, based on the conditions associated with a remote operator remotely controlling a machine using a VR device, such as the Internet bandwidth available to different remote operators performing activities from a remote location using the VR device, the scope of the VR collaboration environment to be displayed on the VR device is determined and output. In this way, the latency in the execution of the workflow sequence performed at the manufacturing location can be minimized. For example, a first operator may have access to a relatively slow Internet bandwidth, while a second operator may have access to a relatively fast Internet bandwidth. Thus, for the first operator, only the target machine actively controlled by the first operator can be shown within the scope of the VR environment displayed on the first operator's VR device. Meanwhile, based on the second operator having access to a relatively fast Internet bandwidth, each machine involved in the manufacturing process can be shown within the scope of the VR environment displayed on the second operator's VR device.

[0045] Now referring to Figure 2A , a flowchart of method 201 according to one method is shown. According to the present invention, method 201 can be executed in any environment described in Figures 1-3 , where it can be executed in various ways. Of course, as those skilled in the art will understand upon reading this specification, method 201 may include more or fewer operations than those specifically described in Figure 2A .

[0046] Each step of method 201 can be performed by any suitable component of the operating environment. For example, in various methods, method 201 can be performed partially or fully by a computer or some other device having one or more processors therein. A processor (e.g., a processing circuit, chip, and / or module implemented in hardware and / or software and preferably having at least one hardware component) can be utilized in any device to perform one or more steps of method 201. Illustrative processors include, but are not limited to, a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., combinations thereof, or any other suitable computing device known in the art.

[0047] Operation 202 includes identifying machines involved in performing a manufacturing process at a manufacturing location, such as machine type, machine identification, machine set, etc. In some methods, the identification is performed by sending a query requesting the operation responsibility of each machine to multiple machines. In another method, the machines can be identified from network information. In yet another method, the machines can be identified by observing the manufacturing location and determining which machines present at the manufacturing location are actively involved in performing machine operations related to the manufacturing process of the product. The machines can be of known types, such as robots, welding machines, sewing machines, hammering machines, molding machines, etc., which are configured to receive commands, for example, issued by a VR device controlled by a remote operator, and perform one or more machine operations in response to the command.

[0048] For context, the VR device can include VR viewing devices of known types, such as, augmented reality (AR) glasses, device displays, VR glasses, front camera devices with display components, etc. In some preferred methods, the VR device is configured to display a perspective view of the manufacturing location, for example, a perspective view viewed by one or more cameras mounted above the machine, mounted to the machine, integrated in the machine, etc. As will be described in more detail elsewhere herein, the manufacturing location can be modeled as a VR collaborative environment for display on one or more VR devices. More specifically, each VR collaborative environment can be a representation of the manufacturing site, which can include all or less than all of the content actually present at the manufacturing location, such as machines, obstacles, clarity, light, etc. For example, the VR collaborative environment displayed on a given one of the VR devices can be modeled with respect to the perspective seen by the camera of the associated machine controlled by the VR device at any given time. In this way, a remote operator wearing the VR device can operate and control the machine as if the remote operator were actually working at the manufacturing location, while avoiding any potential hazards physically present at the manufacturing location.

[0049] It is possible to identify the workflow sequence executed by a machine. For example, refer to operation 204. As described elsewhere herein, a "workflow" can be defined as one or more operations performed to complete a predetermined task, where the operations performed establish the "executed workflow sequence". More specifically, these tasks can be completed as a result of one or more machines performing machine operations according to the "workflow sequence". Within this workflow sequence, one or more machines perform machine operations in a parallel and / or sequential manner in an interconnected way. In some cases, each machine can be assigned a specific work task to perform, while in some other cases, multiple machines can additionally and / or alternatively operate together to perform a single machine operation. At any industrial site, the relative position and orientation of the machines can be monitored and controlled to align with the workflow sequence of the machines. Thus, in some methods, the workflow sequence can be identified by monitoring the operations of the machines, such as monitoring a predetermined amount of time until it is determined that a product has been produced for a predetermined amount of time, until the operation responsibilities of at least a predetermined number of machines have been observed and determined, and so on. In some other methods, the workflow sequence can be identified based on commands issued to the machines from a VR device controlled by a remote operator. For example, the workflow sequence can include each machine command issued by the remote operator via the VR device to the machines.

[0050] Receive conditions associated with a remote operator remotely controlling a machine using a VR device to fulfill an executed workflow sequence at a manufacturing location. For example, refer to operation 206. For context, in some preferred methods, these conditions associated with a remote operator using a VR device are conditions that may cause delays in transmitting commands via the VR device to perform machine operations. For example, in one preferred method, the condition is the relative internet bandwidth of the VR device, such as the internet bandwidth available to the VR device, the internet bandwidth to which one or more VR devices are connected, the internet bandwidth through which the VR device outputs commands for the machine to perform machine operations, etc. For context, different VR devices with different internet bandwidths may introduce delays in the command transmission via the VR device based on the order of received commands not matching the order of previously sent commands. For the purpose of illustration, it can be assumed that the first VR device has a relatively smaller internet bandwidth than the second VR device. Based on this bandwidth difference, the first VR device can issue the first command before the second VR device issues the second command, but the second command can be received before the first command is received. In some other methods, the conditions can additionally and / or alternatively include, for example, the operator's safety record, the amount of time since the operator last logged off from work, the operator's relative skill level and / or proficiency level, and so on.

[0051] In some methods, to account for these conditions and thereby reduce the latency in the execution of the workflow sequence performed at the manufacturing location, the scope of the VR collaboration environment displayed on the VR device can be determined based on these conditions. For example, operation 208 includes determining, for each of the VR devices, the scope of the VR collaboration environment to be displayed. For context, as described below, accounting for these conditions can include customizing a unique scope of the VR collaboration environment for each VR device to ensure that the conditions do not introduce latency into the workflow sequence being executed at the manufacturing location. This thereby reduces the latency in the execution of the workflow sequence performed at the manufacturing location, which would otherwise exist without using the techniques described herein. The reduction in latency improves the performance of the computing devices used in the execution of the workflow sequence. This also results in less waste in the production process at the manufacturing location, as errors in the production process that would otherwise result from such latency are avoided.

[0052] Various illustrative techniques for determining the scope of the VR collaboration environment will now be described below. For example, referring to Figure 2B , an example sub - operation for determining the scope of the VR collaboration environment is shown according to one method, one or more of which can be used to perform Figure 2A operation 208. However, it should be noted that Figure 2B the sub - operations are illustrative of one method and are in no way intended to limit the present invention.

[0053] Now referring to Figure 2B, shows a sub - process where bandwidth conditions are considered to determine the scope of a VR collaboration environment in order to optimize VR collaboration effectiveness and manage latency in the execution of a workflow sequence. Sub - operation 240 includes determining a first machine in the machine that is remotely controlled by a first VR device in the VR devices, where the first VR device has a relatively low Internet bandwidth, e.g., relatively high latency. This determination can be based on comparing the Internet bandwidth with the Internet bandwidth of one or more other VR devices (e.g., a second VR device). In sub - operation 242, a second machine in the machine that is remotely controlled by a second VR device in the VR devices is determined, where the second VR device has a relatively high Internet bandwidth, e.g., relatively low latency. In some methods, to mitigate the latency otherwise caused by the first VR device outputting machine commands using a relatively low Internet bandwidth, the scope of the VR collaboration environment can be reduced, e.g., to include only one or more relevant parts of the VR collaboration environment. For example, the machine remotely controlled by the first VR device, e.g., the target device, is preferably considered relevant, while the machines not remotely controlled by the first VR device can be considered irrelevant, e.g., non - target devices. For example, in sub - operation 244, the first machine is included in the scope of the VR collaboration environment for the first VR device having a relatively low Internet bandwidth, and the second machine is not included. This inclusion and exclusion (respectively) can be performed in response to determining that the first VR device has a relatively low Internet bandwidth. Since the second machine is not included in the scope of the VR collaboration environment for the first VR device having a relatively low Internet bandwidth, potential bandwidth is conserved. This translates to a relative reduction in the latency of the execution of the workflow sequence performed at the manufacturing location because the first VR device receives a relatively smaller bandwidth - intensive scope of the VR collaboration environment, allowing the output of machine commands to be relatively more efficient, e.g., without delay. Conversely, sub - operation 246 includes including the first machine and the second machine in the scope of the VR collaboration environment of the second VR device in response to determining that the second VR device has a relatively high Internet bandwidth. Note that the second VR device can access a relatively larger Internet bandwidth and thus can bear the cost associated with downloading the additional scope of the VR collaboration environment without increasing the latency of the execution of the workflow sequence performed at the manufacturing location.

[0054] It should be noted that although various methods are described herein from the perspective of the first VR device and the second VR device, in some other methods, method 201 can optionally include a hierarchical scope of the VR collaboration environment for more than two VR devices based on multiple relative bandwidth speeds, e.g., the relative bandwidth speeds of five VR devices, fifty VR devices, one hundred VR devices, etc.

[0055] Continue to refer to Figure 2B, in some methods, alternatives for including some machines and excluding other machines within the scope of the VR collaboration environment include increasing the relative clarity of some machines (e.g., target machines) and decreasing the relative clarity of some machines (e.g., target machines within the VR collaboration environment). For example, an alternative method for determining the scope of the VR collaboration environment may include, in response to determining that the first VR device has a relatively low internet bandwidth, specifying that the first machine has a relatively higher clarity within the scope of the VR collaboration environment of the first VR device than the second machine. Additionally, in response to determining that the second VR device has a relatively high internet bandwidth, specifying that the first machine has approximately the same clarity as the second machine within the scope of the VR collaboration environment of the second VR device.

[0056] Now refer to Figure 2C , according to one method, example sub-operations for determining the scope of the VR collaboration environment are shown, one or more of which can be used to perform Figure 2A the operation 208. However, it should be noted that Figure 2C the sub-operations are illustrated according to one method, which is in no way intended to limit the present invention.

[0057] Now refer to Figure 2C , in some methods, based on the level of available bandwidth, the proposed VR collaboration technology can identify which remote activities may be restricted for one or more remote operators with a relatively low level of available bandwidth. As a result, the bandwidth of at least some of these VR devices of these different remote operators can be reserved. In response to this, work can be dynamically assigned to different staff members, e.g., other staff members with a relatively high level of available bandwidth. Based on the internet bandwidth available to different VR devices, in the case where different staff members remotely initiate the execution of activities in a collaborative manner, the proposed technology can dynamically assign different work to different remote operators and can also identify which work assignments will be disabled and / or re-assigned. For example, it can be determined that a first machine in the machines is remotely controlled by a first VR device with a relatively low internet bandwidth among the VR devices, e.g., see sub-operation 240. Additionally and / or optionally, it can be determined that a second machine in the machines is remotely controlled by a second VR device with a relatively high internet bandwidth among the VR devices, e.g., see sub-operation 242. In some methods, based on one or more of these determinations, at least some of the work tasks currently assigned to a first remote operator among the remote operators using the first VR device can be re-assigned to a second remote operator among the remote operators using the second VR device, e.g., see sub-operation 248. In an alternative method, based on one or more of these determinations, at least some of the work tasks currently assigned to the first remote operator can additionally and / or alternatively be cancelled.

[0058] In some alternative methods, method 201 may include training an artificial intelligence (AI) model to determine and / or update the scope of a VR collaboration environment. More specifically, in one or more such methods, the inputs to the trained model may include information such as conditions, the current assignment of work tasks, any other information for determining the scope described elsewhere herein, etc., and the output of the trained model may include the scope. In some methods, a subject matter expert (SME) may examine the inputs and outputs of the model and apply rewards and / or corrective feedback as part of the training process.

[0059] As will be described elsewhere herein, the determination and / or update of the scope of a VR collaboration environment may additionally and / or alternatively be based on predetermined safety criteria, e.g., see judgment 228 - operation 232.

[0060] Referring again to Figure 2A , the determined scope is output to an associated VR device in the VR device, e.g., see operation 210.

[0061] The VR remote collaboration techniques described herein may additionally and / or alternatively evaluate the time delay of receiving commands from VR devices with different latencies in the communication. In some methods, based on the delay, one or more operations may be performed to verify the operations with that delay using the workflow sequence of the execution of the machine. In this way, it is ensured that the commands executed in different machines are corrected to the appropriate execution time, e.g., an execution time that does not cause the order of the workflow sequence executed to be disordered due to the delay.

[0062] In some methods, timestamps associated with machine commands for different machines received from different VR devices can be analyzed. Based on this analysis, in response to determining that the timestamps deviate from the actual execution sequence, a correction can be performed on the sequence. Additionally, in response to this, additional buffer delays can be incorporated into the workflow sequence being executed to allow for a relatively smooth (e.g., delay-free) execution of operations. For the purpose of an illustrative example, it can be assumed that a first command is received from a first VR device to perform a first machine operation, e.g., see operation 212. A second command can be received from a second VR device to perform a second machine operation, e.g., see operation 214. In the current method, it can be assumed that the first command is received before the second command. The timestamp information of the commands can be analyzed to determine whether the second command was output by the second VR device before the first command was output by the first VR device, e.g., see operation 216. For example, techniques that would be understood by a person of ordinary skill in the art upon reading the description herein can be used to compare the timestamp information to determine which commands were output by the VR devices first. For context, this analysis can be performed to determine whether Internet bandwidth latency is at least one of the reasons for receiving the first command before the second command. More specifically, the analysis can be performed to determine whether the second VR device initiated the output of the second command before the first VR device initiated the output of the first command, but due to the relatively high Internet bandwidth of the first VR device, the first command was received before the second command. In other words, the second machine operation can actually be performed before the first machine operation in the workflow sequence being executed.

[0063] Operation 218 includes causing the machine operations to be performed in an order determined based on the analysis of the timestamp information. In one method, in response to determining based on the analysis of the timestamp information that the second command was output by the second VR device before the first command was output by the first VR device, the second machine operation can be caused to be performed before the first machine operation. This operation mitigates the delays that occur in the order of receiving the commands from being incorporated into the order in which the machines perform the machine operations. In some methods, the analysis can additionally and / or alternatively compare the timestamp information of the same VR device and / or machine. For the purpose of an illustrative example, it can be assumed that a first operator has submitted a machine command for a first machine at time T1, and thereafter a machine command for a second machine is received from the first operator's VR device at time T2. In some methods, ideally, the second machine command should be executed first. Thus, method 201 can include causing the correction to be made, e.g., issuing an instruction to issue the second machine command before the first machine command.

[0064] In some methods, one or more remote operators may have a scope of job responsibilities that enables the remote operator to operate with more than one machine at a manufacturing location using a VR device. Thus, in some methods, the determination and / or update of the scope of a VR collaboration environment may additionally and / or alternatively be based on which machines are currently being remotely controlled by the VR device. For example, a remote operator may need to perform activities with multiple machines from time to time, and thus, based on the virtual mobility of the personnel in the VR collaboration environment, the clarity levels of different parts of the VR collaboration environment (e.g., based on the order of activities of the personnel) may change from time to time. Thus, it can be determined whether a VR device, such as a first VR device that currently controls a first machine, has obtained control of a second machine, e.g., see determination 220. In one example, it can continue to be assumed that the first machine is remotely controlled by a VR device with a relatively low Internet bandwidth, and the second machine is remotely controlled by a VR device with a relatively high Internet bandwidth. Additionally, it can be assumed that the first VR device initially controls the first machine and does not control the second machine. In some methods, in response to determining that the first VR device has obtained control of the second machine, e.g., as part of a work sequence, the remote operator of the first VR device is authorized to control the second machine, and the relative clarity of the second machine can be increased in an updated version of the scope of the VR collaboration environment of the first VR device, e.g., see operation 222. Additionally, in response to determining that the first VR device has obtained control of the second machine, the relative clarity of the first machine can be decreased in an updated version of the scope of the VR collaboration environment of the first VR device, e.g., see operation 224. The updated version of the scope of the VR collaboration environment of the first machine can be output to the first VR device, e.g., see operation 226. For the purpose of examples that include more than two machines, it can be assumed that a first operator first operates machine A via a first VR device, but in the surrounding area at the manufacturing location, e.g., within a predetermined proximity, there are also other machines, such as machines B, C, D, E, etc. Thus, in response to determining that the first VR device experiences relatively poor bandwidth, the scope of the VR collaboration environment of the first VR device can include machine A with the highest clarity, while the remaining machines are included with relatively the lowest clarity. Thereafter, in response to determining that the first operator has a next activity scheduled on machine B and / or the first VR device has obtained control of machine B, machine B can be included in the scope of the VR collaboration environment of the first VR device with relatively the highest clarity.

[0065] In some methods, one or more areas at a manufacturing location can be subject to one or more predefined safety standards. These predefined safety standards can be set by, for example, a manager at the manufacturing location, regulatory laws, a safety officer at the manufacturing location, etc., and can be enforced to ensure that machine operations are relatively safe while collaboratively performing an executed workflow sequence. For example, when building a collaborative VR environment for different VR devices with varying Internet bandwidths, some methods can include identifying whether information about the industrial site surroundings is also included within the scope of the VR collaborative environment along with the target machine. Thus, the clarity of different parts of the VR collaborative environment can be determined based on the degree of safety required in different areas at the manufacturing location and included within the scope of the VR collaborative environment. For example, a first area at the manufacturing location may contain hazardous waste and thus be subject to a safety protocol that specifies that machines within the first area maintain a predefined proximity to each other. This can help ensure that collisions do not occur between two or more machines, which could otherwise result in a hazardous waste spill event. Thus, in such instances, it is important for a remote operator controlling a machine within the first area to view the scope of the VR collaborative environment that includes the machine the remote operator is controlling as well as other machines within the first area that may enter within a predefined proximity of the first machine. For the purpose of another example, it can be assumed that a predefined safety standard applies to a first area at the manufacturing location. Decision 228 includes determining whether the first machine has entered an area to which the predefined safety standard applies. In response to determining that the first machine (e.g., the target device of the first VR device) remotely controlled by the first VR device has entered and / or is scheduled to enter the first area, for example, as shown by the "yes" logic path of decision 228, a predefined range of relative clarity of the peripheral area of the first machine can be increased in an updated version of the scope of the VR collaborative environment for the first VR device, for example, see operation 230. In the view of the machine, one or more other areas at the manufacturing location to which the predefined safety standard does not apply can have reduced associated clarity in the updated version of the scope of the VR collaborative environment for the first VR device. The updated version of the scope of the VR collaborative environment for the first machine can be output to the first VR device, for example, see operation 232. Conversely, in response to determining that the first machine has not entered an area to which the predefined safety standard applies, for example, as shown by the "no" logic path of decision 228, the method optionally ends, for example, see "end". Note that although the method is described as optionally ending, in some other methods, additional monitoring can be performed. For example, throughout the executed workflow sequence of the machine, monitoring of areas at the manufacturing location that are subject to one or more predefined safety rules can be performed.

[0066] In some other methods, in addition to and / or as an alternative to one or more predetermined areas of the manufacturing site, one or more machines can be made to specifically comply with predetermined safety standards. For example, assuming that a remote operator needs to view the entire surrounding environment of a target machine, when initiating a command to be executed on the target machine, the associated range of the VR collaboration environment can be made to have a relatively high clarity. Then, based on the required security level at the manufacturing location, the clarity level can vary across the entire range of the VR collaboration environment. For example, in some methods, the target machine remotely controlled by the VR device of the remote operator can be set to have a relatively highest clarity, and the surrounding machines and / or the remote operator subject to relatively limited safety parameters may cause other machines to have a relatively low clarity within the range of the VR collaboration environment.

[0067] In some additional methods, the presence of personnel at the manufacturing location may result in the application of some additional safety standards. For example, personnel within a predetermined proximity of a machine that performs machine operations according to an executed workflow sequence may pose a safety threat to the personnel. Therefore, in some methods, in response to determining that personnel have entered within a predetermined distance of the machine, the relative clarity of the personnel can be increased within the range of the VR collaboration environment output to a VR device with remote control of the machine. In some other methods, in response to determining that personnel have entered a predetermined area at the manufacturing location that complies with one or more predetermined safety standards, the relative clarity of the personnel can be increased within the range of the VR collaboration environment output to a VR device with remote control of one or more machines that are also in the predetermined area.

[0068] It should be noted that although the various operations described consider conditions relative to Internet bandwidth, such operations can be performed additionally and / or alternatively with respect to one or more other conditions. For example, it can be assumed that the conditions include relative operator experience. In response to determining that the first of the remote operators is not relatively very experienced, method 201 can include reassigning at least some of the work tasks currently assigned to the first of the remote operators using the first VR device to the second of the remote operators using the second VR device. In yet another method, it can be additionally and / or alternatively assumed that the conditions include whether a given machine is performing a machine operation on a predetermined fragile product part. In response to determining that the first machine is performing a machine operation using the predetermined fragile product component, method 201 can include increasing the relative clarity of the scope of the VR collaboration environment output to the VR device remotely controlling the first machine. In some other methods, it can be additionally and / or alternatively assumed that the conditions include the relative safety record of the remote operator. For example, the relative safety record of the first remote operator using the first VR device to control the first machine can be monitored. In response to determining that the safety record is below a predetermined threshold, method 201 can include increasing the relative clarity of the scope of the VR collaboration environment output to the first VR device and / or reassigning at least some of the work tasks currently assigned to the first remote operator.

[0069] Implementing the techniques described herein at a manufacturing location realizes many benefits. For example, conditions such as different relative Internet bandwidths of VR devices are prevented from introducing delays into the execution of the workflow sequence performed at the manufacturing location. This is because these conditions are considered by customizing the unique scope of the VR collaboration environment for each VR device to ensure that these conditions do not introduce delays into the workflow sequence being executed at the manufacturing location. Thus, the operations described herein allow machine operations to be performed efficiently and in some cases synchronously according to the workflow sequence, where the operations would otherwise be performed asynchronously due to delays remaining unaccounted for. This thereby reduces the delay in the execution of the workflow sequence performed at the manufacturing location, which would otherwise be present without using the techniques described herein. The reduction in delay improves the performance of the computer devices used in the execution of the workflow sequence. This also results in less waste in the production process at the manufacturing location, as errors in the production process due to such delays are avoided. It should also be noted that determining the scope of the VR collaboration environment displayed on the VR device based on conditions has not been considered in traditional applications heretofore. Instead, conditions that cause delays remain unaccounted for, which undermines manufacturing process efficiency and product quality. Thus, the invention disclosed herein regarding determining the scope of the VR collaboration environment displayed on the VR device based on conditions runs counter to conventional wisdom.

[0070] Now refer to Figure 3, shows a flowchart of method 300 according to a method. Method 300 may be performed in any environment described in Figures 1-3 , including various methods. Of course, as those skilled in the art will understand when reading this specification, method 300 may include more or fewer operations than those specifically described in Figure 3 .

[0071] Each step of method 300 may be performed by any suitable component of the operating environment. For example, in various methods, method 300 may be performed partially or completely by a computer or some other device having one or more processors. A processor (e.g., a processing circuit, chip, and / or module implemented in hardware and / or software and preferably having at least one hardware component) may be utilized in any device to perform one or more steps of method 300. Illustrative processors include, but are not limited to, a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., combinations thereof, or any other suitable computing device known in the art.

[0072] First of all, it should be noted that Figure 3 the flowchart of method 300 discusses different remote operators who are able to cooperate with each other and operate machines at a manufacturing location (e.g., an industrial workshop). In some methods, the operation of machines on an industrial site may be based on changing configurations, changing operating parameters, start commands, stop commands, or any other commands, and / or based on the available Internet bandwidth of different VR devices used by different remote operators. Various operations of method 300 may be used to identify the scope of the VR environment, which is output for display on different VR devices.

[0073] Method 300 may be implemented on any industrial site where one or more different types of machines perform activities according to a workflow sequence, and where different machines have different specifications. In some methods, when a machine is performing an activity, the output of one machine may be used as the input for another machine performing an activity according to the workflow. In this industrial setting, the workflow sequence of the manufacturing process may be identified, and the roles of the machines may be identified, e.g., see operations 302 and 304 of the initialization series of operation 310. For context, the role of a machine is determined by the activity performed and its importance to the workflow. In some methods, these roles may be determined based on the work tasks currently assigned to remote operators using VR devices to control the machines. For example, Figure 3Shown are a plurality of remote operators each wearing a VR glasses device. For example, see remote operator 314, remote operator 316, remote operator 318, and remote operator 320. Using the VR device, the staff can remotely configure, control, and change the operating parameters of different associated machines. Additionally and / or alternatively, the relative positions and orientations of different machines on the industrial site can be identified, for example, the direction and footprint in which the machine operates during the process of performing the machine operation.

[0074] When the machine is performing an activity, the performance, capabilities, operating parameters, etc. of the machine can be considered and associated with the VR content and the configuration specified by the activity, for example, see operation 306. The VR content can be based on the identified scope of the surrounding environment, which will be output and displayed on different VR devices at a threshold level of clarity, for example, see operation 308. In this way, the machine operation can be synchronized and executed according to the workflow sequence. However, it should be noted that different conditions associated with the VR device controlling the machine may introduce delays into the execution of the workflow sequence. As will be described below, method 300 may include performing operations to reduce the delays incorporated into the execution of the workflow sequence.

[0075] It can be identified which machine is being controlled from a remote location by the VR device. Additionally, the available Internet bandwidth of different remote VR devices can be identified, for example, see operation 322. These conditions and outputs of the initialization operation series 310 can be input into the VR collaboration system 312, which can be a trained AI model (as described elsewhere in this document). The scope of the VR collaboration environment to be displayed can be determined for each of the VR devices, where the scope is determined based on the conditions, for example, see operation 324. By determining these scopes based on these conditions, the delay in the execution of the workflow sequence performed at the manufacturing location is reduced. In some methods, determining the scope of the VR collaboration environment includes identifying the manufacturing process and identifying which machines should be processed in sequence and performing a timestamp. In other words, in cases where the conditions introduce delays and cause the operations to be executed out of order, the order of operations of the executed workflow sequence can be determined and established as a reference. Additionally and / or alternatively, the amount of bandwidth required for different types of VR collaboration can be determined in order to determine the scope of the VR collaboration environment. For example, work tasks to be executed may not be issued to VR devices that do not have at least a predetermined minimum Internet bandwidth threshold for the workflow sequence to be executed. In some methods, the operations can include estimating the bandwidth requirements and also using historical learning about the required bandwidth of the executed workflow sequence.

[0076] In some methods, the scope of the VR collaboration environment to be displayed can be determined by analyzing the workflow sequence of the manufacturing process and identifying which machines are being actively remotely controlled by the VR device. In this way, the work task re-specification can be output to the VR devices available and active at the manufacturing location.

[0077] In some other methods, the determination of the scope of the VR collaboration environment can consider one or more safety factors to which one or more machines and / or areas at the manufacturing location are subject. For example, based on the safety level to which one or more machines and / or areas at the manufacturing location are subject, one or more machines can be included in the scope of the VR collaboration environment with relatively high clarity. Conversely, machines that are not present in areas at the manufacturing location that are affected by safety factors may not be included in the scope of the VR collaboration environment, or may be included with relatively low clarity.

[0078] In some methods, the types of activities assigned to different remote operators can be based on the available Internet bandwidth. For example, see operation 326. For example, in response to determining that a remote operator is using a VR device with access to relatively low Internet bandwidth, relatively few work tasks and / or relatively less resource-intensive work tasks can be assigned to the remote operator. Additionally, these work task assignments can be changed at any time, for example, in response to determining that a VR device that previously had relatively low Internet bandwidth now has access to relatively high Internet bandwidth.

[0079] Operation 328 includes reordering machine commands in response to determining that commands received from different VR devices include latency. For example, one or more operations of method 300 can include receiving timestamps associated with commands received from different remote operators. The timestamps can be evaluated relative to the manufacturing process sequence to determine whether machine commands are received out of order based on the latency. In response to determining that machine commands are received out of order, one or more reordering instructions can be issued to correct the command execution timeline. In this way, the latency is reduced, and the machine execution sequence is aligned with the manufacturing process.

[0080] It is clear that the various features of the foregoing systems and / or methods can be combined in any way, resulting in multiple combinations from the description presented above.

[0081] It should also be understood that the method of the present invention can be provided in the form of a service representing a customer deployment, so as to provide services on demand.

[0082] The description of the various methods of the present invention has been given for illustrative purposes, but it is not intended to be exhaustive or limited to the disclosed methods. Many modifications and variations will be obvious to those of ordinary skill in the art without departing from the scope and spirit of the described methods. The terms used herein were chosen to best explain the principles of the methods, the practical application, or the technological improvements found in the marketplace, or to enable other ordinary skilled artisans in the art to understand the methods disclosed herein.

Claims

1. A computer-implemented method, comprising: identifying machines involved in performing a manufacturing process at a manufacturing location; identifying a workflow sequence executed by the machines; receiving conditions associated with a remote operator using a virtual reality (VR) device to remotely control the machines to execute the executed workflow sequence at the manufacturing location; and for each VR device among the VR devices, determining a scope of a VR collaboration environment to be displayed, wherein the scope is determined based on the conditions, thereby reducing latency in the execution of the executed workflow sequence at the manufacturing location; and outputting the scope to the VR device.

2. The computer-implemented method according to claim 1, wherein, The conditions include the relative Internet bandwidth of the VR device.

3. The computer-implemented method according to claim 2, wherein, Determining the scope of the VR collaboration environment includes: determining a first machine among the machines remotely controlled by a first VR device having a relatively low Internet bandwidth among the VR devices, and determining a second machine among the machines remotely controlled by a second VR device having a relatively high Internet bandwidth among the VR devices.

4. The computer-implemented method according to claim 3, wherein, Determining the scope of the VR collaboration environment includes: within the scope of the VR collaboration environment for the first VR device having the relatively low Internet bandwidth, including the first machine and not including the second machine, and within the scope of the VR collaboration environment for the second VR device having the relatively high Internet bandwidth, including the first machine and the second machine.

5. The computer-implemented method according to claim 3, wherein, Determining the scope of the VR collaboration environment includes: within the scope of the VR collaboration environment for the first VR device, specifying that the first machine has a relatively higher clarity than the second machine, and within the scope of the VR collaboration environment for the second VR device, specifying that the first machine has the same clarity as the second machine.

6. The computer-implemented method according to claim 5, comprising: In response to determining that the first VR device obtains control of the second machine, increasing the relative clarity of the second machine in an updated version of the scope of the VR collaboration environment for the first VR device; and outputting the updated version to the first VR device.

7. The computer-implemented method according to claim 6, comprising: In response to determining that the first VR device obtains control of the second machine, reducing the relative clarity of the first machine in the updated version of the scope of the VR collaboration environment for the first VR device.

8. The computer-implemented method according to claim 2, wherein, Determining the scope of the VR collaboration environment includes: determining a first machine among the machines remotely controlled by a first VR device having a relatively low Internet bandwidth among the VR devices, and determining a second machine among the machines remotely controlled by a second VR device having a relatively high Internet bandwidth among the VR devices, and including: reassigning at least some work tasks currently assigned to a first remote operator using the first VR device among the remote operators to a second remote operator using the second VR device among the remote operators.

9. The computer-implemented method according to claim 1, wherein, A predetermined safety standard is applied to a first area of the manufacturing location, and the method includes: in response to determining that a first machine remotely controlled by a first VR device in the VR device has entered and / or is scheduled to enter the first area, increasing the relative clarity of the peripheral area of the first machine in an updated version of the scope of the VR collaboration environment for the first VR device; and outputting the updated version to the first VR device.

10. The computer-implemented method according to claim 1, comprising: Receiving a first command from a first VR device in the VR device to perform a first machine operation; Receiving a second command from a second VR device in the VR device to perform a second machine operation, wherein the first command is received before the second command; analyzing the timestamp information of the commands to determine whether the second command is output by the second VR device before the first command is output by the first VR device; and in response to determining based on the analysis that the second command is output by the second VR device before the first command is output by the first VR device, causing the second machine operation to be performed before the first machine operation.

11. A computer program product, the computer program product including a computer-readable storage medium having program instructions embodied thereon, the program instructions being readable and / or executable by a computer to cause the computer to: Identify, by the computer, machines involved in performing a manufacturing process at a manufacturing location; Identify, by the computer, the workflow sequence executed by the machines; Receive, by the computer, conditions associated with a remote operator using a virtual reality (VR) device to remotely control the machines to perform the executed workflow sequence at the manufacturing location; And Determine, by the computer, for each VR device in the VR device, the scope of the VR collaboration environment to be displayed, wherein the scope is determined based on the conditions, thereby reducing latency in the execution of the executed workflow sequence at the manufacturing location; And Output, by the computer, the scope to the VR device.

12. The computer program product according to claim 11, wherein, The conditions include the relative Internet bandwidth of the VR device.

13. The computer program product according to claim 12, wherein, Determining the scope of the VR collaboration environment includes: determining a first machine in the machines remotely controlled by a first VR device having a relatively low Internet bandwidth among the VR devices, and determining a second machine in the machines remotely controlled by a second VR device having a relatively high Internet bandwidth among the VR devices.

14. The computer program product according to claim 13, wherein, Determining the scope of the VR collaboration environment includes: including the first machine and not including the second machine within the scope of the VR collaboration environment for the first VR device having the relatively low Internet bandwidth, and including the first machine and the second machine within the scope of the VR collaboration environment for the second VR device having the relatively high Internet bandwidth.

15. The computer program product according to claim 13, wherein, Determining the scope of the VR collaboration environment includes: within the scope of the VR collaboration environment for the first VR device, specifying that the first machine has a relatively higher clarity than the second machine, and within the scope of the VR collaboration environment for the second VR device, specifying that the first machine has the same clarity as the second machine.

16. The computer program product according to claim 15, wherein the program instructions are readable and / or executable by the computer to cause the computer to: in response to determining that the first VR device has obtained control of the second machine, increase the relative clarity of the second machine in an updated version of the scope of the VR collaboration environment for the first VR device; and output the updated version to the first VR device.

17. The computer program product according to claim 12, wherein, Determining the scope of the VR collaboration environment includes: determining a first machine among the machines that is remotely controlled by a first VR device having a relatively low Internet bandwidth among the VR devices, and determining a second machine among the machines that is remotely controlled by a second VR device having a relatively high Internet bandwidth among the VR devices, and the program instructions are readable and / or executable by the computer to cause the computer to: reassign at least some of the work tasks currently assigned to a first remote operator using the first VR device among the remote operators to a second remote operator using the second VR device among the remote operators.

18. The computer program product according to claim 11, wherein, A predetermined safety standard is applied to a first area of the manufacturing location, and the program instructions are readable and / or executable by the computer to cause the computer to: in response to determining that a first machine remotely controlled by a first VR device among the VR devices has entered and / or been scheduled to enter the first area, increase the relative clarity of the peripheral area of the first machine in an updated version of the scope of the VR collaboration environment for the first VR device; and output the updated version to the first VR device by the computer.

19. The computer program product according to claim 11, wherein the program instructions are readable and / or executable by the computer to cause the computer to: receive, by the computer, a first command from a first VR device in the VR device to perform a first machine operation; receive, by the computer, a second command from a second VR device in the VR device to perform a second machine operation, where The first command was received before the second command; The computer analyzes the timestamp information of the commands to determine whether the second command was output by the second VR device before the first command was output by the first VR device; And in response to determining based on the analysis that the second command was output by the second VR device before the first command was output by the first VR device, the computer causes the operation of the second machine to be executed before the operation of the first machine.

20. A system, comprising: A processor; And Logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, the logic being configured to: Identify the machines involved in performing the manufacturing process at the manufacturing location; Identify the workflow sequence executed by the machines; Receive conditions associated with a remote operator using a virtual reality (VR) device to remotely control the machine to perform the workflow sequence being executed at the manufacturing location; And For each VR device among the VR devices, determine a scope of a VR collaboration environment to be displayed, wherein the scope is determined based on the conditions, so as to reduce latency in the execution of the workflow sequence being executed at the manufacturing location; And Output the scope to the VR device.

Images