Handsfree documentation of pharmaceutical operations using an extended reality device

Abstract

Documentation of pharmaceutical operations is typically performed as a combination of paper and electronic records. As a result of good manufacturing practice guidelines, operators constantly move between reading, executing, and documenting. The present disclosure describes a method of handsfree accessing and updating of data during execution of operations, using an extended reality (XR) device. The XR device enables verification of an operator's identity, and connection to a private cloud. A task is initiated by the operator or a third-party user, and the task identifier is sent to the cloud. The cloud retrieves and sends the associated data cluster. The display-only fields of the data cluster are displayed on the screen. The editable fields of the data cluster are presented in specific forms, and the operator uses voice commands to fill in each field. The software sends the updated data cluster to the cloud to replace the original data cluster.

Description

HANDSFREE DOCUMENTATION OF PHARMACEUTICAL OPERATIONS USING AN EXTENDED REALITY DEVICECROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 605,819 filed December 4, 2023, the disclosure of which is incorporated by reference herein in its entirety.FIELD

[0002] The present embodiments described here relate generally to documentation of pharmaceutical operations, and in particular to a method of handsfree accessing and updating of data during execution of operations.BACKGROUND

[0003] Documentation of pharmaceutical operations is required by CGMP to follow the ALCOA+ principle. During a batch manufacturing, operators read instructions from a computer screen or a paper, execute the instructions, and record data by typing on the computer or writing on the paper. Movement of pharmaceutical products (i.e., arriving, leaving and / or moving within a company) is currently tracked by using paper forms, followed by transferring data to Enterprise Resource Planning (ERP) systems. In case of critical processes two individuals are required for the execution of an operation (i.e., complying with the four-eyes principle). As a result, operators constantly move between reading, executing, and documenting, in a sequential way.SUMMARY

[0004] Described herein, is a method of handsfree access to a private cloud and updating of data, using an extended reality (XR) device. The XR device enables verification of an operator’ s identity, and connection to a private cloud. In some embodiments, a task is initiated, and theassociated task identifier is sent to the cloud. The cloud requests the software on the XR device to collect task related signals, and sends the associated data cluster. The display-only fields of the data cluster are displayed on the screen. The XR device presents editable fields of the data cluster in specific forms, and the operator uses voice commands, buttons on the device, or hand gestures to fill in each field. The software sends the updated data cluster to the cloud, which replaces the original data cluster with the updated data cluster.

[0005] In one aspect, the present disclosure is directed to a method of hands-free access to a cloud including: logging into software installed on an extended reality (XR) device, by an operator, enabling an authenticated e-signature; establishing a connection, by the XR device, to a private cloud; initiating a task (for example, by scanning a QR code linked to the task), by the operator and / or a third-party user; sending a task identifier associated with the initiated task, by the software on the XR device, to the cloud; sending at least one task related data cluster, by the cloud to the XR; displaying at least one display-only field of the task related data cluster, on the XR device screen; presenting at least one editable field of the task related data cluster, by the XR device, to the operator; updating the editable field of the task related data cluster, by the operator; sending the updated task related data cluster, by the software on XR device, to the cloud; and replacing the original task related data cluster with the updated task related data cluster, by the cloud. In some embodiments, the initiated task includes one or more work packages.

[0006] In some embodiments, logging into the software includes using voice commands.

[0007] In some embodiments, using voice commands includes detecting a wake word by the software on the XR device.

[0008] In some embodiments, the detected wake word prompts automatic speech recognition.

[0009] In some embodiments, the method includes: detecting, by the automatic speech recognition, one or more login credentials; and sending the one or more login credentials, by the software on the XR device, to the cloud.

[0010] In certain embodiments, logging into the software includes using a QR code generated by a computer and / or a mobile phone.

[0011] In some embodiments, the method includes scanning the QR code, by a camera installed on the XR device.

[0012] In some embodiments, the QR code is scanned by a reader integrated with the functionality in a camera.

[0013] In some embodiments, the QR code is scanned by an external application, installed on the XR device.

[0014] In some embodiments, the cloud includes a Wi-Fi network (for example, a Wi-Fi 6 network).

[0015] In some embodiments, connection to the cloud is established using a Wi-Fi 6 router.

[0016] In some embodiments, the cloud includes a private mobile network (for example, a private 5G network).

[0017] In some embodiments, connection to the cloud is established using a Radio Access Network (RAN).

[0018] In some embodiments, initiating a task includes scanning a QR code, by the operator.

[0019] In some embodiments, initiating a task includes detecting a text, by the operator.

[0020] In some embodiments, the method includes detecting the text by a camera installed on the XR device.

[0021] In some embodiments, initiating a task includes making one or more selection(s) on a remote device, by a third-party user.

[0022] In another aspect, the present disclosure is directed to a method of requesting data using a task identifier, including: sending a task identifier, by software on the XR device, to the cloud; retrieving data, by the cloud, using the received task identifier; and sending the retrieved data, by the cloud, to the XR device.

[0023] In some embodiments, the task identifier includes: information about data stored in the cloud associated with the task identifier; information about a data cluster within stored data; and a request to receive the data cluster.

[0024] In some embodiments, the task includes a training session, an inspection session, a work order, a piece of equipment, a product, and / or a quality control session.

[0025] In some embodiments, the method includes sending a request, by the cloud, to the software on the XR device, to collect task related signals.

[0026] In some embodiments, the method includes collecting and sending the requested task related signals, by the software on the XR device, to the cloud.

[0027] In some embodiments, the method includes displaying the editable fields of the data cluster as texts, pictures, audio and / or videos.

[0028] In some embodiments, the method includes presenting the editable field of the task related data cluster, by the XR device, to the operator, using one or more available data forms.

[0029] In some embodiments, the method includes transmitting data between the at XR device and the cloud via a bidirectional data flow, including transmitting data via a first data field using a permanent bidirectional data flow; and transmitting data via a second data field using an alternating bidirectional data flow.

[0030] In some embodiments, the first data field includes the task related data cluster, and the second data field includes editable data field.

[0031] In some embodiments, the data forms enable entry of at least one of a checkbox, numeric values, time values, a dropdown menu, a query, and / or picture / video / audio data.

[0032] In some embodiments, updating the editable data filed includes selecting a value for the data form, by the operator.

[0033] In some embodiments, selecting a value for the data form includes using voice commands, by the operator.

[0034] In some embodiments, selecting a value for the data form includes using hand gestures, by the operator.

[0035] In some embodiments, selecting a value for the data form includes using a navigation button, by the operator.

[0036] In some embodiments, selecting a value for the data form includes using a camera, by the operator, to detect a text (i.e., the camera recognizes the text for the operator).

[0037] In another aspect, the present disclosure is directed to a method of requesting data using a QR code, including: capturing the QR code, by a QR reader; deciphering the captured QR code into a text, by the QR reader; detecting a task identifier from the deciphered text, by the software on the XR device; sending the task identifier, by the software on the XR device, to the cloud; retrieving data, by the cloud, using the received task identifier; and sending the retrieved data, by the cloud, to the XR device.

[0038] In some embodiments, the QR reader is built in a camera.

[0039] In some embodiments, the QR reader is an external application, installed on the XR device.

[0040] In some embodiments, the task identifier includes: information about data stored in the cloud associated with the task identifier; information about a data cluster within stored data; and a request to receive the data cluster. In another aspect, the present disclosure is directed to an XR device, including: a camera; a microphone; a speaker; software installed on the XR device, the software being capable of collecting image (throughout the whole electromagnetic spectrum), audio and video signals; and a processor connecting the software to a private cloud; wherein the software is configured to send a task identifier to the cloud and display at least one task related data cluster, received from the cloud, on the device to be viewed and / or updated by the operator.

[0041] In some embodiments, the XR device further comprises a GPS chip.

[0042] In some embodiments, the XR device further comprises a navigation button.

[0043] In some embodiments, the XR device further comprises one or more IR LEDs.

[0044] In another aspect, the present embodiments are directed to a method of analyzing work pattern data including: compiling a database of work pattern data that includes video data, eye movement data, hand movement data, and / or arm movement data; compiling performance data corresponding with the work pattern data, the performance data including safety parameters, process efficiency data, and / or product performance data; and building a computer model to correlate the work pattern data with the performance data. The computer model may be trained such that any noteworthy trends in the work pattern data that are related to favorable and / or unfavorable performance data are identifiable by the computer model.

[0045] In some embodiments, the computer model employs artificial intelligence, machine learning, and / or deep learning.

[0046] In some embodiments, the method includes compiling at least one process parameter data point that is associated with the work pattern data, and integrating the process parameter data point with the work pattern data. Integrating the process parameter data point with the work pattern data may include integration via at least one common timestamp.

[0047] In another aspect, the present embodiments are directed to a method of identifying hazardous conditions within a workplace, the method including: obtaining videos and / or images of the workplace via an XR device; inputting the videos and / or images into a machine learning module, the machine learning module comprising a hybrid architecture comprising at least (1) a convolutional neural network (CNN) used for recognition of objects, characteristics, and / or other features in the videos and / or images, and (2) a recurrent neural network (RNN) used for sequential pattern recognition of the objects, characteristics, and / or other features in the videos and / or images; convoluting, by the CNN, the videos and / or images to identify the objects, characteristics, and / or other features therein; producing, by the CNN, a time-dependent feature map comprising the (1) objects, characteristics, and / or other features identified by the CNN, and (2) corresponding timestamps; inputting the time-dependent feature map into the RNN; processing the timedependent feature map, by the RNN, to establish weights correlating each of the objects, characteristics, and / or other features of the time-dependent feature map to a likelihood that hazardous conditions are present; and producing for each of the objects, characteristics, and / or other features of the time-dependent feature map, by the RNN, a prediction of a likelihood that hazardous conditions are present in the videos and / or images of the workplace.

[0048] In some embodiments, the method includes comparing each of the predictions produced by the RNN to a control specific to each feature used for training the RNN, thereby producing a weight derivative between each prediction and each control; and back propagating each of the weight derivatives to each corresponding weight, thereby adjusting each corresponding weight such that updated predictions may be produced by the RNN for each of the objects, characteristics, and / or other features of the time-dependent feature map.

[0049] In some embodiments, the CNN is trained via a feature recognition algorithm including at least one of an edge detection algorithm, a feature from accelerated segment test (FAST) algorithm, a local FAST algorithm, a canny edge detector, and a global FAST algorithm.

[0050] In some embodiments, the RNN is trained using videos and / or images that have been tagged as: (1) including or not including an accident; (2) are correlated or not correlated with hazardous conditions such as the occurrences of fires, exposure to hazardous chemicals and / or materials such as carbon monoxide, biological hazards, hazardous materials, the presence of combustible materials, and / or (3) identifying personnel not wearing proper personal protection equipment (PPE).

[0051] In some embodiments, the time-dependent feature map includes a serialized format including multiple linear output layers alternating with the corresponding timestamps, arranged in a single, serialized arrangement.

[0052] In some embodiments, the serialized arrangement includes one or more data files formatted in a JSON format.

[0053] In another aspect, the present embodiments are directed to a method of enhancing quality control of a workplace or optimizing productivity of the workplace including: obtaining videos and / or images of the workplace via an XR device; inputting the videos and / or images into the machine learning module of the method as described herein; and performing the convoluting, producing, inputting, processing, and producing steps of the machine learning module.

[0054] In another aspect, the present embodiments are directed to the machine learning module of the method as described herein.

[0055] In some embodiments, making at least one selection on a remote device includes: (1) selecting a binary input, by the operator, corresponding to a refrigeration status of a selected item; (2) recording a timestamp automatically, by the cloud, corresponding to the binary input; and (3) calculating a time out of refrigeration (TOR) parameter for the selected item.

[0056] In some embodiments, calculating a time out of refrigeration (TOR) parameter includes calculating a total accumulated time out of refrigeration, an uninterrupted time out of refrigeration, and / or a current time out of refrigeration.

[0057] In some embodiments, selecting the binary input includes changing the binary input, thereby reflecting a change in refrigeration status, and calculating a time out of refrigeration (TOR) parameter includes tabulating TOR based at least in part on a timestamp corresponding to the change in the binary input.

[0058] In some embodiments, the method includes transmitting a real-time video feed of from the camera to the cloud.BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Fig. 1 illustrates an example of an extended reality (XR) device, according to aspects of the present embodiments.

[0060] Fig. 2 illustrates an exemplary network connection in a pharmaceutical manufacturing facility featuring a Wi-Fi network (for example, a Wi-Fi 6 network), according to aspects of the present embodiments.

[0061] Fig. 3 illustrates an exemplary network connection in a pharmaceutical manufacturing facility featuring a private mobile network (for example a 5G mobile network), according to aspects of the present embodiments.

[0062] Fig. 4 is a flow chart diagram of a method of handsfree access to a network and updating of data by an operator while performing a task, according to aspects of the present embodiments.

[0063] Fig. 5A is a flow chart diagram of a method of verifying an operator’ s identification using voice recognition, according to aspects of the present embodiments.

[0064] Fig. 5B is a flow chart diagram of a method of verifying an operator’s identification using a QR code, according to aspects of the present embodiments.

[0065] Fig. 6A is a flow chart diagram of a method of establishing a connection to a private cloud using a Wi-Fi network, according to aspects of the present embodiments.

[0066] Fig. 6B is a flow chart diagram of a method of establishing a connection to a private cloud using a private mobile network, according to aspects of the present embodiments.

[0067] Fig. 7 illustrates an example of stored data on a cloud that is associated with a task identifier, an example of a data cluster that is sent to the XR device from the cloud, and data forms that may be used to present editable data fields to the operator by the XR device, according to aspects of the present embodiments.

[0068] Fig. 8A is a schematic of an example of conventional execution and documentation of a batch record.

[0069] Fig. 8B is a schematic of an example of execution and documentation of a batch record using the XR device, according to aspects of the present embodiments.

[0070] Fig. 9 is a flowchart diagram of an example of using the XR device for updating or capturing a batch record, according to aspects of the present embodiments.

[0071] Fig. 10A is a schematic of an example of a conventional on-the-job training session.

[0072] Fig. 10B is a schematic of an example of a training session using the XR device 10, according to aspects of the present embodiments.

[0073] Fig. 11 is a flowchart diagram of an example of using the XR device during a training session, according to aspects of the present embodiments.

[0074] Fig. 12A is a schematic of an example of a conventional inspection session.

[0075] Fig. 12B is a schematic of an example of an inspection session using the XR device, according to aspects of the present embodiments.

[0076] Fig. 13 is a flowchart diagram of an example of using the XR device during an inspection session, according to aspects of the present embodiments.

[0077] Fig. 14A is a schematic of an example of a conventional method of tracking movement of pharmaceutical products.

[0078] Fig. 14B is a schematic of an example of tracking movement of pharmaceutical products using the XR device, according to aspects of the present embodiments.

[0079] Fig. 15 is a flowchart diagram of an example of using the XR device for recording the movement of pharmaceutical products, according to aspects of the present embodiments.

[0080] Fig. 16A is a schematic of an example of conventional storage of eBPR.

[0081] Fig. 16B is a schematic of an example of optimizing work patterns using the XR device, according to aspects of the present embodiments.

[0082] Fig. 17 is a flowchart diagram of an example of using the XR device for optimizing work patterns, according to aspects of the present embodiments.

[0083] Fig. 18 is a schematic of a method of initiating a task by the XR device, according to aspects of the present embodiments.

[0084] Fig. 19 is a flow chart diagram of a method of identifying hazardous conditions within a workplace, according to aspects of the present embodiments.DEFINITIONS

[0085] ALCOA+ Principles: As used herein, the term “ALCOA+ Principles” defines best practice guidelines and methodologies for data management, for example, within the pharmaceutical and biotechnology industries. The acronym ‘ALCOA’ defines that data should be Attributable, Legible, Contemporaneous, Original, and Accurate. In addition, ‘ALCOA+’ guidance (or principles) recommends that data is also Complete, Consistent, Enduring, and Available.

[0086] Atmospheres Explosibles (ATEX): As used herein, the term “Atmospheres Explosibles (ATEX)”, a French term which translates to explosive atmospheres, refers to a set of European Union (EU) directives which covers equipment and protective systems intended for use in potentially explosive atmospheres. The directive defines the essential health and safety requirements and conformity assessment procedures, to be applied before products are placed on the EU market.

[0087] Augmented Reality (AR): As used herein, the term “augmented reality (AR)” refers to the integration of digital information with a user's environment in real time. AR delivers digital visual elements, sound, and / or other sensory information to the user through a device like asmartphone, glasses, and / or headset. AR users experience a real-world environment with generated perceptual information overlaid on top of it.

[0088] Batch Manufacturing: As used herein, the term “batch manufacturing” in pharma refers to a non-continuous production process to manufacture a specific quantity of drugs. As the materials go from step to step, the current batch must be finished before a subsequent batch can be processed.

[0089] Bidirectional Data Flow: As used herein, the term “bidirectional data flow” refers to a form of data exchange where data can be transmitted and received between two or more parties, allowing for a two-way flow of data. In some embodiments, two parties may transmit and received data simultaneously (i.e., in a permanent bidirectional data flow). In some embodiments, the direction of data may alternate between the parties, such that, at any given time, one party may only transmit / edit data and another party may only receive data (i.e., in an alternating bidirectional data flow).

[0090] Cloud, Private Cloud: As used herein, the term "cloud" refers to servers that are accessed over the internet, and the software and databases that run on those servers. Cloud servers are located in data centers all over the world, so that companies do not have to manage physical servers themselves or run software applications on their own machines. A private cloud, as used herein, is defined as computing services offered over a private internal network and only to select users. Private clouds deliver a higher level of security and privacy through both company firewalls and internal hosting to ensure operations and sensitive data are not accessible to third-party providers.

[0091] Current Good Manufacturing Practice (CGMP): As used herein, the term “current good manufacturing practice (CGMP)” refers to a system for ensuring that products are consistently produced and controlled according to quality standards. It is designed to minimize the risks involved in any pharmaceutical production that cannot be eliminated through testing the final product. CGMP covers all aspects of production, from the starting materials, premises, and equipment to the training and personal hygiene of staff. Detailed, written procedures are essential for each process that could affect the quality of the finished product. CGMP includes systems to provide documented proof that correct procedures are consistently followed at each step in the manufacturing process - every time a product is made.

[0092] Enterprise Resource Planning (ERP): As used herein, the term “enterprise resource planning (ERP)” refers to a type of software system that helps organizations automate and manage core business processes for optimal performance. ERP software coordinates the flow of data between a company’s financial, supply chain, operational, commerce, reporting, manufacturing, and human resources activities on one platform. Today, ERP systems are critical for managing thousands of businesses of all sizes and in all industries.

[0093] Extended Reality (XR): As used herein, the term “extended reality (XR)” refers to an umbrella term that encompasses any sort of technology that alters reality by adding digital elements to the physical or real-world environment. XR includes augmented reality (AR), mixed reality (MR), and virtual reality (VR).

[0094] Four-eyes Principle: As used herein, the term “four-eyes principle” requires a certain activity, e.g., a decision, transaction, etc., to be approved by at least two people. This controlling mechanism is used in critical manufacturing steps to facilitate delegation of authority and increase transparency.

[0095] Good automated manufacturing practice (GAMP): As used herein, the term “good automated manufacturing practice (GAMP)” refers to a set of guidelines for manufacturers and users of automated systems in the pharmaceutical industry. The ISPE's guide describes a set of principles and procedures that help ensure that pharmaceutical products have the required quality. GAMP covers all aspects of production, from the raw materials, facility and equipment to the training and hygiene of staff. Standard operating procedures (SOPs) are essential for processes that can affect the quality of the finished product.

[0096] Large Language Models (LLMs), LLM-Based Chatbot: as used herein, the term “large language models (LLMs)” refers to deep learning algorithms that can recognize, summarize, translate, predict, and generate content using very large datasets. LLMs, which are trained on internet-scale datasets (which in some cases include billions (for example, tens or hundreds of billions) of parameters), have unlocked an Al model’s ability to generate human-like content. A “LLMs-based chatbot” enables users to refine and steer a conversation towards a desired length, format, style, level of detail, and language. Successive prompts and replies, known as prompt engineering, may be considered by the LLM-based chatbot at each conversation stage as context for one or more subsequent replies.

[0097] Manufacturing Execution System (MES): as used herein, the term “manufacturing execution system (MES)” refers to a comprehensive, dynamic software system that monitors, tracks, documents, and controls the process of manufacturing goods from raw materials to finished products. Providing a functional layer between enterprise resource planning (ERP) and process control systems, an MES gives decision-makers the data they need to make the plant floor more efficient and optimize production. Regulated industries such as pharmaceuticals, food and beverage, medical devices, and biotechnology may particularly benefit from an MES, as regulated companies must adhere to strict regulations to ensure traceability compliance.

[0098] Mixed Reality (MR): As used herein, the term “mixed reality (MR)” refers to a technology that provides not only the superimposition of digital elements into the real-word environment but also their interactions. MR delivers digital visual elements, sound, and / or other sensory information to the user through a device like a smartphone, glasses, and / or headset. MR users visually interact with and manipulate both physical and digital elements using next generation sensing and imaging technologies. As a result, MR receives input from the environment and changes accordingly.

[0099] 5G Network, Private 5G Network: As used herein, the term “5G network” refers to the fifth generation of wireless technology. 5G technology has a theoretical peak speed of 20 Gbps, and offers lower latency, which can improve the performance of business applications as well as other digital experiences. 5G networks are virtualized and software-driven, and they exploit cloud technologies. A “private 5G network” uses the same underlying network solutions, including hardware and software, encoding schemes, and spectrum, but is dedicated to the use of a single enterprise or organization, especially where the single enterprise or organization includes critical infrastructures and / or applications. The enterprise is responsible for selecting which spectrum to use (licensed, unlicensed, or shared), installing network solutions (for example, Radio Access Network and Core), managing the users, and maintaining the network. The enterprise may enable an extra layer of data safety since all of the data stays within the private 5G network, and can decide whether to allow connection to a public network to allow external users to access the private network.

[0100] SAP: As used herein, the term “SAP” refers to the world's leading ERP software company, based in Walldorf, Germany.

[0101] Time out of Refrigeration (TOR): As used herein, the term “time out of refrigeration (TOR)” refers to the amount of time a drug product is outside of the storage temperature. The maximum TOR is the time the primary container can spend, outside of registered storage conditions without risk to patients and product shelf life.

[0102] United States Pharmacopeia and the National Formulary (USP—NF): As used herein, the term “United States pharmacopeia and the national formulary (USP-NF)” refers to a pharmacopoeia published by the United States Pharmacopoeial Convention. The USP- NF contains monographs and standards for medicines, finished dosage forms, active drug substances, excipients, biologies, compounded preparations, medical devices, dietary supplements, and many other therapeutic goods intended for use in healthcare.

[0103] Virtual Reality (VR). as used herein, the term “virtual reality (VR)” refers to a simulated 3D (i.e., three dimensional) environment that enables users to explore and interact with a virtual surrounding in a way that approximates reality, as it is perceived through the users' senses. VR immerses the users in the simulated environment through the use of interactive devices such as helmets, goggles, gloves and / or bodysuits.

[0104] IVi-Fi 6: As used herein, the term “Wi-Fi 6” refers to systems, equipment, and / or methodologies conforming to the wireless communications protocol (IEEE Standard 802.1 lax) introduced in 2019 and includes an operating frequency of 2.4 GHz or alternatively 5.0 GHz, and a maximum link rate in a range from 600 Mbits / s to 9608 Mbits / s. As used herein, the term “WiFi 6 network” refers to a communications network in which every component of the network is both capable of conforming to the IEEE Standard 802.1 lax and is also actively operating such that it is conforming to the IEEE Standard 802.1 lax. Wi-Fi 6 networks according to the present embodiments may therefore include routers (for example, Wi-Fi 6 routers) spatially arranged in the required frequency to enable link rates of at least 600 Mbits / s throughout the entire network. In addition, Wi-Fi 6 networks according to the present embodiments may include one or more routers and / or extenders being physically wired to other routers and / or extenders such that signals can be carried as needed through walls, floors, and other obstacles within the network.DETAILED DESCRIPTION OF CERTAIN EMBODIMENTS

[0105] Documentation of pharmaceutical operations, which may include production, analytical testing, logistics, etc., must follow the ALCOA+ principles. In certain instances, the European and US Pharmacopoeia (i.e., USP-NF) require documentation of critical processes to be executed by two individuals, i.e., to comply with the four-eyes principle. Currently the documentation of activities is performed as a combination of paper and electronic records. Described herein, is a method of handsfree access to a private cloud and updating of data, using an extended reality (XR) device. This may transfer documentation of pharmaceutical operations to a digital space to facilitate simultaneous documentation and execution of operations, as well as enabling people to work together remotely, in a CGMP compliant manner.

[0106] CGMP is a comprehensive manufacturing system that ensures product consistency and quality by addressing five key components:

[0107] (1) Products: constant testing, comparison, and quality assurance are crucial steps in the product lifecycle to ensure they meet the desired standards before reaching consumers. All primary materials, including raw products and other components, should have clearly defined specifications at every production phase. Standard methods for packing, testing, and allocating sample products should be adhered to, ensuring consistency and quality at all times.

[0108] (2) Processes: clear, consistent, and well -documented processes are essential to effective CGMP. These documented processes should be made accessible to all employees, and regular evaluations should be conducted to ensure compliance and alignment with the organization’s quality standards.

[0109] (3) Procedures: defined as a set of guidelines for performing a critical process or a part of a process, play a pivotal role in achieving consistent results. All employees must be familiar with these procedures and follow them diligently. Any deviation from standard procedures should be immediately reported, thoroughly investigated, and adequately addressed to prevent compromise on product quality.

[0110] (4) Premises: the condition and maintenance of premises directly influence product quality. Premises should be maintained clean at all times to prevent cross-contamination, accidents, or other undesirable outcomes. Equipment placement, proper storage, and regular calibration are crucial to ensure their optimal functionality and reliability, thereby producing consistent results and preventing equipment failure risks.

[0111] (5) People: a company’s personnel form the foundation of successful CGMP implementation. All employees are expected to strictly follow manufacturing processes and regulations. To ensure this, regular CGMP training are expected for all employees to enhance their understanding of their roles, responsibilities, and the importance of their contributions to product quality and safety. Regular performance assessments further aid in boosting productivity, efficiency, and competency.

[0112] An XR device may fulfil all CGMP and work safety regulatory requirements, such as GAMP, ATEX, FDA part 11 (electronic record and electronic signature), the four-eyes principle, audit trail, IT-security, and data storage.

[0113] Fig. 1 illustrates an example of an extended reality (XR) device 10, according to aspects of the present embodiments. In some embodiments, the device 10 may include a head mount 12. In some embodiments, the head mount 12 may be used to mount glass 26 on an operator’s head. In some embodiments, the device 10 may include one or more speakers 14, a GPS chip 28, a communications module 30, a processor 31, and / or one or more batteries 32, all of which may be disposed on or within the head mount 12. In some embodiments, the one or more speakers 14 may be used to play audio instructions for the operator. In some embodiments, the one or more speakers 14 may be used by the operator to interact with internal and / or external third- party users. In some embodiments, the device 10 may include one or more cameras 16, 20 and 22. For example, in some embodiments, the XR device 10 may include a first camera 16 located on a top rim 17 of the device 10, and a second camera 22 located on a bottom portion of the glass 26. In some embodiments, the one or more cameras 16 may be used in the IR range for measuring surface temperature, as further described in connection with Example 3 / Fig. 13. In some embodiments, the one or more cameras 16 may be used in the IR range for detecting the operator’s hands, as further described in connection with Fig. 7 and / or Example 5 / Fig. 17, as well as for room scanning for identification of equipment, and other use such as determining surface temperatures of equipment, materials, fluids, products, etc. In some embodiments, the one or more cameras 22 may be used in the IR range for eye-movement detection, as further described in connection with Example 5 / Fig. 17. In some embodiments, the device 10 may include one or more IR LEDs 18 located on glass 26. In some embodiments, the one or more IR LEDs 18 may be used for eye-movement detection, as further described in connection with Example 5 / Fig. 17.

[0114] Still referring to Fig. 1, in some embodiments, the XR device 10 may include a camera 20 located on the top rim 17 of the device 10. In some embodiments, the device 10 includes a wearable device 10. In some embodiments, the device 10 may include a headset 10 as shown in Fig. 1, thereby enabling handsfree operation. In some embodiments, the XR device 10 is capable of performing augmented reality (AR), mixed reality (MR), and / or virtual reality (VR) functionality. In some embodiments, the camera 20 may be used to capture picture and / or video by the operator. In some embodiments, the camera 20 may be used for capturing a live video feed for third-party user(s). In some embodiments, the camera 20 may be used to scan QR codes and / or detect texts. In some embodiments, the XR device 10 may include a microphone 24. In some embodiments, the microphone 24 may be used by the operator to interact with third-party users. In some embodiments, the microphone 24 may be used by the operator to record audio data. In some embodiments, the microphone 24 may be used by the operator to use voice commands. In some embodiments, the microphone 24 may be used by the operator to interact with an LLM- based chatbot (e.g., ChatGPT). In some embodiments, the XR device 10 may include a navigation button 33. In some embodiments, the navigation button 33 may be used to select the method of initiating a task. The operator, for example, may use the navigation button 33 to select text recognition for initiating a task. In some embodiments, the navigation button 33 may be used to update data. The operator, for example, may use the navigation button 33 to move between editable fields, select an option, save the input data, etc. In some embodiments, the GPS chip 28 may be used to detect the operator’s geographical position, as further described in connection with Example 3 / Fig. 13 and / or Example 5 / Fig. 17. In some embodiments, the GPS chip 28 may be used to detect a geographical position or location of a drug, as further described in connection with Example 4 / Fig. 15.

[0115] Referring still to Fig. 1, in some embodiments, the XR device 10 may be configured to display one or more data clusters 34, according to aspect of the present embodiments. Each data cluster 34 may include a number of parameter values including (1) editable data fields 38 shown within dashed boxes, and (2) display-only data fields 36 (not within dashed boxes). The display-only fields 36 may be used to display data on the XR device 10 for an operator to view. The editable data fields 38 may be used by the operator to send information to an external system (for example, a local server, a remote server, the cloud, etc.). The editable data field 38 may be presented to the operator in one or more data forms 40. The data forms 40, as displayed to theoperator, may include a dropdown menu, picture / video / audio, a numeric value, checkboxes, a time value, and / or a query. These are described in further detail in connection with Fig. 7.

[0116] Fig. 2 illustrates an exemplary network connection 42 in a pharmaceutical manufacturing facility featuring a Wi-Fi network, according to aspects of the present embodiments. In some embodiments, the network connection 42 may include the manufacturing facility 60, one or more external third-party users 64, and global internet 62. In some embodiments, the manufacturing facility 60 may include a Wi-Fi network 44, a distributed control system 52, local controllers 54, one or more operators 58, and one or more internal third-party users 56. In some embodiments, the Wi-Fi network 44 may include a private cloud 46, a Wi-Fi router 50, and a modem 48 (i.e., a Wi-Fi modem). In some embodiments, the private cloud 46 may perform computationally extensive tasks (for example, Al computing and machine learning algorithms) and store all manufacturing data. In some embodiments, the private cloud 46 may be accessed via the Wi-Fi router 50. In some embodiments, the Wi-Fi router 50 may connect the distributed control system 52, the one or more operators 58, and one or more internal third-party users 56 to the private cloud 46 and / or the modem 48. In some embodiments, the modem 48 may connect the Wi-Fi router 50 to global internet 62. In some embodiments, the one or more operators 58 may use an XR device 10, a mobile phone 57 and / or a sensor glove 59. In some embodiments, the one or more operators 58 may use the Wi-Fi router 50 to access the private cloud 46 and to connect to global internet 62 via the modem 48. In some embodiments, the one or more operators 58 may connect directly to the global internet 62. In some embodiments, the one or more internal third-party users 56 may include engineers, scientists, technicians, inspectors, examiners, regulatory authorities, compliance specialists, analysts, operators, and / or other parties. In some embodiments, the one or more internal third-party users 56 may use the Wi-Fi router 50 to access the private cloud 46 and to connect to the global internet 62 via the modem 48. In some embodiments, the one or more internal third-party users 56 may connect directly to the global internet 62. In some embodiments, the one or more external third-party users 64 may include trainers, and / or inspectors.

[0117] Fig. 3 illustrates an exemplary network connection 66 in a pharmaceutical manufacturing facility featuring a private mobile network, according to aspects of the present embodiments. In some embodiments, the manufacturing facility 60 may include a private mobilenetwork 68, a distributed control system 52, local controllers 54, one or more operators 58, and one or more internal third-party users 56. In some embodiments, the private mobile network 68 may include a private cloud 46, a radio access network (RAN) 72, and a packet core network 70. In some embodiments, the private cloud 46 may be accessed via the packet core network 70. In some embodiments, the RAN 72 may connect the distributed control system 52, the one or more operators 58, and one or more internal third-party users 56 to the packet core network 70. In some embodiments, the packet core network 70 may connect the RAN 72 to global internet 62. In some embodiments, the one or more operators 58 may use the RAN 72 to access the private cloud 46 and connect to the global internet 62 via packet core network 70. In some embodiments, the one or more internal third-party users 56 may use the RAN 72 to access the private cloud 46 and connect to global internet 62 via packet core network 70.

[0118] Fig. 4 is a flow chart diagram of a method 73 of handsfree access to a private cloud and updating of data by an operator while performing a task, according to aspects of the present embodiments. At step 74, the method 73 may include verifying the operator’s identity (for example, via a login by the operator, to the device). At step 76, the method 73 may include establishing a connection, by the XR device, to a private cloud. At step 78, the method 73 may include initiating a task, by the operator and / or an admin / supervisor / trainer / inspector. In some embodiments, the initiated task may include a training session 100, an inspection session 102, a work order 104, equipment 106, a product 108, and / or quality control session 110. In some embodiments, a bar code may be used instead of a QR code. In some embodiments, using a QR code 354 may be preferable to a bar code since a QR code reader is enabled to identify a QR code 354 at virtually any orientation (i.e., due to the square shape and other attributes of the QR code 354) whereas using a bar code scanner may require extra time since the bar code scanner needs to be aligned with the alignment of the bar code. At step 80, the method 73 may include sending a task identifier 356, by the software on the XR device 10 (for example, via the communications module 30), to the cloud. At step 82, the method 73 may include sending a request, by the cloud, to the software on the XR device 10, to collect task related signals 112. In some embodiments the task related signals 112 may be collected by a camera 16, 20, 22, a microphone 24, a GPS 28, and / or an endoscope 114. At step 84, the method 73 may include starting to collect and send requested signals 112, by the software on the XR device 10, to the cloud. In some embodiments,the cameras 16, 20, 22 may include one or more conventional cameras and / or one or more IR cameras.

[0119] Still referring to Fig. 4, at step 86, the method 73 may include sending an associated data cluster 34 (shown in Fig. 1), by the cloud, to the XR device. At step 88, the method 73 may include displaying of the data cluster 86 on the XR device screen. In some embodiments, the data cluster 34 may be displayed as texts, pictures, audio and / or videos. For example, the data cluster may be associated with, and specific to, the task that was initiated at step 78. In some embodiments, the data cluster 86 may be displayed on the XR device 10 such that it appears holographically, that is, as a 3D rendering superimposed over the actual (physical world background) the operating is seeing, such that the operator may visually interact with the data cluster, for example via hand gestures and / or via sensor gloves, as described herein. In some embodiments, the data cluster 34 may be associated with a station, module, or process step within the facility 60 where a particular pharmacological process or subprocess is being carried out. In some embodiments, the QR code 354 associated with the process or sub-process is located at or near (for example, within 20 feet) of the process or sub-process within the facility 60. In some embodiments, when a task is initiated, the associated data cluster 34 that is displayed includes data fields that are specific to the process or sub-process, and that need to be populated / filled in, updated, and / or edited. In some embodiments, the data cluster 34 and / or fields needing to be updated or edited include a picture or video of the specific process or subprocesses. In some embodiments, the data cluster 34 and / or fields needing to be updated or edited include other types of data, as described herein. At step 90, the method 73 may include presenting editable fields of the data cluster 34 in a specific order, by the XR device 10, to the operator. In some embodiments, editable fields 36 of the data cluster 34 may include one or more of data forms 170. In some embodiments, the data forms 170 may include checkboxes 178 (for example, selectable binary options), a numeric value 176, a time value 180, a dropdown menu 172, a query 182, and / or picture / video / audio upload fields 174. At step 92, the method 73 may include using voice commands, by the operator, to fill in each presented editable field 36. In some embodiments, the method 73 may include using hand gestures, by the operator, to fill in each presented editable field 36. In some embodiments, the method 73 may include using video analytics to identify the hand gestures (for example, in connection with the cameras 16, 20, 22 and / or IR LEDs 18) such that editable fields 36 may be edited or updated without the need for using a second glove 59. In someembodiments, the method 73 may include using the navigation button 33, by the operator, to fill in each presented editable field 36. In some embodiments, the method 73 may include using the camera 20, by the operator, to detect a text to fill in each presented editable field 36. In some embodiments, the method 73 may include manipulating a sensor glove 59 to fill in each presented editable field 36. In some embodiments, the method 73 may include using a combination of voice commands, hand gestures, using the navigation button 33, using the camera20 and / or sensor glove 59 manipulations to fill in each presented editable field 36. At step 94, the method 73 may include updating each editable field 36, by the software on the XR device 10, according to the data form 170. At step 96, the method 73 may include sending the updated data cluster, by the software on XR device 10, to the cloud. At step 98, the method 73 may include replacing the original data with the updated data cluster, on the cloud.

[0120] Fig. 5A is a flow chart diagram of a method 134 of verifying an operator’s identification using voice recognition, according to aspects of the present embodiments. At step 136, the method 134 may include saying a wake word, by the operator. At step 138, the method 134 may include detecting the wake word, by a voice recognition module on the XR device. The voice recognition module may require a wake word to transition from passive listening to active listening. In some embodiments, the voice recognition module may be continuously listening for an auditory signature associated with the wake word. In some embodiments, the auditory signature may look for a pattern of different audit frequencies or pitches repeated in a specific order or timeframe. For example, in some embodiments, the voice recognition module may categorize sounds according to frequency with low frequency sounds falling into a range of about 20 Hz to about 150 Hz, middle frequencies falling in range from about 150 Hz to about 350 Hz, and high frequencies falling with a range of about 350 Hz to about 1050 Hz. The voice recognition module may then be listening for a specific pattern of low, medium, and high frequency sounds within a specific time frame (for example, about 0.3 seconds to about 2.0 seconds, about 0.4 seconds to about 1.5 seconds, about 0.5 seconds to about 1.2 seconds, about 0.5 seconds to about 1.0 seconds, about 0.5 seconds to about 0.9 seconds, and / or about 0.6 seconds to about 0.8 seconds). In some embodiments, the pattern the voice recognition module is looking for may include sounds in 2 or more (for example, 2, 3, 4, 5, 6, more than 6) different sounds categories repeated within the time frame such as low-high-low, high-low-high, medium-high, low-high, low-medium-high, high- low-medium, low-high-low-high, medium-high-low-high, etc.).

[0121] Referring still to Fig, 5A, in some embodiments, the voice recognition module may use (for example, in addition to frequency / pitch) tone to characterize the sound pattern it is listening for (i.e., in the wake word). In some embodiments, tone may be described as the fullness or quality of a sound within a given pitch or frequency. For example, the voice recognition module may quantify how continuous or choppy a sound is while it remains within a given pitch / frequency as a measure of the sounds tone. The quantified tone can then also be used to assess whether or not a wake word has been used. At step 140, the method 134 may include beginning of the automatic speech recognition, by the XR device, once the wake word is detected 138. In some embodiments, automatic speech recognition may include beginning to record voice sounds once the wake word is detected, and sending them to the cloud for voice recognition. Because full voice / text recognition requires far greater computing power, the full / voice recognition module may “live” on the cloud, while the local voice recognition module can “live” on the device, and is only responsible for detecting the wake word and performing the subsequent voice / text recording, thereby allowing it to be much less computationally intensive. At step 142, the method 134 may include 1) detecting login credentials, provided by the operator using voice commands, and 2) sending the detected login credentials, by the XR device 10, to the cloud, for identity verification.

[0122] Fig. 5B is a flow chart diagram of a method 144 of verifying the operator’s identification using a QR code 354, according to aspects of the present embodiments. At step 146, the method 144 may include generating a QR code 354 for the operator. In some embodiment, the QR code 354 may be generated on a mobile phone. In some embodiment, the QR code 354 may be generated on a personal computer. At step 148, the method 144 may include 1) scanning the QR code 354, by the operator, using the camera 20 on the XR device 10, and 2) sending the task identifier 356, by the XR device 10, to the cloud, for identity verification.

[0123] Fig. 6A is a flow chart diagram of a method 150 of establishing a connection to a private cloud using a Wi-Fi 6 network 44, according to aspects of the present embodiments. At step 152, the method 150 may include permitting a processor 31 to connect to a Wi-Fi 6 network by the software on the XR device 10. At step 154, the method 150 may include connecting the XR device 10 to a Wi-Fi 6 router 50, by the processor 31. At step 156, the method 150 may include connecting the XR device 10 to the cloud, by the Wi-Fi 6 router 50.

[0124] Fig. 6B is a flow chart diagram of a method 158 of establishing a connection to a private cloud using a private mobile network, according to aspects of the present embodiments. At step 160, the method 158 may include permitting a processor 31 to connect to a private mobile network, by the software on XR device 10. At step 162, the method 158 may include connecting the XR device 10 to a radio access network (RAN), by the processor 31. At step 164, the method 158 may include connecting the XR device 10 to the cloud, by the RAN.

[0125] Fig. 18 is a schematic of a method 350 of initiating a task by the XR device 10, according to aspects of the present embodiments. The method 350 may include selecting a task 278, by the operator 272, from a task list 352. A QR code 354 may then be generated for the selected task 278. The method 350 may further include scanning the QR code 354, by the operator 272, using the camera 20. In some embodiments, the QR code 354 may be captured by a built-in QR reader. In some embodiments, a QR reader application may be used by the XR device to capture the QR code 354. In some embodiments, the stored data in the QR code 354 may be numbers and / or characters (i.e., text). The numbers and / or characters / text may include a task identifier 356 associated with the selected task 278. In some embodiments, a text 353 (numbers and / or characters) may be presented for the selected task 278. The method 350 may further include scanning the text 353, by the operator 272, using the camera 20. The method 350 may include detecting the text 353, by a text recognition module on the XR device. The detected text may include the task identifier 356 associated with the selected task 278. In some embodiments, the QR code 354 and the text 353 are generated when a task is selected. In some embodiments, the QR code 354 and the text 353 are printed and placed next to a station. In some embodiments, the task 272 may be selected by an admin / supervisor / trainer / inspector on a remote device to generate the task identifier 356. In some embodiments, the task identifier 356 may be configured to include three pieces of information: 1) data stored 166 in the cloud associated with the task identifier 356, 2) a unique data cluster 34 within the stored data 166, and 3) a request to receive the unique data cluster 34. In some embodiments, data cluster 34 may be already created with predetermined fields including a selection of one or more editable fields 36, one or more display-only fields 36, and the field types 170 (dropdown menu 172, numerical 176, checkbox 178 (binary), etc.), as further described in connection with Fig. 7. The method 350 may include sending the task identifier 356, by the software on XR device 10, to the cloud. The receipt of the task identifier 356, may prompt the cloud to retrieve the requested data cluster 34, and send it to the XR device 10.

[0126] Fig. 7 illustrates an example of stored data 166 in a cloud that is associated with a task identifier 356, an example of a data cluster 34 that is sent or transmitted (for example, as illustrated with arrow 167 in Fig. 7) to the XR device 10 from the cloud, and the data forms 170 that may be used to present (via data transmission 169 to the XR device 10) the editable data fields 38 to the operator by the XR device 10, according to aspects of the present embodiments. In some embodiments, the task identifier 356 may be determined by a QR code 354 linked to a training session 100, an inspection session 102, a work order 104, a piece of equipment 106, a product 108, and / or identity verification 110 as further described in connection with Fig. 4. In some embodiments, one or more data clusters 34 may be sent, by the cloud, to the XR device 10. Each data cluster 34 may include a number of parameter values including (1) editable data fields 38 shown within dashed boxes, and (2) display-only data fields 36 (not within dashed boxes). The display-only fields 36 may be used to display data on the XR device 10 for the operator to view. The software running on the XR device 10 may be configured to display data clusters 34 of various shapes and comprising various types of data. In some embodiments, the data cluster 34 may be displayed as texts, pictures, audio and / or videos. The software on XR device 10 may be able to display the data cluster 34 in a way that distinguishes the editable data fields 38 from the display- only data fields 36 for the operator (for example, by highlighting them in different colors and / or by greying out the display-only data fields 36).

[0127] Still referring to Fig. 7, the editable data fields 38 may be used by the operator to send information to an external system (for example, a local server, a remote server, the cloud, etc.). The editable data fields 38 may be presented to the operator in one or more data forms 170. The data forms 170, as displayed to the operator, may include a dropdown menu 172, picture / video / audio upload fields 174, a numeric value 176, checkboxes 178, a time value 180, and / or a query 182. In some embodiments, one data form 170 may be displayed and wait for the operator to update. In some embodiments, the operator may use voice commands to update the displayed data from 170. The software on XR device 10 may be configured to include predetermined voice commands for the operator to navigate through editable data fields 38. The predetermined voice commands may, for example, include “next field”, “previous filed”, “update field”, “save update”, “close cluster”, etc. The software on XR device 10 may display the predetermined voice commands next to the editable data fields 38 to the operator.

[0128] In some embodiments, the editable data fields 38 may establish a bidirectional data flow between an operator and an external system (for example, a local server, a remote server, the cloud, etc.). In some embodiments, at any given time, the editable data fields 38 may only be edited by either the operator or the cloud in an alternating bidirectional data flow configuration, i.e., while the operator updates an editable data field 38, the cloud cannot update the same editable data field 38. For example, in an alternating bidirectional data flow configuration, when the editable data field is sent from the cloud to the operator (via the XR device 10) editing is locked such that the cloud cannot edit the editable data field 38 and only the operator can update the editable data field 38. Once editing by the operator is complete, in some embodiments, the editable data field 38 is sent back to the cloud at which point the editable data field 38 can only be edited by the cloud, and not by the operator. In some embodiments, the direction of data flow may be supervised by the operator, an algorithm on the external system, or a person operating the external system. In some embodiments, the editable data fields 38 may be edited by the operator and the cloud simultaneously, while the timestamp of each edit is recorded. The recorded timestamps, for example, may allow tracking and / or synching of data. Timestamps corresponding to different editable data fields 38 may be used to identify and / or verify data discrepancies. For example, synching timestamp of a data entry with timestamp of an accompanying video may be used to correct discrepancies in character recognition. In some embodiments, one or more editable data fields 38, at any given time, may only be edited by either the operator or the cloud, while one or more editable data fields 38 may be edited by the operator and cloud simultaneously. In some embodiments, one or more data clusters may be transmitted between the XR device 10 and the cloud (i.e., in both directions) in a permanent bidirectional data flow configuration in which the same data field(s) (i.e., the one or more data clusters) may be simultaneously transmitted from the XR device 10 to the cloud and from the cloud to the XR device 10.

[0129] Referring still to Fig. 7, in some embodiments, the operator may use hand gestures to update the displayed data form 170. In some embodiment, the one or more IR LEDs 18 may be used to illuminate the operator’s immediate surrounding, while the camera 16 may be used in the IR range to capture the IR light interrupted by the operator’s hands. The changes in the IR light intensity received by the camera 16 may be used to detect a spatial position and / or the motion of the hand. The collected data may be processed in the cloud using one or more algorithms (for example, complex algorithms, involving machine learning and artificial intelligence). In someembodiments, camera 20 may be used to for depth-sensing and creating a 3D representation of the hand. The collected data may be processed in the cloud using computer vision techniques to capture and analyze hand movements. The software on XR device 10 may be configured to include predetermined hand gestures for the operator to navigate through editable data fields 38. The predetermined hand gestures may, for example, include “swipe right” for next field, “swipe left” for previous filed, “swipe up” for update field, “swipe down” for save update, “pinch” to close cluster, etc. In some embodiments, the same gestures (for example, sipe right, swipe left, swipe up, swipe down, pinch) and / or other gestures may be used in various combinations to perform these functions (i.e., next field, previous field, update field, save update, and / or close cluster). The software on XR device 10 may display the predetermined hand gestures next to the editable data fields 38 to the operator. In some embodiments, the operator may use the navigation button 33 to update the displayed data from 170. The operator, for example, may press the navigation button 33 once to navigate between available options (e.g., “next field”, “previous filed”, “update field”, “save update”, “close cluster”, etc.) and twice to select an option. Pressing the button twice to select an option may include pressing the button twice within a pre-determined period of time (for example, about 0.2 seconds, about 0.3 seconds, about 0.4 seconds, about 0.5 seconds, about 0.6 seconds, about 0.7 seconds, about 0.8 seconds, and / or other suitable periods of time such that pressing the button twice to indicate the selection of an option can be distinguished from successive pressings of the button a single time to scroll between available options. Stated otherwise, in some embodiments, pressing the button once to scroll between available options (i.e., rather than to select an option) may include successive pressings of the button at time intervals greater than the pre-determined period of time or threshold.

[0130] Still referring to Fig. 7, in some embodiments, the operator may also use the navigation button 33 to select a value for a dropdown menu 172, checkboxes 178 and / or a query 182. In some embodiments, the operator may use the camera 20 to detect a text. In some embodiments, the operator may use the camera 20 to capture a picture and / or a video. In some embodiments, the operator may use the microphone 24 to record an audio report and / or surrounding sounds. In some embodiments, after a displayed data form 170 is updated by the operator, the editable field 38 may be updated on data cluster 34, by the software on XR device 10. The software on XR device 10 may wait for the operator to indicate the completion of data updates. The operator, for example, may use a predetermined voice command (“close cluster”,“update complete” etc.) to end the data update. In some embodiments, once the operator indicates the completion of data updates, the software on XR device 10 may send the updated data cluster 34 back to the cloud. The cloud may replace the original data cluster 34 with the updated data cluster 34 within the stored data 166.Example 1

[0131] During conventional batch manufacturing, operators read instructions from a computer screen or a paper, execute the instructions, and record data by typing it into a data field on the computer or writing on the paper. As a result, operators constantly move between reading, executing, and documenting, in a sequential way. When required by the European and US Pharmacopoeia, a second operator executes and / or observes the execution of critical processes to comply with the four-eyes principle.

[0132] In contrast to conventional processes, using the XR device 10 and associated methodologies of the present embodiments, the operator follows the instructions on display which may be presented as text, audio, pictures, and / or videos. The operator completes the batch record using voice commands, during and / or after the execution. The system transforms the information into a written document for batch release. Furthermore, when required, multiple operators are able to work simultaneously, while having access to the same information, which enables complying with the four-eyes principle.

[0133] Fig. 8A is a schematic of an example of conventional execution and documentation of a batch record. In some embodiments, the first step 270 may include reading instructions from a computer screen 274, by the operator 272. In some embodiments, the next step 276 may include executing the instructions 278, by the operator 272. In some embodiments, the last step 280 may include recording data on the computer 274, by the operator 272.

[0134] Fig. 8B is a schematic of an example of execution and documentation of a batch record using the XR device, according to aspects of the present embodiments. In some embodiments, the operator 272 may follow the instructions on display 282 and complete the batch record using voice commands, during and / or after the execution 278, as described herein.

[0135] Fig. 9 is a flowchart diagram of an example (or method) 184 of using the XR device10 for updating or capturing a batch record, according to aspects of the present embodiments. At step 186, the method 184 may include logging in to the XR device 10, by the operator, to establish a connection to a cloud. In some embodiments, step 186 may enable an authenticated e-signature. At step 188, the method 184 may include initiating a workorder 104, by the operator. At step 190, the method 184 may include sending a request to the software on the XR device 10, by the cloud, to collect signals from the microphone 24 and the camera 20. At step 192, the method 184 may include sending the instructions, by the cloud, to the software on the XR device 10. At step 194, the method 184 may include logging in to the cloud, by a third-party user, using a link. In some embodiments, step 194 may include enabling authenticated e-signature and gaining remote access, by the third-party, to the software on the XR device 10. The third-party, for example, may be an additional operator wearing a XR device 10 as described herein. In some embodiments, the remote access to the XR device 10 may allow the third-party user to see everything the operator is seeing, on a screen (a computer, a tablet, a mobile phone, etc.). For example, the data cluster 34 displayed on the holographic glass 26 may be superimposed on a live video feed captured by camera 20. At step 196, the method 184 may include observing, by the third-party, while the operator executes a task and fills in the batch record. In some embodiments, the third-party may confirm and / or help as the operator updates the data cluster 34, by talking to the operator through the speakers 14. The operator may save the updates once the accuracy is confirmed by the third-party. In some embodiments, step 196 complies with the four-eyes principle. At step 198, the method 184 may include sending the batch record to the cloud for a batch release. At step 199, the method 184 may include ensuring data integrity in real-time by a logfile.Example 2

[0136] On the job training, according to current standards, is typically performed in three steps: 1) A trainer demonstrates a task to the operator. The operator is expected to train / familiarize her / himself by reading and understanding Standard Operation Procedures (SOPs) specific to the task prior to training. 2) The operator performs the task while the trainer provides feedback. 3) The operator performs the task, observed by the trainer. Upon correct completion of the taskwithout any feedback from the trainer, the operator is certified to perform the activity on her / his own.

[0137] In contrast, using the XR device 10 and associated methodologies of the present embodiments, on the job training may be performed as follows: 1) the operator trains her / himself by extended reality content of the trainer performing the task, using the XR device 10. Instructions in the video may be standardized and reviewed by one or more instructors and quality assurance (QA) specialist, which may be independent from the personal experience of the operator and eliminate training content variability as a source of error. 2) The operator performs the task while the trainer / instructor / QA specialist observes the execution of the task remotely and provides audio feedback. 3) The operator performs the task while the trainer / instructor / QA specialist observes. This process allows remote training, i.e., while the operator is located at a production site or laboratory, the trainer can conduct the training session from anywhere.

[0138] Fig. 10A is a schematic of an example of a conventional training session. In some embodiments, the first step 284 may include demonstrating the task 278, by the trainer / instructor 286, to the operator 272, by the trainer 286. In some embodiments, the next step 288 may include performing the task 278, by the operator 272 and providing feedback by the trainer 286. In some embodiments, the third step 290 may include performing the task 278, by the operator 272, and observing the performance, by the trainer 286.

[0139] Fig. 10B, in contrast, is a schematic of an example of a training session using the XR device 10, according to aspects of the present embodiments. In some embodiments, the first step 292 may include extended reality presentations of the task execution 278. In some embodiments, the next step 294 may include performing the task 278, by the operator 272 and observing remotely and / or providing audio feedback, by the trainer 286. In some embodiments, the third step 296 may include performing the task 278, by the operator 272 while the trainer 286 observes.

[0140] Fig. 11 is a flowchart diagram of an example (or method) 200 of using the XR device 10 during a training session, according to aspects of the present embodiments. At step 202, the method 200 may include logging in to the XR device 10, by an operator, to establish a connection to a cloud. In some embodiments, step 202 may enable an authenticated e-signature. At step 204, the method 200 may include initiating a training session 100, by the operator. At step206, the method 200 may include sending the training content, by the cloud, to the software on the XR device 10. At step 208, the method 200 may include extended reality training, by the operator, and executing a task. At step 210, the method 200 may include logging in to the cloud, by a trainer, using a link. In some embodiments, step 210 may include enabling authenticated e-signature and gaining remote access, by the trainer, to the software on XR device 10. In some embodiments, the remote access to the XR device 10 may allow the trainer to see everything the operator is seeing, on a screen (a computer, a tablet, a mobile phone, etc.). For example, the data cluster 34 displayed on the holographic glass 26 may be superimposed on a live video feed captured by camera 20. At step 212, the method 200 may include sending a request to the software on the XR device 10, by the cloud, to collect signals from the microphone 24 and the camera 20. At step 214, the method 200 may include observing remotely, by the trainer, while the operator executes the task and providing feedback, by the trainer. In some embodiment, the feedback may be displayed as text and / or audio. At step 216, the method 200 may include observing remotely, by the trainer, while the operator executes the task. At step 217, the method 200 may revert to step 214 if the operator fails to execute the task correctly. At step 218, the method 200 may include storing the training session in the cloud.Example 3

[0141] Pharmaceutical manufacturing sites are regularly inspected by authorities to ensure excellence in product quality. Inspections are currently performed in person where a team of inspectors visits a production site and inspects Current Good Manufacturing Practice (CGMP) relevant topics. An inspection may include a walkdown (or walk-though) of the site followed by an inspection of documents. When processes are located in multiple buildings and / or sites, additional travelling may be required, reducing the efficacy of the inspection. Onboarding of new CGMP-relevant equipment requires a similar process. According to predefined requirements, the purchasing CGMP manufacturer travels to the vendor and performs factory acceptance tests (FATs) as part of the qualification. Depending on the locations of pharmaceutical sites and inspectors and / or vendors, onsite inspections may potentially be associated with high travel costs. In addition, there is a need to periodically recertify sites to verify CGMP compliance, thereby necessitating additional travel.

[0142] In contrast, using the XR device 10 and associated methodologies of the present embodiments enables inspections and / or FATs to be more efficient by virtually bringing the inspector to different locations in real-time. The capability to guide inspectors through sites and / or equipment makes traveling obsolete. The inspectors may prepare a list of points that need to be fulfilled by the standards and / or the factory (e.g., electricity plans, documentation on welding, etc.). The FAT documents may be filled out in an ALCOA+ compliant manner while the operator is completing the list of points that need to be fulfilled. The documentation and the equipment can be viewed remotely. The XR device 10 may also be capable of connecting additional cameras (e.g., endoscopes) to inspect hard-to-reach areas of equipment. In some embodiments, the software may be capable of implementing multiple video sources, for example, camera, endoscopes, etc.

[0143] Fig. 12A is a schematic of an example of a conventional inspection session. In some embodiments, step 298 may include creating written records and / or inspecting a process and / or equipment 302, by inspectors 300. In some embodiments, step 304 may include creating written records and / or inspecting associated documentation 306, by the inspectors 300 and / or the inspected company. In some embodiments, step 308 may include creating written records and / or inspecting the next process and / or piece of equipment 310, by the inspectors 300. In some embodiments, step 312 may include creating written records and / or inspecting associated documentation 314, by the inspectors 300.

[0144] Fig. 12B, in contrast, is a schematic of an example of an inspection session using the XR device 10, according to aspects of the present embodiments. In some embodiments, the inspection 316 may include inspecting processes and / or equipment 302, 310 and associated documentation 306, 314, by the operator 272, and viewing remotely, by the inspectors 300.

[0145] Fig. 13 is a flowchart diagram of an example (or method) 220 of using the XR device during an inspection session, according to aspects of the present embodiments. At step 222, the method 220 may include logging in to the XR device 10, by an operator, to establish a connection to a cloud. In some embodiments, step 222 may enable an authenticated e-signature. At step 224, the method 220 may include initiating an inspection session 102, by the operator. In some embodiments, initiating an inspection session 102 may include scanning a QR code, scanning a bar code, detecting one or more text strings, making a selection by an admin (or power userand / or by an operator), and / or other initiating a task via one or more other identifiers. At step 226, the method 220 may include sending a request to the software on the XR device 10, by the cloud, to collect signals from the microphone 24, the camera 20, the camera 16, the GPS chip 28, and an endoscope 114. At step 228, the method 220 may include logging in to the cloud, by one or more inspectors, using a link. In some embodiments, step 228 may include enabling authenticated e- signature and gaining remote access, by the inspectors, to the software on XR device 10. In some embodiments, the remote access to the XR device 10 may allow the inspectors to see everything the operator is seeing, on a device (a computer, a tablet, a mobile phone, etc.). For example, the data cluster 34 displayed on the holographic glass 26 may be superimposed on a live video feed captured by camera 20. At step 230, the method 220 may include walking to different locations of production site, by the operator, and performing actions that are requested by the inspectors. In some embodiments, the camera 20 may be used to aid the visual aspects of the inspection session. The operator, for example, may direct the camera 20 toward a piece of equipment, an ongoing process, another operator performing a task, a document, etc. In some embodiments, an endoscope 114 may be used to aid the visual inspection of hard-to-reach places. The operator, for example, may direct the endoscope 114 to the sides and / or behind the piece of equipment. In some embodiments, the camera 16 may be used in the IR range to measure surface temperature. The operator, for example, may measure the temperature of a sample, a piece of equipment, the surrounding surfaces of a process, etc. In some embodiments, the GPS chip 28 may be used to detect the operator’s geographical position. At step 232, the method 220 may include storing a record of the inspection session in the cloud.Example 4

[0146] Pharmaceutical products that arrive and / or leave CGMP companies or move within a company need to be tracked. It must be ensured that, for example, temperature sensitive materials are refrigerated, and when out of refrigerator, stay within defined time periods. In general, the traceability of all pharmaceutical products must be confirmed. Movement of products is currently tracked by using paper forms, followed by transferring data to Enterprise Resource Planning (ERP) systems (e.g., SAP). This enables real time tracking of products, necessary by ALCOA+ requirements, which is critical for batch release. Time out of Refrigeration (TOR)values, for example, need to be tracked when handling frozen products. TOR is the time between two locations where / when the product is not refrigerated. Intermediate TOR tabulations are routinely performed whenever the product is in transit (e.g., freezer to refrigerated trucks) such that a predicted final TOR calculation may be provided (for example, in the event products need to be rerouted if the predicted final TOR exceeds one or more threshold critical to ensuring product viability, etc.).

[0147] In contrast, using the XR device 10 and associated methodologies of the present embodiments, products may be scanned when picked up by the operator. The time of scan and the operator’s location may be recorded simultaneously (e.g., using GPS or the data of the picked product). This enables automated storage of the information associated with the product when leaving the temperature-controlled area.

[0148] Fig. 14A is a schematic of an example of a conventional method of tracking movement of pharmaceutical products. In some embodiments, tracking movement of products 318 may include picking up product 320 and using paper forms 322, by the operator 272, followed by transferring information to SAP 324 on a computer, by the operator 272.

[0149] Fig. 14B is a schematic of an example of tracking movement of pharmaceutical products using the XR device 10, according to aspects of the present embodiments. In some embodiments, tracking the movement of products 326 may include scanning and picking up the product 320, by the operator 272, while information 328 associated with the product is stored automatically by software on the XR device 10.

[0150] Fig. 15 is a flowchart diagram of an example (or method) 234 of using the XR device 10 for recording the movement of pharmaceutical raw materials / materials / probes / products, according to aspects of the present embodiments. At step 236, the method 234 may include logging in to the XR device 10, by an operator, to establish a connection to a cloud. In some embodiments, step 236 may enable an authenticated e-signature. At step 238, the method 234 may include initiating a workorder 104, by the operator. At step 240, the method 234 may include sending a request to the software on the XR device 10, by the cloud, to collect signals from the camera and the GPS chip 28. In some embodiments, data may be received by the XR device 10 via an enterprise resource planning (ERP) system (for example, a cloud-based ERP system). At step 242, the method 234 may include scanning the QR code (or in some embodiments a barcode or text identifier) of a pharmaceutical product 320 that is picked up by the operator. In some embodiments, the location of the operator may be detected, by the GPS chip 28, when the pharmaceutical product is scanned. In some embodiments the detected location and time of scanning the pharmaceutical product may be sent, by the XR device 10, to the cloud. At step 244, the method 234 may include calculating critical parameters, by the cloud. In some embodiments, time out of refrigeration (TOR) or other process related information may be calculated to confirm the next destination of the pharmaceutical product. For example, if the calculated TOR is within an acceptable range, the pharmaceutical product may be sent to the next destination according to schedule. However, if the calculated TOR is out of acceptable range, the pharmaceutical product may be discarded and / or rerouted to a temporary refrigeration area to prevent TOR exceeding a critical threshold. At step 246, the method 234 may include storing a record of movements of pharmaceutical products in the cloud.

[0151] Referring still to FIG. 15, in some embodiments, calculating TOR (for example, at step 244) may include calculating a total accumulated time out of refrigeration (TOR), an uninterrupted time out of refrigeration, and / or a current time out of refrigeration (TOR). In each case, the TOR calculation may include using a binary indicator (i.e., a 0 or 1) to indicate whether the item in question is in refrigeration or out of refrigeration, in connection with the corresponding timestamps. In some embodiments, a “0” signifies out of refrigeration and a “1” corresponds to in refrigeration. For example, in some embodiments, calculating a total accumulated time out of refrigeration (TOR) may include tabulating a time duration of each period when the item is out of refrigeration (i.e., by calculating the difference between the timestamp when the “1” becomes a “0” and the subsequent timestamp when the “0” becomes a “1” again) and adding these periods together to arrive at a total accumulated time out of refrigeration (TOR). In some embodiments, calculating an uninterrupted time out of refrigeration may include tabulating a time duration of each period when the item is out of refrigeration, and taking a maximum value of the tabulated time durations. The uninterrupted time out of refrigeration may be proportional to a maximum temperature reached by the item in question, and therefore may be indicative of whether or not the item in question has reached a spoliation threshold (and therefore does or does not need to be thrown out). In some embodiments, calculating a current time out of refrigeration (TOR) may include tabulating a difference between the present timestamp and the timestamp corresponding to the most recent instance in which a “1” became a “0.” The current time out of refrigeration(TOR) may be useful in understanding how long an item has before it must be returned to refrigeration (i.e., in order to avoid spoliation). At each point where an operator is selecting an item, the operator may have the opportunity to toggle between “in refrigeration” and “out of refrigeration” using the XR headset 10, as described herein (i.e., the operator may select the appropriate binary corresponding to the appropriate refrigeration status for the item in question). The system may automatically capture a timestamp corresponding to each time a binary is entered, whether or not the binary changes. According to aspects of the present disclosure, each of the various calculations of TOR disclosed herein may include tabulating TOR based at least in part on a timestamp corresponding to a binary change reflecting a change in refrigeration status.Example 5

[0152] Data associated with the processes in the pharmaceutical industry are typically collected in paper-based and / or electronical systems. If deviations from the predefined conditions occur, they are tracked, and improvements are implemented (Corrective and Preventive Action (CAPA)). This process triggers the implementation of improvements by failure.

[0153] In contrast, using the XR device 10 and associated methodologies of the present embodiments, operator’s work pattern data may be collected during the process. Work pattern data may include hand movement, eye movement, location, etc. The collected data may be transmitted and analyzed in real time using machine learning algorithms. Recommendations to ensure efficient, safe, and compliant execution of tasks within the validated parameters or alarms may be sent back to the operator. Alignment of process trends with quality control data enables the system to predict and prevent critical deviations and maximize the product quality in real-time. The information that is gained may be used to optimize processes before product quality is impacted.

[0154] Example 5 may be further applied to improve execution and documentation of a batch record, as discussed in Example 1. In some embodiments, processing may be improved by proposing next batch steps. In some embodiments, resources may be improved by prediction of raw material input. In some embodiments, product quality may be improved by calculating a machine setting for next steps (e.g., pumping speed, temperature, stirring, cycle time) according to collected data.

[0155] Example 5 may be further applied to improve a training session, as discussed in Example 2. In some embodiments, repetitive errors made by the operator may be tracked to optimize a training session. In some embodiments, the training material may be adapted according to the operator’s learning type.

[0156] Example 5 may be further applied to improve tracking of movement of pharmaceutical products, as discussed in Example 4. In some embodiments, walking paths may be tracked to optimize the paths and / or layout of a warehouse. In some embodiments, picking of products with the shortest shelf life may be prioritized.

[0157] Fig. 16A is a schematic of an example of conventional optimization of work patterns. In some embodiments, step 330 may include executing process 278, by the operator 272. In some embodiments, step 332 may include collecting and recording associated data 334, by the operator 272. In some embodiments, step 336 may include tracking of potential deviations from the predefined conditions, and implementing suggested improvements.

[0158] Fig. 16B, in contrast, is a schematic of an example of optimizing work patterns using the XR device 10, according to aspects of the present embodiments. In some embodiments, step 338 may include executing process 278, by the operator 272. In some embodiments, step 340 may include collecting and transmitting work pattern data 342 (for example hand movement, eye movement, location, etc.), by the XR device 10. In some embodiments, step 344 may include analyzing the data 342 in real time, by Al algorithms 345 running in the cloud. In some embodiments, step 346 may include (1) sending recommendations to optimize the process 278 and / or (2) alarms being sent to the operator 272, by the cloud. In some embodiments, step 346 may revert to step 338, which may include adjusting work the pattern, by the operator 272. In some embodiments, step 338 may include opposing the recommended action, by the operator 272.

[0159] Fig. 17 is a flowchart diagram of a method 248 of using the XR device 10 for optimizing work patterns, according to aspects of the present embodiments. At step 250, the method 248 may include logging in to the XR device 10, by the operator, to establish a connection to the cloud. In some embodiments, step 250 may enable an authenticated e-signature. At step 252, the method 248 may include initiating a task by the operator. In some embodiments, the task may include a quality control or business process reengineering (BPR or eBPR in this case) session (for example, to understand if improvements can be made to operator workflows from a safetyand / or optimization standpoint). At step 254, the method 248 may include sending a request to the software on the XR device 10, by the cloud, to collect signals from the microphone 24, the camera 20, one or more cameras 22, one or more IR LEDs 18, and the GPS chip 28. At step 256, the method 248 may include analyzing the operator’s work pattern data while performing a task. In some embodiments, analyzing the operator’s work pattern data may include: (1) compiling a database of work pattern data (for example, video data showing worker location within a work station or relative to a process or equipment, eye movement data, hand and arm movement data, hand gesture data, and other types of data); (2) compiling any corresponding process parameters associated with the work pattern data (for example, integrating the process parameter data with the work pattern data according to timestamp); (3) compiling performance data corresponding with the work pattern data, the performance data including (but not limited to) safety parameters (including the occurrence of any work accidents / injury / equipment damage / product damage / product waste / supply waste, etc.), process efficiency and optimization data, product performance data and / or acceptance information, etc., and / or other forms of performance data); (4) building a model (for example, an Al / machine learning / deep learning computer model, etc.) to correlate the work pattern data and / or process parameter data to the performance data, where the model may be trained such that any noteworthy trends in the work pattern data that are related to favorable and / or unfavorable performance data can be identified and used both for future business process reengineering (BPR) efforts, as well as in real time to avoid accidents, injuries and / or productivity loss, as described herein. In some embodiments, the camera 20 may be used to capture the operator’s hand movements. In some embodiments, the camera 20 may be used to capture the direction and / or background of the operator’s gaze.

[0160] Still referring to Fig. 17, in some embodiments, the one or more IR LEDs 18 may be used to illuminate the operator’s cornea and pupil while the one or more cameras 22 may be used in the IR range to record the reflections from the operator’s cornea and pupil. The recorded reflections from the cornea and pupil may be used to detect the eye movement. For example, the cornea reflection may be used as a reference point and pupil reflection as a moving point (i.e., to detect movement of the eye and / or to detect where the eye is focused). The positional relationship between the reference point and the moving point may be used to define the eye movement.

[0161] Referring still to Fig. 17, in some embodiments, the GPS chip 28 may be used to detect the location of the operator (for example, within the facility 60 and / or relative to pieces of equipment and process locations). One or more combinations of the detected movements may be used, by the cloud, to compare the operator’s movements to a set of optimized standards, for example, to ensure the operator is looking in the right spot (i.e., paying attention) when performing a specific task. The comparison between the operator’s movements and the standards, may be used by the cloud, to predict critical deviations. At step 258, the method 248 may include sending recommended actions, by the cloud, to the software on the XR device 10, for the operator to optimize the execution of a task. In some embodiment, an alarm may be sent, by the cloud, to the software on the XR device 10, to warn the operator that a movement deviation has occurred (and / or is impending). At step 260, the method 248 may include adjusting work patterns, by the operator, according to the received recommendations. In some embodiments, the operator may oppose the recommended action, using a voice command. At step 261, the method 248 may revert to step 256 until the task is completed. At step 262, the method 248 may include storing a record of work pattern data and the results of the adjustments, in the cloud for future analysis.Optimization Models

[0162] Certain embodiments described herein make use of a model to correlate the work pattern data and / or process parameter data to the performance data. In certain embodiments, the model includes a machine learning module, also referred to herein as artificial intelligence or artificial intelligence software. As used herein, a machine learning module refers to a computer implemented process (e.g., a software function) that implements one or more specific machine learning algorithms, such as an artificial neural network (ANN), random forest, decision trees, support vector machines, and the like, in order to determine, for a given input, one or more output values. In certain embodiments, the input comprises alphanumeric data which can include numbers, words, phrases, lengthier strings, pictures, audio, or videos, for example. In certain embodiments, the one or more output values comprise values representing numeric values, words, phrases, other alphanumeric strings, pictures, audio, or videos. In certain embodiments, the one or more output values comprise an identification of one or more response strings (e.g., selected from a database).

[0163] In certain embodiments, machine learning modules implementing machine learning techniques are trained, for example using datasets that include categories of data described herein. Such training may be used to determine various parameters of machine learning algorithms implemented by a machine learning module, such as weights associated with layers in neural networks. In certain embodiments, once a machine learning module is trained, e.g., to accomplish a specific task such as identifying certain response strings, values of determined parameters are fixed and the (e.g., unchanging, static) machine learning module is used to process new data (e.g., different from the training data; e.g., infer a result) and accomplish its trained task without further updates to its parameters (e.g., the machine learning module does not receive feedback and / or updates). In certain embodiments, machine learning modules may receive feedback, e.g., based on automated review of accuracy or human user review of accuracy, and such feedback may be used as additional training data, to dynamically update the machine learning module. In certain embodiments, two or more machine learning modules may be combined and implemented as a single module and / or a single software application. In certain embodiments, two or more machine learning modules may also be implemented separately, e.g., as separate software applications. A machine learning module may be software and / or hardware. For example, a machine learning module may be implemented entirely as software, or certain functions of an ANN module may be carried out via specialized hardware (e.g., via an application specific integrated circuit (ASIC), field programmable gate arrays (FPGAs)).

[0164] In certain embodiments, machine learning modules implementing machine learning techniques may be composed of individual nodes (e.g., units, neurons). A node may receive a set of inputs that may include at least a portion of a given input data for the machine learning module and / or at least one output of another node. A node may have at least one parameter to apply and / or a set of instructions to perform (e.g., mathematical functions to execute) over the set of inputs. In certain embodiments, node instructions may include a step to provide various relative importance to the set of inputs using various parameters, such as weights. The weights may be applied by performing scalar multiplication (e.g., or other mathematical function) between a set of inputs values and the parameters, resulting in a set of weighted inputs. In certain embodiments, a node may have a transfer function to combine the set of weighted inputs into one output value. A transfer function may be implemented by a summation of all the weighted inputs and the addition of an offset (e.g., bias) value. In certain embodiments, a node may have an activation function tointroduce non-linearity into the output value. Non-limiting examples of the activation function include Rectified Linear Activation (ReLu), logistic (e.g., sigmoid), hyperbolic tangent (tanh), and softmax. In certain embodiments, a node may have a capability of remembering previous states (e.g., recurrent nodes). Previous states may be applied to the input and output values using a set of learning parameters.

[0165] A layer is a building block in a deep learning architecture composed of nodes. In particular, a layer is a set of nodes that receives data input (e.g., weighted or non-weighted input), transforms it (e.g., by carrying out instructions, e.g., applying a set of functions e.g., linear and / or non-linear functions), and passes transformed values as output (e.g., to the next layer). In certain embodiments, the set of nodes in a particular layer may share the same parameters and instructions without interacting with each other. A machine learning module may be composed of at least one layer (e.g., ordered). Examples of types of layers include convolutional layers (e.g., layers with a kernel, a matrix of parameters that is slid across an input to be multiplied with multiple input values to reduce them to a single output value); fully connected (FC) layers (e.g., all nodes are connected to all outputs of the previous layer); recurrent layers, long / short term memory (LSTM) layers, gated recurrent unit (GRU) layers (e.g., nodes with the various abilities to memorize and apply their previous inputs and / or outputs); batch normalization (BN) layers (e.g., layers that normalize a set of outputs from another layer, allowing for more independent learning of individual layers); activation layer (e.g., layers with nodes that only contain an activation function); (un)pooling layers [e.g., layers that reduce (increase) dimensions of an input by summarizing (splitting) input values in defined patches).

[0166] In certain embodiments, the performance of a machine learning module may be characterized by its ability to produce an output data that reproduces an input data with specific accuracy. To achieve specific accuracy, a training process is performed to find optimal parameters, such as weights, for every node in every layer of the machine learning module. In certain embodiments, the training process of a machine learning module may involve using output data to calculate an objective function (e.g., cost function, loss function, error function) that needs to be optimized (e.g., minimized, maximized). For example, a machine learning objective function may be a combination of a loss function and regularization parameter. The loss function is related to how well the output is able to predict the input. The loss function may take various forms, likemean squared error, mean absolute error, binary cross-entropy, categorical cross-entropy, for example. The regularization term may be needed to prevent overfitting and improve generalization of the training process. Typical regularization techniques include LI Regularization or Lasso Regression, L2 Regularization or Ridge Regression, and Dropout (e.g., dropping layer outputs at random during training process).

[0167] In certain embodiments, objective function optimization of a machine learning module may involve finding at least one (e.g., all) of the present global optima (e.g., as opposed to local optima). A typical algorithm for objective function optimization follows principles of mathematical optimization for a multi-variable function and relies on achieving specific accuracy of the process. Examples of objective function optimization algorithms include gradient descent, nonlinear conjugate gradient, random search, Levenberg-Marquardt algorithm, limited-memory Broyden-Fietcher-Goldfarb-Shanno algorithm, pattern search, basin hopping method, Krylov method, Adam method, genetic algorithm, particle swarm optimization, surrogate optimization, and simulated annealing.

[0168] In certain embodiments, available input data includes training data and validation data, e.g., where the validation data is separate and non-overlapping with the training data. Training data is used during the training process to optimize a model, whereas validation data is used to check the accuracy of the model while operating on previously unseen data. In certain embodiments, training data is divided into batches (e.g., portions) that is sequentially used (e.g., in random order) as sets of inputs to train a model. In certain embodiments, a model is trained multiple times (e.g., epochs) on the entire set of training data.

[0169] In some embodiments according to the present disclosure, the machine learning module may include a hybrid architecture that includes a convolutional neural network (CNN) for recognition of objects and other features in video recordings (i.e., captured by the XR device 10), as well as a recurrent neural network (RNN) for sequential pattern recognition in order to predict patterns that may be correlated with accidents (in one embodiment). For example, the convolutional neural network (CNN) may be trained via recorded XR device 10 video that is tagged with certain labels describing the equipment that is involved and / or the process(es) that is / are being carried out. The CNN may be trained via feature recognition algorithms (i.e., edge detection algorithms, Feature from accelerated segment test (FAST) algorithms, local FASTalgorithms, global FAST algorithms, and / or other methods to identify visual features / objects / characteristics in the recorded videos. In connection with feature and / or characteristic recognition, the CNN may employ RGB (red-green-blue) color space as well as HSV (Hue image, Saturation image, and Value image) techniques.

[0170] Once a CNN has run across (i.e., convoluted) each pixel with an image / frame and each image / frame withing of a video to identify features and characteristics, the hybrid architecture can generate a time-dependent feature map (that is, a sequential mapping of the features and characteristics identified by the CNN as a function of time). The time-dependent feature map may then be fed into (i.e., inputted into) the RNN, which evaluates the time-dependent feature map for patterns and / or sequences of features that are correlated with the occurrence of accidents and / or other hazardous or unsafe conditions (in one embodiment). In embodiments where the CNN produces a time-dependent feature map based solely on one or more images, a time stamp associated with each of the one or more images may be used to establish the time-dependency (i.e., the feature sequence). In some embodiments, the CNN produces a linear output layer (for example, an output layer that connects each neuron to corresponding neurons from previous layers in a linear series). The time-dependent feature map may then be generated by adding a time stamp to each linear output layer (LOL) of the CNN. In some embodiments, the CNN produces one linear output layer for each frame of a video.

[0171] In some embodiments, the CNN may use one linear output layer to cover multiple frames of the video (for example, in instances where the video is unchanging across multiple frames). In some embodiments, the CNN may use multiple linear output layers to cover a single frame of the video (for example, in instances where a single frame of the video is so involved or detailed that multiple linear output layers are required). In each case, for each CNN linear output layer, the time-dependent feature map will include at least one time stamp indicating the time or range of times associated with the corresponding linear output layer, such that the RNN may arrange the linear output layers in a sequential or time-dependent manner. In some embodiments, the time-dependent feature map includes the plurality of CNN linear output layers (LOL) and accompanying time stamps arranged in a serialized format with alternating linear output layers and accompanying timestamps (i.e., LOL1, Timestampl, LOL2, Timestamp2, LOL3, Timestamp3) for inputting into the RNN. In some embodiments, the time-dependent feature map comprises aserialized data format (for example, including but not limited to a serialized data format comprising one or more JavaScript Object Notation (JSON) data files (i.e., data files that include a JSON format)).

[0172] In some embodiments, the RNN may be trained using videos and / or images that have been tagged as (1) including or not including an accident, (2) are correlated with hazardous conditions such as the occurrences of fires, exposure to hazardous chemicals and / or materials such as carbon monoxide, biological hazards, other hazardous materials, the presence of combustible materials, and / or (3) identifying personnel not wearing the proper personal protection equipment (PPE). The RNN ultimately arrives at a plurality of weighting factors or weights that relate the features and characteristics on the time-dependent feature map to one or more outcomes (i.e., the likelihood of an accident, in one example).

[0173] In operation, in some embodiments, the machine learning module sends an alert (i., a visible alert and / or an audible alert) to the XR device 10 so the operator is aware of the hazardous conditions. In some embodiments, the machine learning module also sends an alert to the cloud. In some embodiments, the CNN is trained and operates in an unsupervised fashion (for example, rather than identifying various preselected features or objects, the CNN simply identifies features (i.e., without “knowing” or identifying what the feature is) that are correlated with the occurrence of accidents and / or unsafe conditions). As such, in some embodiments, the machine learning module may be considered to be at least partially unsupervised.

[0174] In some embodiments, the machine learning module includes a CNN and RNN that work iteratively. For example, in some embodiments, after a CNN has generated the timedependent feature map, and after the RNN has evaluated the time-dependent feature map to identify patterns and / or sequences that are correlated with the occurrence of accidents and / or hazardous conditions, the RNN may first generate weights (i.e., weighting factors) that correlate the inputs (i.e., features from the time-dependent feature map) to the outputs (i.e., the likelihood of an accident occurring). For example, the RNN generates a set of unique weights, each weight correlating one of the objects, characteristics, and / or other features of the time-dependent feature map to a likelihood that hazardous conditions are present. Weight derivatives can then be generated to quantify the difference between the prediction value from the RNN and the training data (i.e., the control for each feature, etc.). These weight derivatives can then be back propagatedsuch that the weights determined by the RNN can be updated based on the generated weight derivatives.

[0175] In the above example, the machine learning module including the hybrid architecture (i.e., including both a CNN and an RNN) has been explained using a safety / accident prevention use case. However, in some embodiments, other use cases are possible in connection with the disclosed hybrid architecture. For example, in some embodiments, the machine learning module including the hybrid architecture (i.e., including both a CNN and an RNN) may be used with the end goal of optimizing productivity of a facility. In some embodiments, the machine learning module including the hybrid architecture (i.e., including both a CNN and an RNN) may be used with the end goal of optimizing the product quality (i.e., avoiding quality control issues) of a production line or manufacturing facility. In some embodiments according to the present disclosure, the machine learning module including the hybrid architecture (i.e., including both a CNN and an RNN) may be used with other end goals in mind.

[0176] Fig. 19 is a flow chart diagram of a method 360 of identifying hazardous conditions within a workplace, according to aspects of the present embodiments. At step 362, the method 360 may include obtaining videos and / or images of the workplace, by an XR device 10. At step 364, the method 360 may include inputting the videos and / or images into a machine learning module. In some embodiments, the machine learning module includes a hybrid architecture including a convolutional neural network (CNN) and a recurrent neural network (RNN). In some embodiments, the CNN is used for recognition of objects, characteristics, and / or other features in the videos and / or images. In some embodiments, the CNN is trained via a feature recognition algorithm, such as an edge detection algorithm, a Feature from accelerated segment test (FAST) algorithm, a local FAST algorithm, a canny edge detector, and a global FAST algorithm. In some embodiments, the RNN is used for sequential pattern recognition of the objects, characteristics, and / or other features in the videos and / or images. In some embodiments, the RNN is trained using videos and / or images that have been tagged as: (1) including or not including an accident, (2) are correlated or not correlated with hazardous conditions such as the occurrences of fires, exposure to hazardous chemicals and / or materials such as carbon monoxide, biological hazards, hazardous materials, the presence of combustible materials, and / or (3) identifying personnel not wearing proper personal protection equipment (PPE).

[0177] Still referring to Fig. 19, at step 366, the method 360 may include convoluting, by the CNN, the videos and / or images to identify the objects, characteristics, and / or other features therein. At step 368, the method 360 may include producing, by the CNN, a time-dependent feature map including the objects, characteristics, and / or other features identified by the CNN, and the corresponding timestamps. In some embodiments, the time-dependent feature map includes a serialized format including multiple linear output layers alternating with the corresponding timestamps, arranged in a single, serialized arrangement. The serialized arrangement, for example, includes one or more data files formatted in a JSON format. At step 370, the method 360 may include inputting the time-dependent feature map into the RNN. At step 372, the method 360 may include processing the time-dependent feature map, by the RNN, to establish weights correlating each of the objects, characteristics, and / or other features of the time-dependent feature map to a likelihood that hazardous conditions are present. At step 374, the method 360 may include producing for each of the objects, characteristics, and / or other features of the time-dependent feature map, by the RNN, a prediction of a likelihood that hazardous conditions are present in the videos and / or images of the workplace. In some embodiments, method 360 may further include comparing each of the predictions produced by the RNN to a control specific to each feature used for training the RNN, thereby producing a weight derivative between each prediction and each control, and back propagating each of the weight derivatives to each corresponding weight, thereby adjusting each corresponding weight such that updated predictions may be produced by the RNN for each of the objects, characteristics, and / or other features of the time-dependent feature map.EQUIVALENTS

[0178] Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments of the disclosure described herein. Therefore, the scope of the present disclosure is not intended to be limited to the above Description.

Claims

CLAIMSWhat is claimed:

1. A method of hands-free access to a cloud comprising: logging into software installed on an extended reality (XR) device, by an operator, enabling an authenticated e-signature; establishing a connection, by the XR device, to a private cloud; initiating a task, by the operator and / or a third-party user; sending a task identifier associated with the initiated task, by the software on the XR device, to the cloud; sending at least one task related data cluster, by the cloud to the XR device; displaying at least one display-only field of the at least one task related data cluster, on the XR device screen; presenting at least one editable field of the at least one task related data cluster, by the XR device, to the operator; updating the at least one editable field of the at least one task related data cluster, by the operator; sending the at least one updated task related data cluster, by the software on the XR device, to the cloud; and replacing the at least one original task related data cluster with the at least one updated task related data cluster, by the cloud.

2. The method of claim 1, wherein logging into the software comprises using voice commands.

3. The method of claim 2, wherein using voice commands comprises detecting a wake word by the software on the XR device.

4. The method of claim 3, wherein the detected wake word prompts an automatic speech recognition program or application.

5. The method of claim 4, further comprising: detecting, by the automatic speech recognition program or application, one or more login credentials; and sending the one or more login credentials, by the software on the XR device, to the cloud.

6. The method of claim 1, wherein logging into the software comprises using a QR code generated by a computer and / or a mobile phone.

7. The method of claim 6, further comprising scanning the QR code, by a camera installed on the XR device.

8. The method of claim 7, wherein the QR code is scanned by a reader integrated with functionality of the camera.

9. The method of claim 7, wherein the QR code is scanned by an external application, installed on the XR device.

10. The method of claim 1, wherein the cloud comprises a Wi-Fi 6 network.

11. The method of claim 10, wherein connection to the cloud is established using a Wi-Fi 6 router.

12. The method of claim 1, wherein the cloud comprises a private 5G mobile network.13 The method of claim 12, wherein connection to the cloud is established using a Radio Access Network (RAN).

14. The method of claim 1, wherein initiating a task comprises scanning a QR code, by the operator.

15. The method of claim 1 , wherein initiating a task comprises detecting a text, by the operator.

16. The method of claim 15, further comprising detecting the text by a camera installed on the XR device.

17. The method of claim 1, wherein initiating a task comprises making at least one selection on a remote device, by a third-party user.

18. The method of claim 1, comprising transmitting data between the at XR device and the cloud via a bidirectional data flow, wherein transmitting data via the bidirectional data flow comprises: transmitting data via a first data field using a permanent bidirectional data flow; and transmitting data via a second data field using an alternating bidirectional data flow.

19. The method of claim 18, wherein the first data field comprises the at least one task related data cluster, and wherein the second data field comprises the at least one editable data field.

20. The method of claim 1, wherein the task includes a training session, an inspection session, a work order, inspecting a piece of equipment, inspecting a product, and / or a quality control session.

21. The method of claim 1, further comprising sending a request, by the cloud, to the software on the XR device, to collect task related signals.

22. The method of claim 1 , further comprising collecting and sending the requested task related signals, by the software on the XR device, to the cloud.

23. The method of claim 1, further comprising displaying the editable fields of the at least one task related data cluster as texts, pictures, audio and / or videos.

24. The method of claim 1, further comprising presenting at least one editable field of the at least one task related data cluster, by the XR device, to the operator, using one or more available data forms.

25. The method of claim 24, wherein the data forms enable entry of at least one of a checkbox, numeric values, time values, a dropdown menu, a query, and / or picture / video / audio data.

26. The method of claim 1, wherein updating the editable data field comprises selecting a value for the data form, by the operator.

27. The method of claim 26, wherein selecting a value for the data form comprises using voice commands, by the operator.

28. The method of claim 26, wherein selecting a value for the data form comprises using hand gestures, by the operator.

29. The method of claim 26, wherein selecting a value for the data form comprises using a navigation button, by the operator.

30. The method of claim 26, wherein selecting a value for the data form includes using a camera, by the operator, to detect at least one text string.

31. A method of requesting data using a QR code, comprising: capturing the QR code, by a QR reader; deciphering the captured QR code into a text, by the QR reader; detecting a task identifier from the deciphered text, by software installed on an extended reality (XR) device; sending the task identifier, by the software installed on the XR device, to the cloud; retrieving data, by the cloud, using the received task identifier; and sending the retrieved data, by the cloud, to the XR device.

32. The method of 31 , wherein the QR reader is integrated within the functionality in a camera.

33. The method of 31, wherein the QR reader is an external application, installed on the XR device, and wherein the XR device comprises an XR headset or XR glasses.

34. The method of 31, wherein the task identifier comprises:information about data stored in the cloud associated with the task identifier; information about a data cluster within a stored data set; and a request to receive the data cluster.

35. An XR device, comprising: software installed on the XR device, the software being capable of collecting image, audio and video signals; a processor connecting the software to a private cloud; a camera communicatively coupled to the processor; a microphone communicatively coupled to the processor; and a speaker communicatively coupled to the processor; wherein the software is configured to send a task identifier to the cloud, and display at least one task related data cluster, received from the cloud, on the XR device to be viewed and / or updated by the operator.

36. The system of claim 35, wherein the XR device further comprises a GPS chip.

37. The system of claim 35, wherein the XR device further comprises a navigation button.

38. The system of claim 35, wherein the XR device further comprises one or more IR LEDs.

39. A method of analyzing work pattern data comprising: compiling a database of work pattern data, the work pattern data comprising at least one of video data, eye movement data, hand movement data, and arm movement data;compiling performance data corresponding with the work pattern data, the performance data comprising at least one of: safety parameters, process efficiency data, and product performance data; and building a computer model to correlate the work pattern data with the performance data, wherein the computer model is trained via a hybrid machine learning architecture comprising at least a convolutional neural network (CNN) and a recurrent neural network (RNN) such that any noteworthy trends in the work pattern data that are related to favorable and / or unfavorable performance data are identifiable by the computer model.

40. The method of claim 39, wherein the computer model employs at least one of artificial intelligence, machine learning, and deep learning.

41. The method of claim 39, further comprising compiling at least one process parameter data point, the at least one process parameter data point being associated with the work pattern data, and integrating the at least one process parameter data point with the work pattern data, wherein integrating the at least one process parameter data point with the work pattern data comprises integration via at least one common timestamp.

42. A method of identifying hazardous conditions within a workplace, the method comprising: obtaining videos and / or images of the workplace via an XR device; inputting the videos and / or images into a machine learning module, the machine learning module comprising a hybrid architecture comprising at least (1) a convolutional neural network (CNN) used for recognition of objects, characteristics, and / or other features in the videos and / or images, and (2) a recurrent neural network (RNN) used for sequential pattern recognition of the objects, characteristics, and / or other features in the videos and / or images;convoluting, by the CNN, the videos and / or images to identify the objects, characteristics, and / or other features therein; producing, by the CNN, a time-dependent feature map comprising the (1) objects, characteristics, and / or other features identified by the CNN, and (2) corresponding timestamps; inputting the time-dependent feature map into the RNN; processing the time-dependent feature map, by the RNN, to establish weights correlating each of the objects, characteristics, and / or other features of the time-dependent feature map to a likelihood that hazardous conditions are present; and producing for each of the objects, characteristics, and / or other features of the timedependent feature map, by the RNN, a prediction of a likelihood that hazardous conditions are present in the videos and / or images of the workplace.

43. The method of claim 42, further comprising: comparing each of the predictions produced by the RNN to a control specific to each feature used for training the RNN, thereby producing a weight derivative between each prediction and each control; and back propagating each of the weight derivatives to each corresponding weight, thereby adjusting each corresponding weight such that updated predictions may be produced by the RNN for each of the objects, characteristics, and / or other features of the time-dependent feature map.

44. The method of claim 42, wherein the CNN is trained via a feature recognition algorithm comprising at least one of an edge detection algorithm, a feature from accelerated segment test (FAST) algorithm, a local FAST algorithm, a canny edge detector, and a global FAST algorithm.

45. The method of claim 42, wherein the RNN is trained using videos and / or images that have been tagged as: (1) including or not including an accident; (2) are correlated or not correlated with hazardous conditions such as the occurrences of fires, exposure to hazardous chemicals and / or materials such as carbon monoxide, biological hazards, hazardous materials,the presence of combustible materials, and / or (3) identifying personnel not wearing proper personal protection equipment (PPE).

46. The method of claim 42, wherein the time-dependent feature map comprises a serialized format comprising multiple linear output layers alternating with the corresponding timestamps, arranged in a single, serialized arrangement.

47. The method of claim 46, wherein the serialized arrangement comprises one or more data files formatted in a JSON format.

48. A method of enhancing quality control of a workplace or optimizing productivity of the workplace comprising: obtaining videos and / or images of the workplace via an XR device; inputting the videos and / or images into the machine learning module of claim 42; and performing the convoluting, producing, inputting, processing, and producing steps of claim 42.

49. The machine learning module of claim 42.

50. The method of claim 17, wherein making at least one selection on a remote device comprises:(1) selecting a binary input, by the operator, corresponding to a refrigeration status of a selected item;(2) recording a timestamp automatically, by the cloud, corresponding to the binary input; and(3) calculating a time out of refrigeration (TOR) parameter for the selected item.

51. The method of claim 50, wherein calculating a time out of refrigeration (TOR) parameter comprises calculating at least one of a total accumulated time out of refrigeration, an uninterrupted time out of refrigeration, and a current time out of refrigeration.

52. The method of claim 50, wherein selecting the binary input comprises changing the binary input, thereby reflecting a change in refrigeration status, and wherein calculating a time out of refrigeration (TOR) parameter comprises tabulating TOR based at least in part on a timestamp corresponding to the change in the binary input.

53. The method of claim 7, further comprising transmitting a real-time video feed of from the camera to the cloud.

54. A method of requesting data using a task identifier, comprising: sending a task identifier, by software on an extended reality (XR) device, to the cloud, where the task identifier is received; retrieving data, by the cloud, using the received task identifier; and sending the retrieved data, by the cloud, to the XR device.

55. The method of claim 54, wherein the task identifier comprises: information about data stored in the cloud associated with the task identifier; information about at least one data cluster within the stored data; and a request to retrieve the at least one data cluster.

Images