Method and apparatus for augmented reality capture and display using distributed architecture

Abstract

A method and apparatus for capturing and displaying using a distributed architecture with LLM using an XR device is disclosed. A method includes capturing sensor data from a plurality of sensors in an environment and using an XR device, wherein the sensor data includes image data, audio data, and depth data. Using the sensor data, a world grid corresponding to a three-dimensional geometric representation of the environment is formed. Based on the image data, pose data indicative of respective positions and orientations of a plurality of objects present in the world grid is generated. Speech in the audio data is converted into text. Using the text data and the image data, a prompt is created. LLM is performed using the hint and a textual output is generated, which is provided to the XR device to produce a visual output.

Description

Technical Field

[0001] This disclosure relates to extended reality (XR) headsets, and more particularly to the use of multimodal large language models (MLLM) with XR headsets. Background Technology

[0002] Extended reality encompasses several different immersive technologies, including virtual reality (VR), augmented reality (AR), and mixed reality (MR). Using these technologies, the physical and digital (or virtual) worlds can be merged to some extent to create a variety of interactive experiences.

[0003] VR encompasses fully immersive experiences where users are placed within a virtual environment (from a sensory perspective). Users wear head-mounted displays that block at least some real-world sensory input (such as visual and auditory input), replacing it with input from a simulated environment. Visually, this environment can include a simulated three-dimensional space within the user's field of view. VR is commonly used in gaming and training simulations.

[0004] In AR, digitally generated elements are overlaid on a view of the real world. AR users can use devices that do not obstruct real-world sensor input (such as head-mounted displays or smartphones). Alternatively, AR can add computer-generated content to, for example, a view of the real world. A notable AR game offered as a mobile app is Pokémon Go.

[0005] MR combines elements of both AR and VR, allowing for real-time interaction between real-world objects (e.g., in the user's field of view) and virtual-world objects (generated using devices capable of generating VR).

[0006] XR technologies using various combinations of the aforementioned elements can be implemented, for example, in certain types of head-mounted viewers. Using such technologies, spatial mapping can be performed to allow images of digital objects to be overlaid on a real-world view, while allowing users to interact with them using gestures, eye tracking, voice commands, etc. Such devices can be used in fields such as entertainment, training, healthcare, and industrial applications, thereby providing immersive experiences that bridge physical and virtual environments. Summary of the Invention

[0007] A method and apparatus for capturing and displaying data using an XR device with a distributed architecture featuring LLM are disclosed. In one embodiment, the method includes capturing sensor data from multiple sensors of an extended reality device in an environment, including image data, audio data, and depth data. The method further includes forming a world mesh corresponding to a three-dimensional geometric representation of the environment using the sensor data, and generating pose data based on the image data indicating the corresponding positions and orientations of multiple objects present in the world mesh. The method also includes converting speech in the audio data into text data. Subsequently, the method includes creating a cue using the text data and selected image data, and performing multiple Large Language Models (LLMs) in parallel using the cue to generate text output. Based on this execution, the method includes combining the text output with a three-dimensional projection generated based on the world mesh and pose data in the extended reality device to produce visual output, and providing the visual output to a display of the extended reality device. Attached Figure Description

[0008] Figure 1 A system 100 for training neural networks is shown.

[0009] Figure 2 A computer-implemented method 200 for training a neural network is shown.

[0010] Figure 3A This is a diagram illustrating one embodiment of a system utilizing LLM and XR devices.

[0011] Figure 3B This is a diagram illustrating a method for operating a system utilizing LLM and XR devices according to one embodiment.

[0012] Figure 3C This is a flowchart of one embodiment of a method for operating an XR device.

[0013] Figure 4A This is a diagram illustrating the operation of an application using XR devices and LLM.

[0014] Figure 4B This is a diagram further illustrating the operation of applications using XR devices and LLM.

[0015] Figure 4C This is a diagram illustrating an example of a web-based dashboard used for a manual evaluation of the performance of the system disclosed herein.

[0016] Figure 5 A diagram depicting the interaction between the computer-controlled machine 510 and the control system 512 is provided. Detailed Implementation

[0017] Embodiments of this disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various forms and alternative forms. The figures are not necessarily to scale; some features may be enlarged or minimized to show details of particular components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to adopt the embodiments in various ways. As will be understood by those skilled in the art, various features illustrated and described with reference to any of the figures may be combined with features illustrated in one or more other figures to produce embodiments not explicitly illustrated or described. The combinations of illustrated features provide representative embodiments for typical applications. However, various combinations and modifications of features consistent with the teachings of this disclosure may be desirable for a particular application or implementation.

[0018] As used herein, “a,” “an,” and “the” refer to both the singular and plural indicators, unless the context clearly specifies otherwise. As an example, a “processor” programmed to perform various functions refers to one processor programmed to perform each of the various functions, or more than one processor programmed together to perform each of the various functions.

[0019] Extended reality (XR) devices enhance user interaction with their environment in an immersive way. Types of XR devices include head-mounted displays, glasses, smartphones, tablets, and head-up displays, and include local processors for processing various types of data, such as image and audio data. However, the processing power of XR devices is limited in some functionalities. For example, typical XR devices lack the processing power to run large language models (LLMs), which include multimodal LLMs (MLLMs, which ingest multiple data types, such as image and audio data). Typical LLMs are resource-intensive and cannot be run efficiently on mobile XR devices.

[0020] As development of general-purpose LLMs continues to advance, considerable effort has been invested in making them operable on mobile platforms. These efforts may include quantization and sparsification. While this improves memory utilization, reduces latency, and decreases network requirements, it does so at the cost of reduced descriptivity, accuracy, and general applicability. Robust LLMs, on the other hand, can utilize billions and trillions of parameters. Such LLMs can be trained on massive datasets, enabling them to provide more detailed and imaginative responses across a wide range of topics. However, their size prevents them from running on edge devices such as mobile XR devices. Instead, the current processing power required to run LLMs necessitates running them on powerful servers.

[0021] Recognizing the limitations of running LLM on edge devices, such as XR headsets, this disclosure combines the functionality of LLM processing in the cloud while handling XR-related tasks on the headset of an XR device. This, in turn, provides a balance between performance and resource usage.

[0022] An example application of the method disclosed in this paper is a "cognitive assistant" that displays step-by-step instructions for performing a task, updating the instructions only when it has confirmed that the user has completed a specific step of the process. While significant progress has been made in developing such instruction-following assistants in XR, many such systems require substantial human effort to write XR instructions for each specific task. These processes are labor-intensive and lack adaptability across different tasks. Furthermore, these systems typically provide fixed responses and lack the ability to dynamically adjust the guidelines based on user interaction. Various implementations of the method disclosed in this paper overcome these problems.

[0023] LLM has demonstrated significant potential to power virtual assistants across diverse domains, such as programming, personal tasks, and medical diagnostics. However, recent advances in multimodal artificial intelligence (AI) and perception technologies have led to “intelligent” assistants capable of interpreting and responding to the physical environment using customized responses. For example, certain types of glasses can leverage AI inference to analyze images captured by the glasses and can provide insights into the user’s surroundings. Similarly, some AI assistants may use visual and audio cues to perceive and respond within the context of the user’s physical environment. While these types of systems excel at perceiving their environment, they are limited to displaying fixed, static outputs and lack the ability to dynamically project and anchor responses back to the environment.

[0024] The methods and systems disclosed herein overcome these problems by partitioning processing tasks between XR devices and LLM processing in the cloud or data center. Various embodiments will now be discussed in further detail.

[0025] Figure 1 A system 100 for training a neural network (e.g., a deep neural network) is shown. The neural network or deep neural network shown and described is merely an example of the types of machine learning networks or neural networks that can be used. System 100 may include an input interface for accessing training data 102 of the neural network. For example, as Figure 1As illustrated, the input interface can be comprised of a data storage interface 104, which can access the training data 102 from the data storage device 106. For example, the data storage interface 104 can be a memory interface or persistent storage interface, such as a hard disk or SSD interface, but it can also be a private, local area, or wide area network interface, such as a Bluetooth, Zigbee, or Wi-Fi interface, or an Ethernet or fiber optic interface. The data storage device 106 can be an internal data storage device of the system 100, such as a hard disk drive or SSD, but it can also be an external data storage device, such as a network-accessible data storage device.

[0026] In some embodiments, data storage device 106 may further include a data representation 108 of an untrained version of the neural network, which system 100 can access from data storage device 106. However, it will be appreciated that the training data 102 and the data representation 108 of the untrained neural network may each be accessed from different data storage devices, for example, via different subsystems of data storage interface 104. Each subsystem may have the type of data storage interface 104 as described above. In other embodiments, the data representation 108 of the untrained neural network may be generated internally by system 100 based on the design parameters of the neural network, and therefore may not be explicitly stored on data storage device 106. System 100 may further include a processor subsystem 110, which may be configured to provide an iterative function during operation of system 100 as an alternative to a stack of neural network layers to be trained. Here, the corresponding layers of the replaced layer stack may have mutually shared weights and may receive the output of the previous layer as input, or, for the first layer of the layer stack, receive the initial activation and a portion of the input of the layer stack as input. The processor subsystem 110 can be further configured to iteratively train the neural network using training data 102. Here, the training iterations of the processor subsystem 110 may include a forward propagation part and a backward propagation part.

[0027] The processor subsystem 110 can be configured to perform the forward propagation portion by: determining an equilibrium point of the iterative function, where the iterative function converges to a fixed point, in addition to defining other operations that can be performed on the forward propagation portion, wherein determining the equilibrium point includes finding the root solution of the iterative function by subtracting its input using a numerical root-finding algorithm; and by providing the equilibrium point as an alternative to the output of the layer stack in the neural network.

[0028] System 100 may further include an output interface for outputting a data representation 112 of the trained neural network, which may also be referred to as trained model data 112. For example, as well as Figure 1As illustrated, the output interface can be comprised of a data storage interface 104, which in these embodiments is an input / output (“IO”) interface through which trained model data 112 can be stored in data storage device 106. For example, the data representation 108 defining an “untrained” neural network can be replaced, at least partially replaced, with the data representation 112 of the trained neural network during or after training, because the parameters of the neural network (such as the network weights, hyperparameters, and other types of parameters) can be adapted to reflect training on the training data 102. Figure 1 This is also illustrated by reference numerals 108 and 112, which refer to the same data records on data storage device 106. In other embodiments, data representation 112 may be stored separately from data representation 108, which defines the "untrained" neural network. In some embodiments, the output interface may be separate from data storage interface 104, but it can generally have the type of data storage interface 104 as described above.

[0029] Systems for training neural networks can be used in applications including those performing LLM processing for XR devices. Training can be performed on audio and image data from the XR device. This training may include generating cues using the image and audio data from the XR device, where the cues are used for further training on one or more LLMs, and in some instances, further training along with text (speech-to-text conversion) and image data. In some embodiments, the LLM may be an MLLM, and it may generate text output that can be provided back to the XR device. The text output can be used, for example, to provide instructions or guidance to the user of the XR device, generate AR overlays, etc.

[0030] Figure 2 A system 200 is described that implements the machine learning models described herein (e.g., deep neural networks used in conjunction with XR devices, such as LLM / MLLM used in conjunction with XR devices). Other types of machine learning models can be used, and the DNN described herein is not the only type of machine learning model that can be used in the systems disclosed herein. For example, a CNN can be utilized if the input image associated with a process controlled by a PLC contains an ordered sequence of pixels after the CSI values ​​have been converted into pixels in the image.

[0031] System 200 may be implemented to perform one or more of the operational phases described herein, wherein LLM / MLLM processing is performed in conjunction with data received from an XR device, and the result of the processing is returned to the XR device. System 200 may include at least one computing system 202. Computing system 202 may include at least one processor 204 operatively connected to memory unit 208. Processor 204 may include one or more integrated circuits implementing the functions of central processing unit (CPU) 206. CPU 206 may be a commercially available processing unit implementing an instruction set such as ×86, ARM, Power, or MIPS instruction set families. During operation, CPU 206 may execute stored program instructions retrieved from memory unit 208. The stored program instructions may include software controlling the operation of CPU 206 to perform the operations described herein. In some examples, processor 204 may be a system-on-a-chip (SoC) that integrates the functions of CPU 206, memory unit 208, network interface, and input / output interface into a single integrated device. The computing system 202 can implement an operating system for managing various aspects of operations. Although in Figure 2 The diagram shows a processor 204, a CPU 206, and a memory 208, but of course, more than one of each can be used in the overall system.

[0032] Memory cell 208 may include volatile and non-volatile memory for storing instructions and data. Non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 202 is disabled or loses power. Volatile memory may include static and dynamic random access memory (RAM) for storing program instructions and data. For example, memory cell 208 may store machine learning model 210 or algorithm, training dataset 212 of machine learning model 210, and original source dataset 216.

[0033] The computing system 202 may include a network interface device 222 configured to provide communication with external systems and devices. For example, the network interface device 222 may include wired and / or wireless Ethernet interfaces as defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 series of standards. The network interface device 222 may include a cellular communication interface for communicating with cellular networks (e.g., 3G, 4G, 5G). The network interface device 222 may be further configured to provide a communication interface to an external network 224 or the cloud.

[0034] External network 224 may be referred to as the World Wide Web or the Internet. External network 224 can establish standard communication protocols between computing devices. External network 224 can allow for the easy exchange of information and data between computing devices and the network. One or more devices 230 (such as XR devices) can communicate with external network 224. External network can facilitate bidirectional communication between one or more devices and computing system 202. For example, external network 224 can facilitate communication between the computing system and the XR device disclosed herein, receiving image and audio data, and returning data from the execution of one or more LLM / MLLMs by the computing system 202.

[0035] The computing system 202 may include an input / output (I / O) interface 220, which may be configured to provide digital and / or analog inputs and outputs. The I / O interface 220 is used to transfer information between internal storage devices and external input and / or output devices (e.g., HMI devices). The I / O interface 220 may include associated circuitry or a bus (BUS) network to transfer information to or between one or more processors and storage devices. For example, the I / O interface 220 may include digital I / O logic lines that can be read or set by one or more processors, handshake lines for monitoring data transfers via the I / O lines, timing and counting facilities, and other known structures for providing such functionality. Examples of input devices include keyboards, mice, sensors, etc. Examples of output devices include monitors, printers, speakers, etc. The I / O interface 220 may include additional serial interfaces (e.g., Universal Serial Bus (USB) interfaces) for communicating with external devices. I / O interface 220 can be referred to as an input interface (because it transmits data from external inputs (such as sensors)) or an output interface (because it transmits data to external outputs (such as displays)).

[0036] The computing system 202 may include a human-machine interface (HMI) device 218, which may include any device that enables the system 200 to receive control input. Examples of input devices may include HMI inputs such as XR devices (e.g., glasses, head-mounted displays), but may also include keyboards, mice, touchscreens, voice input devices, and other similar devices. The computing system 202 may include a display device 232. The computing system 202 may include hardware and software for outputting graphical and textual information to the display device 232. The display device 232 may include an electronic display screen, projector, printer, or other suitable device for displaying information to a user or operator. The computing system 202 may be further configured to allow interaction with remote HMIs and remote display devices via a network interface device 222.

[0037] System 200 can be implemented using one or more computing systems. While this example depicts a single computing system 202 implementing all the described features, the intention is to decouple and implement various features and functions through multiple computing units communicating with each other. The specific system architecture chosen can depend on a variety of factors.

[0038] System 200 can implement machine learning algorithm 210 configured to analyze raw source dataset 216. Raw source dataset 216 may include raw or unprocessed sensor data that may represent the input dataset for the machine learning system. Raw source dataset 216 may include video, video clips, images, text-based information, audio or human speech, time-series data (e.g., pressure sensor signals over time), raw or partially processed sensor data (e.g., radar maps of objects), and wireless signals according to CSI, RSSI, or CIR. Furthermore, raw source dataset 216 may be input data derived from associated sensors, such as cameras, lidar, radar, ultrasonic sensors, motion sensors, thermal imaging cameras, wireless receivers, or any other type of sensor that produces associated data with spatial dimensions (where some concept of "foreground" and "background" exists within those spatial dimensions). The input or input "image" mentioned herein does not necessarily originate from a camera but may originate from any of the sensors listed above. Other types of sensors (such as temperature and pressure sensors) may also provide various inputs to the system. Several different examples of inputs are shown and described with reference to other figures in this disclosure. In some examples, the machine learning algorithm 210 may be a neural network algorithm (e.g., a deep neural network) designed to perform a predetermined function. For example, the neural network algorithm may be configured to identify defects (e.g., cracks, stresses, bumps, etc.) in a part after it has been manufactured but before it leaves the factory.

[0039] Computer system 200 may store a training dataset 212 for machine learning algorithm 210. Training dataset 212 may represent a previously constructed dataset used to train machine learning algorithm 210. Machine learning algorithm 210 can use training dataset 212 to learn weighting factors associated with a neural network algorithm. Training dataset 212 may include a source dataset having a corresponding outcome or result that machine learning algorithm 210 attempts to replicate through the learning process.

[0040] Machine learning algorithm 210 can operate in learning mode using training dataset 212 as input. Machine learning algorithm 210 can be executed in multiple iterations using data from training dataset 212. For each iteration, machine learning algorithm 210 can update its internal weighting factors based on the achieved results. For example, machine learning algorithm 210 can compare its output (e.g., a reconstructed or supplemented image in the case of image data as input) with those results included in training dataset 212. Since training dataset 212 includes expected results, machine learning algorithm 210 can determine when performance is acceptable. After machine learning algorithm 210 reaches a predetermined performance level (e.g., 100% consistency with results associated with training dataset 212) or converges, machine learning algorithm 210 can be executed using data not in training dataset 212. It should be understood that in this disclosure, "convergence" can mean that a set (e.g., predetermined) number of iterations has occurred, or the residual is sufficiently small (e.g., the change in approximate probability within an iteration is less than a threshold), or other convergence conditions. The trained machine learning algorithm 210 can be applied to new datasets to generate annotated data.

[0041] Machine learning algorithm 210 can be configured to identify specific features in raw source data 216. Raw source data 216 may include multiple instances or input datasets expected to supplement the results. For example, machine learning algorithm 210 may be configured to identify certain aspects of a manufacturing process performed by automated equipment under the control of a program executed by a PLC. In another example, machine learning algorithm 210 may be configured to identify the presence of defects in a part produced by an automated process under the control of a PLC program by capturing images of the manufactured part. Machine learning algorithm 210 may be programmed to process raw source data 216 to identify the presence of specific features. Machine learning algorithm 210 may be configured to identify features in raw source data 216 as predetermined features (e.g., obstacles, pedestrians, road signs, etc.). Raw source data 216 can be derived from various sources. For example, raw source data 216 may be actual input data collected by a machine learning system. Raw source data 216 may be machine-generated for testing a system. As an example, raw source data 216 may include raw video images from a camera and raw audio data received from an XR device.

[0042] Figure 3AThis diagram illustrates one embodiment of a system utilizing an LLM and an XR device. In the example shown, system 300 includes an XR device 305 that interacts with an environment 340 and is also configured to communicate with a server 325. In various embodiments, the server 325 is located remotely from the XR device 305, although bidirectional communication occurs between the two via various channels during the operations described herein. Data sent from the XR device 305 to the server 325 is applied to one or more LLMs (e.g., MLLMs), which return text information to the XR device 305. Using the text information processed from the LLMs, the XR device can perform various actions, such as displaying instructions on a user-viewable AR overlay by the XR device 305.

[0043] like Figure 3A As shown, XR device 311 includes an audio / video processing unit 311 and physical input / output (I / O) 306, which includes various sensors, including cameras, microphones, and depth-sensing cameras. These sensors may be, for example, part of a head-mounted viewer or smart glasses, which may also include a head-up display (HUD). XR device 305 also includes the audio / video processing unit 311 and a decoder unit 315. These units can be implemented in XR device 311 using any suitable combination of software, hardware, and / or firmware. Server 325 includes an encoder 328, a cue generator 329, and an LLM processor 327. These units can also be implemented using any suitable combination of software, hardware, and / or firmware.

[0044] Various sensors, part of the XR device 305, capture information about the physical world, which is then passed to a server 325 for further processing. This information is processed by a head-mounted viewer to generate a world mesh and estimate the camera's pose, as well as perform basic filtering on the streaming data. The world mesh comprises several distinct points in three-dimensional (3D) space, corresponding to points on objects in the environment viewed by the XR device.

[0045] The audio / video processing unit 311 in the illustrated embodiment can capture both video and audio data and convert them into digital formats for further processing. Additionally, information regarding the depth of various objects observed in the real world within environment 340 can also be determined. The audio / video processor 311 is also configured to perform 3D world modeling to allow the XR device 305 to anchor virtual content to the physical world, such that the virtual content appears in the same place relative to the physical world, regardless of how the user moves their XR device 305. To achieve 3D world modeling with low network bandwidth overhead, the XR device 311 performs 3D modeling locally. Image frames from the XR device are tagged with unique identifiers, timestamps, and camera poses indicating their captured positions before transmission to the server. The camera pose is determined by a tracking algorithm specific to the XR device 311, which calculates its position relative to the local origin of the tracking space. The audio / video processing unit 311 is also configured to generate a world mesh for environment 340 using both captured image frames and depth sensing information from a depth sensor.

[0046] Frames captured by the audio / video processing unit 311 can be added to a user's text query for contextual information and forwarded to the encoder 328. Since the position and orientation of the XR device 305 may constantly change, the world mesh can be updated accordingly. The output of the decoder 328 can be paired with the world mesh map to project content into the user's (potentially changing) physical world view.

[0047] In the illustrated embodiment, the encoder 328 of server 325 is coupled to receive audio and image data from the audio / video processing unit 311 of XR device 305. The decoder 315 of XR device 305 is coupled to receive text output from the LLM processor 327 of server 325. Communication between XR device 305 and server 325 can be performed using any suitable communication method / protocol. For example, XR device 305 and server 325 can communicate with each other via a cellular network, WiFi, or both.

[0048] In the illustrated embodiment, encoder 328 is responsible for converting audio to text and capturing frames of interest, which should be passed to server 325 for processing in encoder 328 and LLM processor 327. As shown here, image data (which may be video frames) and audio data are transmitted from the sensors of XR device 311 to encoder 328 running on server 325. Note that in some embodiments, encoder 328 may be implemented on XR device 311 instead of server 325, thereby reducing network overhead. Speech present in the audio data received by encoder 328 can be converted into text data using a speech-to-text module. To convert the audio data, encoder 328 may perform ambient noise filtering and divide the audio stream into segments of predetermined duration that may contain user queries. Each segment may be augmented with additional context. For example, in the cognitive assistant application discussed below, the transcribed text query may be augmented to include supplementary information, such as the current set of instructions being executed.

[0049] When transforming image data generated by the XR device 305, multiple image frames are selected at a time for processing in the LLM processor 327. These frames can show a current view of the physical world as seen through the XR device 305, as well as snapshots of events that have previously occurred in the physical world. Frames of the past and current grounds of the LLM, executed in the LLM processor 327, are provided within the context of what has already happened.

[0050] The processed image and audio data can then be passed to the LLM processor 327 for processing by one or more LLMs, which may include MLLMs capable of processing audio and image data in addition to text data in the form of cues. In addition to the processing of the image and audio data by the encoder 328, the cue generator 329 can transform the processed data into cues that are provided to various LLMs running by the LLM processor 327.

[0051] The cues generated in the cue generator 329 and the audio / video data processed in the encoder 328 are both sent to the LLM processor 327. Since the server 325 has ample computing resources compared to XR devices, the provided API can enable the parallel execution of multiple MLLMs and potential remote cloud calls. An example implementation can use GPT-4V

[36] on both an OpenAI server and Ferret

[32] . Ferret is a specialized MLLM that can identify objects, discern relationships between multiple regions in an image, and provide 2D bounding boxes for object locations. In such an implementation, Ferret's spatial understanding can be combined with GPT-4V for inference, thereby optimizing the responsiveness of both. The responses of the two models can be combined using the LLM processor 327 to query both models simultaneously, using the response of GPT-4V for text feedback and the response of Ferret for object locations for anchoring AR content. Therefore, the inference time is limited by the slowest MLLM (which is typically GPT-4V). Faster models like GPT-4o could potentially reduce this overhead.

[0052] As shown here, decoder 315 is responsible for packaging the output received from LLM processor 327 into an intuitive visual interface. Simple language can be used for AR annotations provided by a cueing system of XR device 305, which can draw graphical primitives at 3D locations anchored in a world grid based on environment 340. This system can include a small dictionary of arrows and text boxes, but can also include more complex models and scripted interactions. Decoder 315 can use world grid and camera pose information received from audio / video processing unit 311 and the output from LLM processor 327 to generate 3D projections and augmented reality overlays. These can be projected onto the HUD of XR device 305 to display information usable by the wearer of the device. In the example shown here, the information includes arrows and text indicating the location where the wearer of XR device 311 can place a potted plant.

[0053] One of the main challenges in processing image data is converting 2D coordinates to 3D coordinates. For example, a 2D box generated by a Ferret model may need to be projected onto 3D data to anchor AR content to a scene. In the example shown, XR device 305 stores all previous camera poses and their associated image IDs in a lookup table. Once XR device 311 receives a response with the image ID from server 325, it looks up the associated camera pose. Using this pose, it projects rays into a stored 3D world mesh to obtain the 3D coordinates of the object when the image was captured. Text output is also displayed on the screen for the user in AR. Because XR device 305 tracks each virtual 3D object internally within its local tracking space, it does not need to process each image consecutively to update the object's position.

[0054] In one embodiment, since the object is not actively tracked after a frame is captured, the anchor can be presented at a fixed position and updated after a new frame is processed. Embodiments in which a post-processing step is added to the XR device 305, which performs continuous tracking of certain objects known to be dynamic, are possible and anticipated.

[0055] Figure 3B This is a diagram illustrating a method for operating a system utilizing LLM and XR devices according to one embodiment. The method 350 shown herein can be derived from the above references. Figure 3A Various embodiments of the XR device 305 and server 325 discussed are used to perform this.

[0056] Method 350 involves a user wearing a head-mounted viewer (XR device) and capturing data from the 3D physical world (box 352). The captured 3D physical world data is stored as a mesh in the XR device (box 354). Audio data and egocentric image data (i.e., image data from the user's perspective on the XR device) – which are labeled with camera pose (i.e., the position and orientation of objects) – are then streamed to a server (boxes 356 and 358). In the server, a high-level task description is combined with the audio data and egocentric image data to generate a cue, which is fed to multiple LLMs (GPT-4V and Ferret) to generate a combined textual and coordinate response (box 360).

[0057] The LLM response and 2D image space coordinate information are then provided back to the XR device (box 362). In this example, the LLM response includes the text instruction “Put the red wire [obj1] into the rightmost slot [obj2]”, with a corresponding overlay. This response is received at the XR device and projected by light into the mesh of the previously generated 3D world (box 364). The XR device then receives the virtual annotations and places them into the environment (box 366), allowing the user to see and execute the given instructions.

[0058] Figure 3C This is a flowchart of one embodiment of a method for operating an XR device. Method 370 can be performed by various embodiments of a system, including those connected to a server or other systems capable of running LLM (including...). Figure 3A The system shown is an XR device that communicates with other devices.

[0059] Method 370 includes capturing sensor data from multiple sensors of an augmented reality device in an environment and using the device, wherein the sensor data includes image data, audio data, and depth data (box 372). The method also includes forming a world mesh corresponding to a three-dimensional geometric representation of the environment using the sensor data (box 374), and generating pose data based on the image data indicating the corresponding positions and orientations of multiple objects present in the world mesh (box 376). The method further includes converting speech in the audio data into text data (box 378).

[0060] The method also includes creating a cue using text data and selected image data (box 380). The method then includes performing multiple Large Language Models (LLMs) in parallel using the cue to generate text output (box 382). Method 370 further includes combining the text output with a 3D projection generated based on world mesh and pose data in the extended reality device to produce visual output (box 384). The method then continues by displaying the visual output on the extended reality device's display (box 386). Method 370 can be repeated during operation of the XR device.

[0061] Figure 4A This is a diagram illustrating the operation of an application using XR devices and LLM. More specifically, Figure 4A This illustration depicts an instruction generation pipeline for an example cognitive assistant application utilizing various methods and apparatus embodiments disclosed herein. The illustration includes example input prompts and example output responses from the system (including an LLM) at each step. For this example, a specific task includes generating a summary based on a first input prompt on the left and generating a task step based on a second input prompt on the right.

[0062] In box 402, several first-person view image frames (still images or frames from video) of the capture task are captured. The image frames can be captured using a camera from an XR device (e.g., a head-mounted viewer or glasses with video capabilities). In some embodiments, a depth-sensing camera can be used to capture depth information of various objects appearing in the frame, and the image frame can therefore be labeled accordingly.

[0063] The captured images are used to perform a task summation in box 404. In this non-limiting example, the task summation involves utilizing 10-second blocks of video / image frames and applying these blocks to at least one MLLM. The example here utilizes at least one of MLMMGPT-4V or Video-LLaVA, although other MLLM types are possible and anticipated. For each of the 10-second segments, a summary can be generated from the utilized MLLM(s).

[0064] Then, instruction generation occurs in box 406. This includes an LLM, such as GPT-4, which receives a list of task summaries and the expected tasks provided as context. These inputs are provided to the LLM, which responds with output containing instructions to perform the tasks.

[0065] Figure 4B This is a further illustration of an application using an XR device and an LLM operating according to process 410. The task of generating instructions for it in this non-limiting example is similar to that in... Figure 4A The task used for illustration in the provided example.

[0066] In box 412, the encoder (referred to here as the "reality encoder") receives input from the physical world via the XR device's sensors, which includes audio and video / image data. The encoder uses past data from time t-1 (for context) and current data from time t to generate instructions. In the generation of the current instruction, pre-generated instructions provided to the user of the XR device are also provided for context. Generated prompts are also generated based on the data provided to the encoder.

[0067] In box 414, LLM processing is performed using data provided from encoder step 412. This data may include selected video / image data (e.g., selected video frames captured by an XR device) and speech-to-text derived from audio data. Two different LLMs receive the generated prompts and perform processing based on those prompts according to their respective functions. In this non-limiting example, the GPT-4V MLLM generates an answer to a true / false query present in the generated prompts, while the Ferret MLLM generates an answer based on information from the video / image data, indicating the next part of the task to be performed. The results of the MLLMs' execution are then combined into a unified response.

[0068] In box 416, the decoder on the XR device (referred to here as the "reality decoder") receives a combined response generated from the execution of the MLLM based on previously generated prompts. Using the combined output from the MLLM, the processor on the XR device can retrieve video / image data with the relevant camera pose, convert the coordinates within the image to world coordinates, and generate an AR overlay to be provided to the HUD of the XR device. Text with instructions is also generated for display within the AR overlay. The AR overlay and the corresponding text are then displayed on the HUD of the XR device to instruct the user on the next step of the task.

[0069] Figure 4C This is a diagram illustrating an example of a network-based dashboard used for manual evaluation of the performance of the system disclosed herein, according to dashboard 440. Specifically, Figure 4C An example dashboard is shown to allow humans to evaluate the operation of various LLMs as they perform the intended operation of the system described in this paper, so that any further training can be performed if desired.

[0070] In the example shown, frames 442-456 present image data from various times in a sequence starting at time t-4 up to the current time t. Images 454 and 456 also correspond to time t and can be used to train an LLM to generate AR overlays at specific locations within the images for a specific task. In image 454, and based on this instruction, the dashboard user indicates the location where the AR overlay will be presented via a sketch on the image. In image 456, the corresponding AR overlay based on the sketch is shown.

[0071] Dashboard 440 also displays commands and responses with a true / false selection, where an LLM trainer can indicate whether a specific command has been executed as expected. The trainer can then press the send button to store the response. This can also be executed for other commands, allowing the trainer to train the LLM and thus configure the various systems discussed herein to assist users of XR devices in performing specific tasks under the guidance of the LLM.

[0072] Figure 5A diagram depicting the interaction between a computer-controlled machine 500 and a control system 502 is provided. The computer-controlled machine 500 includes actuators 504 and sensors 506. Actuators 504 may include one or more actuators, and sensors 506 may include one or more sensors. Sensors 506 are configured to sense the condition of the computer-controlled machine 500 or processes performed therethere. Sensors 506 may be configured to encode the sensed condition into a sensor signal 508 and transmit the sensor signal 508 to the control system 502. Non-limiting examples of sensors 506 include sensors used with XR devices (e.g., XR head-mounted displays), including audio sensors, video sensors, depth-sensing cameras, etc. Other types of sensors are also possible and contemplated, such as ultrasonic sensors, motion sensors, thermal imaging sensors, etc. Embodiments combining different sensors are possible and contemplated.

[0073] The control system 502 is configured to receive sensor signals 508 from a computer-controlled machine 500. As described below, the control system 502 may be further configured to calculate actuator control commands 510 based on the sensor signals and transmit the actuator control commands 510 to the actuators 504 of the computer-controlled machine 500. The control system 502 may include a PLC as discussed elsewhere herein, while the computer-controlled machine 500 may be an industrial device configured to perform automated industrial processes.

[0074] like Figure 5 As shown, the control system 502 also includes a processor 520 and a memory 522. The processor 520 may include one or more processors, and at least one of these processors may include a PLC. The memory 522 may include one or more memory devices. A classifier 514 of one or more embodiments may be implemented by the control system 502, which includes a non-volatile storage device 516, a processor 520, and a memory 522. As an alternative to or supplement to the classifier 514, one or more LLMs (including MLLMs) may be included.

[0075] Non-volatile storage device 516 may include one or more persistent data storage devices, such as hard disk drives, optical drives, tape drives, non-volatile solid-state devices, cloud storage devices, or any other device capable of persistently storing information. Processor 520 may include one or more devices selected from high-performance computing (HPC) systems, including high-performance cores, microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, or any other device that manipulates signals (analog or digital) based on computer-executable instructions residing in memory 522. Memory 522 may include a single memory device or multiple memory devices, including but not limited to random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information.

[0076] Processor 520 may be configured to read into memory 522 and execute computer-executable instructions residing in non-volatile storage device 516 and embodying one or more ML algorithms and / or methods of one or more embodiments. Non-volatile storage device 516 may include one or more operating systems and applications. Non-volatile storage device 516 may store computer programs compiled and / or interpreted using various programming languages ​​and / or techniques, including but not limited to Java, C, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL, individually or in combination.

[0077] When executed by processor 520, computer-executable instructions of non-volatile storage device 516 can cause control system 502 to implement one or more of the ML algorithms and / or methods disclosed herein. Non-volatile storage device 516 may also include ML data (including data parameters) supporting the functionality, features, and processes of one or more embodiments described herein.

[0078] Program code embodying the algorithms and / or methods described herein can be distributed individually or collectively as a program product in a variety of different forms. The program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to perform aspects of one or more embodiments. Computer-readable storage media, inherently non-transitory, can include volatile and non-volatile, removable and non-removable tangible media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media may further include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technologies, portable compact disc read-only memory (CD-ROM) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be read by a computer. Computer-readable program instructions can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device, or another device via a network, or downloaded to an external computer or external storage device.

[0079] Computer-readable program instructions stored in a computer-readable medium can be used to direct a computer, other type of programmable data processing apparatus, or other device to operate in a particular manner, causing the instructions stored in the computer-readable medium to produce an article of art, including instructions that implement the functions, actions, and / or operations specified in a flowchart or diagram. In some alternative embodiments, the functions, actions, and / or operations specified in the flowcharts and diagrams may be reordered, processed sequentially, and / or processed in parallel, in accordance with one or more embodiments. Furthermore, any of the flowcharts and / or diagrams may include more or fewer nodes or blocks compared to those illustrated in accordance with one or more embodiments.

[0080] While exemplary embodiments have been described above, they are not intended to describe all possible forms covered by the claims. The terms used in this specification are descriptive and not limiting, and it should be understood that various changes may be made without departing from the spirit and scope of this disclosure. As previously described, features of various embodiments may be combined to form further embodiments of the invention, which may not be explicitly described or illustrated. While various embodiments may have been described as offering advantages or superiority over other embodiments or prior art implementations with respect to one or more desired characteristics, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system properties depending on the particular application and implementation. These properties may include, but are not limited to, cost, strength, durability, lifecycle cost, merchantability, appearance, packaging, size, suitability, weight, manufacturability, ease of assembly, etc. Accordingly, to the extent that any embodiment is described with respect to one or more characteristics as less desirable than other embodiments or prior art implementations, such embodiments are not outside the scope of this disclosure and may be desirable for a particular application.

Claims

1. A method for operating an extended reality device, the method comprising: In an environment and using the extended reality device, sensor data from multiple sensors of the extended reality device is captured, wherein the sensor data includes image data, audio data, and depth data; The sensor data is used to form a world grid corresponding to a three-dimensional geometric representation of the environment; Based on the image data, pose data indicating the corresponding positions and orientations of multiple objects present in the world grid is generated; Convert the speech in the audio data into text data; Create a prompt using text data and selected image data; Multiple large language models (LLMs) are executed in parallel and using the aforementioned prompts to generate text output; The text output is combined with a 3D projection based on the world grid and the pose data in the extended reality device to produce a visual output; and The visual output is displayed on the screen of the extended reality device.

2. The method of claim 1, wherein the plurality of LLMs includes at least one multimodal LLM.

3. The method of claim 1, wherein the execution of multiple LLMs is performed on a server remotely relative to the extended reality device.

4. The method of claim 1, further comprising using the extended reality device to form the world mesh and generate the pose data.

5. The method of claim 1, wherein the selected image data comprises selected frames from a plurality of frames of video data captured by the camera of the extended reality device.

6. The method of claim 1, wherein the visual output comprises an augmented reality overlay displayed to a user of the extended reality device.

7. The method of claim 6, wherein the visual output further includes text information associated with the augmented reality overlay.

8. The method of claim 1, further comprising using at least one of a plurality of LLMs of audio data and selected image data.

9. The method of claim 1, further comprising performing the conversion of audio speech to text data and selecting image data on a server remote from the extended reality device.

10. The method of claim 1, further comprising creating a prompt based on an advanced task description.

11. A system for performing extended reality functions, the system comprising: Extended reality device, wherein the extended reality device includes: Multiple sensors configured to capture sensor data include a camera configured to capture image data, a microphone configured to capture audio data, and a depth sensor configured to capture depth data corresponding to objects existing in the environment in which the extended reality device is operating; Processor, the processor being configured to: A world grid is formed using sensor data, corresponding to a three-dimensional geometric representation of the environment. Pose data, indicating the position and orientation of multiple objects existing in the world grid, is generated based on image data. Visual output is generated by combining text output from multiple Large Language Models (LLMs) and a 3D projection based on world grid and pose data. This involves performing multiple LLMs based on speech-to-text data derived from audio data, selected portions of image data, and cues generated using the selected portions of both text and image data. The visual output is displayed on the screen of the extended reality device.

12. The system of claim 11, further comprising a server remotely located relative to the extended reality device, wherein the server is configured to: Generate prompts based on selected portions of text and image data; Perform multiple LLMs using selected portions of text data, audio data, and image data; and Provides text output from the execution of multiple LLMs to an extended reality device.

13. The system of claim 12, wherein the server is further configured to: Convert speech data from audio data into text data; and Select a specific frame from multiple frames of image data to produce a selected portion of the image data.

14. The system of claim 12, wherein the server is further configured to generate the prompt based on an advanced task description.

15. The system of claim 11, wherein the visual output includes an augmented reality overlay that can be viewed on the display of the extended reality device.

16. The system of claim 15, wherein the visual output further includes text associated with the augmented reality overlay.

17. The system of claim 11, wherein the extended reality device is further configured to store previous instances of pose data and associated image identifiers.

18. A method for generating visual content on a display of an extended reality device, the method comprising: The extended reality device uses multiple sensors to capture sensor data, which includes audio data, image data, and depth data corresponding to objects in the environment in which the extended reality device is operating; A world grid is formed using sensor data to create a three-dimensional geometric representation of the environment; Generate pose data based on image data, indicating the position and orientation of multiple objects present in the world grid; Visual output is generated by combining text output from multiple large language models (LLMs) and a 3D projection based on world grid and pose data, wherein multiple LLMs are performed based on text data converted from speech data from audio data, selected portions of image data, and cues generated using selected portions of text data and image data. and Display visual output on the display of an augmented reality device.

19. The method of claim 18, further comprising: Image and audio data are transmitted to a server that is remotely located relative to the augmented reality device; and Execute multiple LLMs on the server.

20. The method of claim 19, further comprising: Convert speech data from audio data into text data on the server; and Select a specific frame from multiple frames of image data on the server to produce a selected portion of the image data.

Images