Augmented display generator, experience station and generative augmented display station

The enhanced display system addresses limitations in virtual display systems by integrating input streams and user interactions with AI and machine learning, enabling immersive and interactive experiences.

JP7583900B2Active Publication Date: 2024-11-14BRELYON INC

Patent Information

Application Number
JP2023197827
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2023-03-30
Filing Date
2023-11-22
Publication Date
2024-11-14
Estimated Expiration
2043-11-22

AI Technical Summary

Technical Problem

Existing virtual display systems lack the ability to effectively integrate multiple input streams, user interactions, and generative content, limiting their capability to provide immersive and interactive experiences.

Method used

An enhanced display system that incorporates input stream modules, functional modules for processing multiple streams, and user interfaces to generate dynamic and interactive content, including augmented and virtual display systems with multiple depths and layers, utilizing AI and machine learning for predictive and generative capabilities.

Benefits of technology

Enables immersive and interactive experiences by integrating multiple input streams, allowing for dynamic content generation and user interactions, enhancing productivity and entertainment through advanced display technologies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007583900000001
    Figure 0007583900000001
  • Figure 0007583900000002
    Figure 0007583900000002
  • Figure 0007583900000003
    Figure 0007583900000003
Patent Text Reader

Abstract

To provide a system and method for improving use of an extended display system.SOLUTION: Some embodiments relate to an extended display generator including an input stream module for generating or receiving an input stream. The input stream may be generated locally (e.g., by a game engine) or remotely (e.g., from the Internet). A function module of the generator provides a function for changing or extracting information from the input stream. Next, the extended display generator applies a template for defining a method for presenting such a display content to a user to outputs of the input stream and the function. A graphical user interface is used to designate which input stream, function and visual template should be used. An extended display displays outputs of the selected input stream and function with a format defined by the visual template.SELECTED DRAWING: Figure 4A
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] [Field of the Invention] The present invention relates to various display system software layer experiences that include predictive and generative elements, and more particularly to new user and multi-user software experiences involving an augmented display system that includes a tandem computer and a generative content engine.

[0002] [Background of the invention] In today's society, in addition to multitasking applications for increased productivity, there is an increase in developments in machine learning based content and applications, or generative artificial intelligence (AI) content and applications, for either predictive modeling or content creation.

[0003] Virtual display systems are designed and implemented using a variety of specifications. For example, in US Patent 11,067,825 B2 and US Patent 11,768,825 B1, Dehkordi describes a virtual display system that provides monocular or binocular depth cues to achieve realistic depth perception effects. In US Patent 11,592,684 B2, Dehkordi disclosed an optical component called a field evolving cavity that folds light back and forth during multiple round trips through the cavity, causing the light source to appear farther away from the viewer compared to the distance to the physical display system. In US Patent 11,196,976 B2, Dehkordi disclosed a virtual display system that is directed to tessellating a light field to a size or depth that extends beyond the size of the pupil of the display system. Finally, in U.S. Patent No. 11,320,668 B2, Dehkordi et al. disclose a method for modifying the optical qualities or characteristics of a display system using optical fusion, which combines computational methods with optical artifacts to remove visual artifacts from images produced by the display system.

[0004] [Summary of the invention] Some aspects relate to an augmented display generator including an input stream module for generating or receiving a plurality of input streams; a functional module for performing a plurality of functions, where each function of the plurality of functions is configured to operate on at least one input stream of the plurality of input streams; a plurality of visual templates, where each template of the plurality of visual templates is configured to format output from the plurality of functions; and a graphical user interface configured to generate a graphical representation of the plurality of functions, the plurality of input streams, and the plurality of visual templates, where user selection of at least one of the plurality of functions, the input streams, and the visual templates via the graphical user interface causes the generator to generate display content, and the display content includes an output corresponding to a presented selected function from the plurality of functions using a visual template from the plurality of visual templates.

[0005] In some embodiments, the augmented display generator further comprises an augmented display system for receiving and displaying the display content.

[0006] In some embodiments of the extended display generator, the multiple input streams include a first input stream and a second input stream, the extended display system has at least one main section and at least one extended section, the extended display system displays first display content based on the first input stream in the main section and displays second display content in the extended section, the second display content being output from one of a plurality of functions configured to operate on the first display content and the second input stream.

[0007] In some embodiments of the extended display generator, the extended display system is a virtual display system, the main section is configured to be displayed at a first depth on the virtual display system, and the at least one extended section is one of a plurality of extended sections configured to be displayed at a plurality of different depths on the virtual display system.

[0008] In some embodiments, the augmented display generator further comprises an input device from the set consisting of an eye tracker, a head tracker, and a localization device.

[0009] In some embodiments of the augmented display generator, multiple augmented parts of the augmented display system communicate wirelessly.

[0010] In some embodiments of the augmented display generator, the augmented display system is a virtual display system, where one template from a plurality of visual templates formats a respective one of a plurality of virtual images at each depth, each depth including at least two or more unique depths.

[0011] In some embodiments of the augmented display generator, the augmented display system is configured to display multiple objects within the display content in multiple focal planes, and the depth of one of the multiple focal planes is calculated by an algorithm that optimizes the user's depth perception.

[0012] In some embodiments of the augmented display generator, one of the multiple functions is configured to operate on a first input stream of the multiple input streams and output an annotation based on detection of multiple features of the first input stream, and the augmented display system is a virtual display system configured to display the annotation in a first focal plane and display content of the first input stream in a second focal plane different from the first focal plane.

[0013] In some embodiments of the augmented display generator, the input stream module includes a sensor configured to generate a first input stream of a plurality of input streams based on information about a user or the environment captured by the sensor, and one of the plurality of functions operates on the first input stream to modify the display content generated by the augmented display system.

[0014] In some embodiments of the augmented display generator, one of the multiple input streams is a video game, and one of the multiple functions is configured to extract a depth map from the video game and display the depth map on the augmented display system.

[0015] In some embodiments of the augmented display generator, the augmented display system is integrated into a phone, a watch, a tablet, a headset, a viewer, a viewfinder, a television, or a vehicle instrument cluster.

[0016] In some embodiments of the augmented display generator, a visual template from a plurality of templates is configured to augment the field of view using a plurality of virtual images tiled along a direction substantially perpendicular to the user's line of sight.

[0017] In some embodiments of the augmented display generator, a template from the plurality of visual templates is user-defined, hi some embodiments, a function from the plurality of functions is a user-defined function.

[0018] In some embodiments of the augmented display generator, the input stream module generates or receives at least one of a plurality of input streams using or from a set including the internet, a game, a game engine, an existing application, a website, a simulation, a training video, a camera video, a camera image, a user input, a sensor input, and a data set.

[0019] In some embodiments of the augmented display generator, one of the multiple input streams is a video game, and one of the multiple functions is configured to geometrically transform shapes within the display content of the video game.

[0020] In some embodiments of the augmented display generator, at least one function from the plurality of functions is an AI function configured to generate at least a portion of the display content based on user input.

[0021] Another aspect relates to an experience station including a virtual display system having a main portion and an extension portion, and a module performing a generation function, the generation function being configured to: (i) receive a plurality of input streams, at least one of the input streams providing display content to the main portion of the virtual display system; (ii) capture a series of actions by a user; and (iii) cause the virtual display system to display a plurality of possible future actions by the user on the plurality of input streams in the extension portion.

[0022] In some embodiments of the experience station, the generation functionality is configured to use a time factor to determine a number of possible future actions for the augmented portion of the virtual display system.

[0023] In some embodiments of the experience station, the generation functionality includes an event-based action trigger that, when triggered, causes generation of content in the augmented portion of the virtual display system. In some embodiments of the experience station, the event-based action trigger is configured to be triggered by user input.

[0024] In some embodiments of the experience station, the generation functionality is further configured to automatically suggest and execute predicted tasks in parallel with the user's actions based on the authority level granted by the user.

[0025] In some embodiments of the experience station, the multiple input streams include at least one of the following: internet, games, existing applications, websites, videos, images, and data visualizers.

[0026] In some embodiments of the experience station, the generation functionality is configured to determine a plurality of possible future actions based at least in part on a probabilistic analysis of a plurality of past actions of the user.

[0027] In some embodiments of the experience station, the generation functionality is configured to cause the virtual display system to output a plurality of possible future actions as an annotation layer.

[0028] In some embodiments of the experience station, the generation functionality is configured to cause the virtual display system to display a plurality of possible future actions at a plurality of respective virtual depths.

[0029] In some embodiments, the experience station further comprises a sensor for detecting a user's gaze and outputting a position of the gaze as one of the multiple input streams, and the generation functionality is configured to dynamically change the display of the multiple possible future actions based on the position of the gaze.

[0030] Another aspect relates to a generative augmented display station comprising an augmented display system including a main portion and an augmented portion and configured to display first display content from an input stream of a plurality of input streams, and a computation module configured to operate on at least one of the plurality of input streams and output second display content to the augmented portion of the augmented display system.

[0031] In some embodiments of the generative augmented display station, at least a portion of the augmented display system is configured to display virtual images to generate a visual environment.

[0032] In some embodiments of the generative augmented display station, the augmented display system is a virtual display system, such that the multi-layer display displays images from multiple input streams in a first layer and a computation module generates the annotation layer.

[0033] In some embodiments, the generative augmentation display station includes a sensor for capturing information from a user, which information is one of a plurality of input streams input to a computation module, which is further configured to modify the first or second display content based on the information.

[0034] In some embodiments, the generative augmentation display station includes an AI module configured to receive as input a user query, such that the second display content is recommendations of a plurality of vertical search engines, the recommendations of a first vertical search engine being determined by the query, and the recommendations of the second vertical search engine being at least partially determined by results of the recommendations of the first vertical search engine.

[0035] In some embodiments of the generative augmentation display station, the second display content is influenced by settings of the user or user history.

[0036] In some embodiments of the generative augmented display station, the computation module is an AI module comprising a neural network or a transformer, which may be configured to compare at least two of the multiple input streams and merge the input streams into a single display content on the augmented display system.

[0037] Another aspect relates to an extended display experience station comprising an extended display system including a main portion and an extended portion for displaying content, where the main portion is configured to be fed directly by one of a plurality of input streams, the input stream including a local source, at least one processor operatively connected to the extended display system, and a non-transitory computer-readable storage medium encoding instructions executable by the at least one processor, the instructions including (i) a plurality of functions configured to receive as an input one of the plurality of input streams, and (ii) a software application configured to generate second display content for display on the extended portion of the extended display system using a function selected from the plurality of functions such that a first display content on the main portion of the extended display system interacts with a second display content on the extended portion.

[0038] In some embodiments, the augmented display experience station further comprises at least one sensor configured to capture further information about the user and input this information into a selected function, which outputs a dynamic change of at least one of the first and second display content on the augmented display system.

[0039] In some embodiments of the augmented viewing experience station, the augmented portion of the augmented viewing system is displayed at a virtual depth that is different from the virtual depth of the main portion.

[0040] In some embodiments of the augmented display experience station, at least a portion of the at least one processor is part of a remote source in communication with the augmented display system, the remote source being configured to execute a software application.

[0041] In some embodiments of the augmented display experience station, at least a portion of the at least one processor is part of the augmented display system and is configured to execute at least in part on the augmented display system.

[0042] In some embodiments of the augmented display experience station, the software application is further configured to communicate with a remote source and receive assistance from the remote source including second display content to be displayed on the extended portion of the augmented display system.

[0043] In some embodiments of the augmented display experience station, an input device is configured to capture information from a user and a software application is configured for the training experience such that multiple functions provide multiple interactive auxiliary image portions.

[0044] In some embodiments of the augmented viewing experience station, the software application is configured to update a second user's training experience based on the first user's usage.

[0045] In some embodiments of the augmented viewing experience station, the software application is a video application and characteristics of the video are influenced by the AI ​​module.

[0046] In some embodiments, the augmented viewing experience station further comprises a user input device, wherein the video application is a clickable video, and based on input to the user input device, the AI ​​function generates one of: (i) information about objects in the clickable video, (ii) a new frame of the clickable video, and (iii) additional content related to the clickable video.

[0047] In some embodiments of the augmented display experience station, the video application is a real-time video or a remote environment, and the AI ​​module is configured to detect people or actions of people in the real-time video and display a sequence of actions simultaneously with the display of the real-time video.

[0048] In some embodiments of the augmented display experience station, one of the functions is a computer vision function configured to detect features of one of the multiple input streams and output visually altered features on the augmented display system.

[0049] In some embodiments of the augmented display experience station, the computer vision functionality is configured to detect content of a first image displayed in a main portion of the augmented display system and generate and display second content on the extended portion of the augmented display system, the second content being identified by a reference in the first content.

[0050] In some embodiments, the augmented display experience station is configured for use in collaborative applications in which multiple users interact with multiple augmented display systems, each of which displays a portion of a visually shared environment.

[0051] In some embodiments of the augmented display experience station, the augmented display system is communicatively coupled to a portion of the augmented display experience station on a mobile device, a tablet, a smartphone, a smartwatch, or a headset.

[0052] Another aspect relates to another augmented viewing experience station including an augmented viewing system, where a module for performing a geometric transformation function is configured to receive an image and output a modified shape of at least a portion of the image.

[0053] In some embodiments of the augmented viewing experience station, the image is an image of a plurality of images in a video, and a geometric transformation function is applied to the video.

[0054] In some embodiments of the augmented viewing experience station, the video is a video game and the geometric transformation function operates on the video game to provide at least one of pose distortion, stride distortion, perspective distortion, orientation distortion, and motion distortion.

[0055] In some embodiments, the augmented display experience station further comprises a camera and a videoconferencing application for capturing images, the images being at least images of the user and the user's environment, and the module generates a second image including the altered shape using a geometric transformation function, and the image and the second image are combined within the shared visual environment.

[0056] In some embodiments, the augmented viewing experience station further comprises a camera configured to capture a user's gesture, and a modified shape of at least a portion of the image output by the geometric transformation function is so modified based on the gesture.

[0057] In some embodiments, the augmented viewing experience station further comprises a camera configured to capture an image, the image being an image of the user's environment, and the geometric transformation function configured to act on a portion of the image such that the portion of the image is displayed within the virtual reality environment.

[0058] Another aspect relates to a generative display experience station including an augmented display system configured to output a multi-layered display, with a plurality of computational modules each configured to receive as input content from each of a plurality of focal planes of the multi-layered display and output generative content, the generative content modifying content within the multi-layered display.

[0059] In some embodiments of the generative display experience station, at least one of the computing modules is an AI module. In some embodiments of the generative display experience station, the content includes a text source, and the AI ​​module outputs logical inferences derived from the text source.

[0060] In some embodiments of the augmented display experience station, a first depth of the multi-layer display is an annotation layer and a second depth displays a computer simulation or a training simulation, hi some embodiments, the second depth displays a training simulation and the annotation layer displays an image of an instructor.

[0061] In some embodiments, the generative display experience station further includes a sensor for detecting a user's gaze, and the user interface is configured to display a menu in one of multiple focal planes of the multi-layered display and to modify the menu based on the detected gaze.

[0062] In some embodiments of the generative display experience station, a subset of depth layers of the multi-layer display displays a plurality of images from at least one input stream, and another layer of the multi-layer display displays generative content, the generative content being a difference between the plurality of images, hi some embodiments, at least one input stream is video, and the difference is based on a time delay between frames of the video.

[0063] In some embodiments, the generative display experience station further comprises at least one sensor for capturing information from a user, the information being one of the input streams input to one of the computational modules, the computational module further configured to modify the first or second display content based on the information. In some embodiments, the sensor is a user input device.

[0064] In some embodiments of the generative display experience station, the multiple computational modules are multiple AI modules probabilistically coupled to each other such that the generative display content is a combination of the possible outputs of each of the multiple AI modules.

[0065] Another aspect relates to a tandem computing system comprising an input stream module for generating or receiving a plurality of input streams, the plurality of input streams including a first input stream, the input stream module configured to receive the first input stream from a remote source, a functional module implementing a plurality of functions configured to operate on the plurality of input streams, the plurality of functions including the first function, and an enhanced display system for displaying content output from the first function.

[0066] In some embodiments of the tandem computing system, the remote source is at least one of a crowd source, the Internet, a distributed network, a sensor, and a local area network. In some embodiments, the remote source is a distributed network, a first input stream from the distributed network is input to a first function, and display content output from the first function is a coherent image formed from information from the distributed network.

[0067] In some embodiments of the tandem computing system, a portion of the displayed content is generated by information solely from remote sources.

[0068] In some embodiments of the tandem computing system, the multiple input streams further include a second input stream, the input stream module is configured to receive the second input stream from a local source, and the first function outputs a portion of the display content generated by overlaying information from at least the first input stream and the second input stream.

[0069] In some embodiments of the tandem computing system, the display content is split into low bandwidth and high bandwidth components, and a remote source relays information regarding one of these components to the augmented display system for later modifying the display content.

[0070] In some embodiments, the tandem computing system further comprises at least one sensor configured to collect input from a user, wherein an output from at least one of the plurality of functions depends on said input.

[0071] In some embodiments of the tandem computing system, one of the functions is a time delay function configured to calculate and display a difference between two of the multiple input streams.

[0072] In some embodiments of the tandem computing system, the remote source is part of a shared visual environment configured for videoconferencing, the first function is an AI module configured to detect characteristics of a conversation during the videoconference, and the displayed content on the augmented display system includes information about these characteristics.

[0073] In some embodiments of the tandem computing system, the extended display system is a multi-layer display (multi-layer display) including a first tier and a second tier, the multiple input streams further include a second input stream, the input stream module is configured to receive the second input stream from a local source, the multiple functions further include a second function configured to receive the second input stream as input and output second display content based on the second input stream, the display content being a first display content based on a remote source and displayed on the first tier, and a second display content based on the local source and displayed on the second tier.

[0074] In some embodiments, the tandem computing system further comprises a camera array coupled to the remotely controlled vehicle for teleoperation applications. In some embodiments, the sensor array is configured to collect data from a user, said data including at least one of head tracking, SLAM, and line of sight from the user, and at least one of the plurality of functions receives said data as an input and outputs an angularly accurate dynamic perspective view of the visual environment.

[0075] In some embodiments of the tandem computing system, the multiple input streams include at least a portion of the multiple input streams each including information about a respective remote user involved in the collaborative application. In some embodiments, an augmented display system is configured to display the shared visual environment, and at least one function of the multiple functions is configured to dynamically transform content shared by a first user into a different form for a second user, the different form being determined by at least one of a second user profile and history.

[0076] In some embodiments of the tandem computing system, the augmented display system displays the shared visual environment in a multi-layer display configured for use by multiple users, a first layer generated by a remote source, the first layer being further configured to be commonly viewable by the multiple users and influenced by a respective second layer for each of the multiple users. In some embodiments, the input sensor is configured to detect a shape or a user, and each second layer for each of the users acts as a window into the first layer, the window being determined by the shape.

[0077] In some embodiments of the tandem computing system, user input of the first user is input to a function that outputs modified content of one of each of the second tiers.

[0078] Another aspect relates to a generative display system comprising an input stream module for generating or receiving a plurality of input streams, a functional module for implementing an AI function configured to receive as an input at least one input stream of the plurality of input streams and output generative visual information, the generative visual information being an annotation of a virtual image derived from at least one of the plurality of input streams, and an augmented display system configured to display the generative visual information.

[0079] In some embodiments, the generative display system further comprises a user input device configured to detect user input, and the AI ​​functionality is configured to continuously update the generative visual information based on at least one of the multiple input streams, and the AI ​​functionality is further configured to receive user input from the user input device and modify the generative visual information based on the user input.

[0080] In some embodiments of the generative display system, the AI ​​functionality receives as input an image of at least one focal plane and outputs annotations as an extended field of view display image.

[0081] In some embodiments of the generative display system, at least one of the multiple input streams comes from a remote source.

[0082] In some embodiments of the generative display system, the generative visual content is modified by a user profile or user history. [Brief description of the drawings]

[0083] [Figure 1] FIG. 1 illustrates a set of elements that are common features in example embodiments of the present disclosure. [Diagram 2] A diagram illustrating a set of software applications using an extended display system, a virtual display system, or a multi-layer display system, with or without remote computing sources. [Diagram 3] 1 is a flow chart of a software generation mechanism for a virtual or multi-layered display. [Figure 4A] FIG. 13 illustrates a detailed drop-down menu of software-generated mechanisms for an extended, virtual, or multi-layer display system. [Figure 4B] FIG. 13 illustrates a detailed drop-down menu of software-generated mechanisms for an extended, virtual, or multi-layer display system. [Figure 5-1] FIG. 1 illustrates a set of example functional blocks that may be used in a software generation mechanism to generate various software experiences. These blocks may be selected by a user or determined based on prompts or inputs in the system at a particular time. [Figure 5-2] FIG. 1 illustrates a set of example functional blocks that may be used in a software generation mechanism to generate various software experiences. These blocks may be selected by a user or determined based on prompts or inputs in the system at a particular time. [Figure 6-1]FIG. 5 shows a set of flowcharts or block diagrams corresponding to the example embodiments described in FIGS. 5-1 and 5-2. [Figure 6-2] FIG. 5 shows a set of flowcharts or block diagrams corresponding to the example embodiments described in FIGS. 5-1 and 5-2. [Figure 6-3] FIG. 5 shows a set of flowcharts or block diagrams corresponding to the example embodiments described in FIGS. 5-1 and 5-2. [Figure 7-1] FIG. 1 illustrates a series of embodiments of generated software experiences configured to display alternative actions that correlate with a current use and that may be performed using a display system. [Figure 7-2] FIG. 1 illustrates a series of embodiments of generated software experiences configured to display alternative actions that correlate with a current use and that may be performed using a display system. [Figure 8-1] FIG. 7 shows a set of flowcharts illustrating an exemplary mechanism for displaying alternative actions for some of the embodiments in FIGS. 7-1 and 7-2. [Figure 8-2] FIG. 7 shows a set of flowcharts illustrating an exemplary mechanism for displaying alternative actions for some of the embodiments in FIGS. 7-1 and 7-2. [Figure 9] FIG. 1 is a block diagram of a neural network that may be used to implement the predictive features of the software described in this disclosure. [Figure 10] FIG. 1 is a block diagram illustrating a self-attention mechanism for use in predicting alternative actions in an augmented, virtual, or multi-layered display system. [Figure 11-1] FIG. 1 illustrates a series of embodiments of software applications for an extended display system, a virtual display system, or a multi-layer display system. [Figure 11-2] FIG. 1 illustrates a series of embodiments of software applications for an extended display system, a virtual display system, or a multi-layer display system. [Figure 12] 1 is a flow chart illustrating a mechanism for simultaneously displaying content from remote and local sources. [Figure 13-1] FIG. 1 illustrates a series of embodiments of generative software applications for an augmented, virtual, or multi-layered display system where content is derived from local or remote sources. [Figure 13-2] FIG. 1 illustrates a series of embodiments of generative software applications for an augmented, virtual, or multi-layered display system where content is derived from local or remote sources. [Figure 13-3] FIG. 1 illustrates a series of embodiments of generative software applications for an augmented, virtual, or multi-layered display system where content is derived from local or remote sources. [Figure 14-1] FIG. 1 illustrates a series of embodiments of generative software applications for an augmented, virtual, or multi-tiered display system for multi-user applications. [Figure 14-2] FIG. 1 illustrates a series of embodiments of generative software applications for an augmented, virtual, or multi-tiered display system for multi-user applications. [Figure 15] 1 is a series of flowcharts of a portion of a generative software application described in this disclosure. [Figure 16-1] 1 is a series of flow charts illustrating the division of an augmented display system into remotely provided and locally provided subsections. [Figure 16-2] 1 is a series of flow charts illustrating the division of an augmented display system into remotely provided and locally provided subsections. [Figure 17]FIG. 1 illustrates an auxiliary embodiment showing how to graphically display information and events that follow a sequence and can branch off from a central event occurring in a generative software application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0084] Modern display devices provide new channels for bandwidth sharing, content creation, and user interaction. Immersive content and hardware such as augmented reality (AR), virtual reality (VR), extended reality (XR), mixed reality (MR), headsets, and autonomous virtual display systems are all approaches that provide untapped methods and software applications for improving human productivity and entertainment. Predictive and generative visual content can be combined with machine learning (ML), artificial intelligence (AI) algorithms, and other software architectures and algorithms to be displayed in new and unique ways to amplify or enhance the user experience. The inventors have recognized and understood that a user's visual experience can be enhanced by leveraging the power of computers running in concert to extend and expand the set of possibilities offered to the user's field of view (FoV). For example, software mechanisms to incorporate such content into various display systems, including but not limited to three-dimensional displays, virtual displays, and multi-layer displays, or even multi-monitor setups. In some embodiments, the displayed image is simply a 2D image that is extended onto the side panel and monitor. In some other embodiments, the display provides an image that includes monocular depth, and the viewer experiences accommodative depth cues to at least one image plane. In some embodiments, the displayed image is a stereoscopic image. In some embodiments, both stereoscopic and monocular depth cues are provided. Users of the disclosed technology may experience improved productivity, entertainment value, or generative suggestions for any application.

[0085] In this disclosure, new software methods and software applications are described. Some embodiments described herein disclose such methods and applications configured for use in an augmented display system, including methods for generating software applications, integration of predictive visual software, collaborative and single user applications, and software applications and displays involving multiple sources, including remote sources. New methods for generating visual bandwidth for user productivity, training, videoconferencing, telepresence, or entertainment are described.

[0086] term A "display system" or "display" is any device that generates an image. The physical source of the display image can be a standard 2D image or video, such as that generated by a display panel or panels, such as a liquid crystal display (LCD), a light emitting diode (LED) display, a micro LED display, an organic light emitting diode (OLED) display, a polymer light emitting diode (POLED) display, an active-matrix organic light emitting diode (AMOLED) display, a MOLED, etc. Such a display technology, or display technologies, may be incorporated into other display systems. In some embodiments, spatial light modulators (SLMs) are used. In some display systems, a light source may be combined with a mask or patterned elements to create a segmented addressable light source. Other light sources may be general light sources, such as one or more LEDs, a backlight, or a laser beam, configured for use in a projection-based display system.

[0087] Furthermore, the display system may be a headset, a handheld device, or a freestanding system, where the term "freestanding" means that the housing of the device may rest on a structure such as a table, hi some embodiments, the display system is configured to be attached to the structure by a mechanical arm.

[0088] In this disclosure, an "extended display" or "extended display system" is any display system that includes a portion of an image or visualization that is allocated to, extended to, or dedicated to the extended content that is not the primary content provided to the display. An extended display includes multi-monitor setups, hybrid setups of monitor projection systems, virtual display systems, AR, VR, and XR headsets including extended head-tracked displays, multi-projection systems, light field display systems, multi-focal display systems, volumetric display systems, tiled video walls, or any display system that is a connected part of the same environment. In some embodiments, an extended display system includes one portion on a monitor and another portion on a mobile phone, tablet, laptop screen, touch screen, advertising screen, or AR / VR / XR / MR device. An extended display system may be divided into any collection of displays on any screen device in any application. An extended display system may be considered a collection of displays or pixels on one or more devices such that there is a primary input set of pixels and an extended set of pixels. The extended set of pixels may be referred to as an "extended portion" or "extended part" of the display content. An augmented display system may be described as including a main portion, the content of which is generated by a primary computer system (a "local source"), and the augmented display system may include a secondary portion (i.e., an augmented portion) that may be generated by an auxiliary or indirect computer system or source (a "remote source").

[0089] A "virtual display system" is an augmented display system that generates images at more than one perceived depth, or at one perceived depth different from the depth of the display panel that generates the images. Such images may rely on monocular depth and may be stereoscopic, autostereoscopic, or (auto)multiscopic. A virtual display system may be a freestanding system such as a computer monitor or a television set. A virtual display system may be part of a mobile phone, tablet, headset, smartwatch, or any portable device. A virtual display system may be single-user or multi-user in any application. A virtual display system may be a volumetric display or a light field display. In some embodiments, the virtual display system is a holographic display, which relies on the wave properties of light to generate images based on manipulating the interference of light.

[0090] The term "display content" is used to describe the final image information or source information as perceived by a viewer. In some embodiments, the virtual display system generates an eyebox having a volume large enough to encompass both eyes of the viewer simultaneously. In other embodiments, the virtual display system generates left and right eyeboxes configured for simultaneous viewing by the left and right eyes, respectively. The size and number of eyeboxes will depend on the particular nature and design of the display.

[0091] Augmented display systems, including virtual display systems, may incorporate any hardware, including liquid crystals or other polarization dependent elements to affect the properties of the display, any type of mirrors or lenses to redirect the light path, affect the size in any dimension, change the depth of focus, or correct aberrations and distortions, any surface coatings, active elements, spectral or spatial filters that aid in image quality, optical cavities, or any type of elements or coatings that act as shielding or anti-reflective layers to reduce unwanted, stray, or ambient light reaching the viewer. In some embodiments, the display system comprises metamaterials and metasurfaces, nonlinear optical elements, photonic crystals, gradient index materials, anisotropic or bianisotropic elements, or electro-optical elements. In some embodiments, the augmented display system is an optical virtual display system. However, the augmented display system can be any manner of augmented display system, including radio frequency or acoustic display systems configured for consumption by the human auditory system. In some embodiments, the display or elements of the display may be curved.

[0092] In some embodiments, a field evolving cavity is included as a component in the augmented display system. A "field evolving cavity", "field evolving cavity (FE)", or "field evolving cavity (FEC)" is an optical cavity in which light can essentially fold. The FEC helps provide a user with depth cues for three-dimensional perception. In some embodiments, the depth cues are monocular depth cues. An example of an FEC includes a first semi-reflective element, an air gap or a dielectric material, and a second semi-reflective element. Light passes through the first semi-reflective element, passes through the gap, reflects off the second semi-reflective element back into the gap, reflects off the first semi-reflective element, travels forward again through the gap, and then transmits through the semi-reflective element to the viewer. As a result, the effective distance traveled by light in this case is three times the gap distance itself. The number of round trips is arbitrary. For example, there may be zero, one, two, or three round trips. In some embodiments, polarization-dependent and polarization-affecting elements, such as polarizers, wave plates, and polarizing beam splitters, may be used to improve efficiency or change the number of round trips. For example, if the light source is a pixel that is approximately a point source, the FEC causes the spherical wavefront of the pixel to be flatter than if the light passed through the gap once.

[0093] In FEC, the number of round trips determines the focal plane of the image and therefore the viewer's monocular depth cues. In some embodiments, different light rays travel different total distances to generate multiple focal planes or multi-focal images with multiple image depths. In some embodiments, the image depth is dynamic or adjustable, for example, via an electro-optical structure that changes the number of round trips.

[0094] A "light field" is a mathematical model of light propagation that relies on approximations of geometric light rays. Some light field models incorporate wave-based effects, such as diffraction. A light field display is a three-dimensional display designed to create a 3D effect for the user using light field modeling. A concentric light field display is a light field display in which for any two pixels of the display at a fixed radius from the viewer, the chief ray of the light cone of the first pixel intersects with the chief ray of the light cone of the second pixel. A concentric light field display produces an image that can be focused on by the eye at all points.

[0095] The display system may generate images, overlay annotations onto existing images, feed one set of display content back into another set of interactive environments, or adapt to the surrounding environment. Users may experience VR, AR, XR, video see-through effects, monitor remote systems to receive simultaneous predictive suggestions, provide avatars with the power to leave an imprint on digital content or online resources, or use AI for the creation of generative content. In some embodiments, subsections of the display content are input into an algorithm to affect another subsection.

[0096] A "subsection" of the display content is a division of the display content generated by the display system. In some embodiments, a subsection is a pixel or a set of pixels. The set of pixels may be disjoint or contiguous. In some embodiments, the subsection corresponds to a type of feature of the display content. For example, a subsection of an image of a person may be the head or arm, and another subsection may be the hand or eye. In some embodiments, a subsection may be an entire layer or a portion of a layer or focal plane of a display that generates multiple focal planes. In some embodiments, a subsection is a portion of the spectral content of an image or portion of an image, in any mathematical basis. The subsections may be divided in different ways at various times.

[0097] The displayed content may be manipulated by the user or by user interaction through various input devices. Input devices are typically a type of sensor that captures user input intentionally rather than automatically. Input devices such as cameras, keyboard and mouse input, touch screens, gesture sensors, head tracking, eye tracking, VR paddles, voice input, voice detection, etc., allow for user feedback in multiple ways. In some embodiments, various biological or health sensors capture information such as heart rate, posture, sitting or standing orientation, blood pressure, gaze or focus, etc., and use that information in algorithms to affect the displayed content.

[0098] In some embodiments, for example, gaze may be detected and the position of gaze may be tracked. Gaze detection may measure the focus of a person, i.e., where the person is looking, what the person is looking at, how the person blinks or winks, or how the person's pupils are responding to a stimulus, image, or other (e.g., change in pupil size). A sensor, such as an infrared sensor, shines infrared light into the eye and detects changes in reflectance based on eye movement. In some embodiments, a camera captures an image of the eye and a convolutional neural network (CNN) is used to estimate the gaze. After the gaze is detected or known by the display system, the display content may change based on the gaze. For example, the gaze may be such that the user is looking at a particular display content that corresponds to an action the user may perform, such as displaying a menu. In another example, the first layer may display a wide-field image of the user's position on a scene or map, and eye-tracking feedback may magnify a particular area or display annotations regarding the area that is the focus of the gaze. This example may be referred to as a telescoping function.

[0099] In some embodiments, user or environmental input may be generated via a variety of other sensors or AI mechanisms that may interface directly or indirectly with the display system. Sensors include cameras of any kind, pressure or tactile sensors, sensors that detect health information, biological information about a person or the environment, clock and other timing sensors, temperature sensors, audio sensors (including microphones of any kind), chemical sensors, or measurement sensors for scientific and engineering purposes.

[0100] The source of the display content may be local or remote. Sources include local workstations, laptops, computers, edge devices, distributed sensors, the Internet, cloud sources, servers or server farms, or any electronic device capable of communicating data. Sources may include microcontrollers, field programmable gate arrays (FPGAs), cloud computers or servers, edge devices, distributed networks, internet of things (IoTs). Sources may operate on data before sending it to the display system, and sources may receive data from the display system and operate on the data.

[0101] Remote sources include, but are not limited to, cloud servers, the Internet, distributed networks or distributed sensors, edge devices, systems connected via wireless networks, or IoT. Remote sources are not necessarily located far away and may include processing units (CPUs, GPUs, or neural processing units (NPUs)) running at stations other than the local source. A local source is hardwired into the user interface system and serves as the primary workstation for the primary display portion of the augmented display.

[0102] A "communication channel" refers to a link between at least two systems or users that allows for the transmission of information and data, for example, between a source and a display. A communication channel may be hardwired or wireless. Communication channels include Ethernet, USB, wireless networks, any short-range wireless technology (such as Bluetooth), fiber optic systems, digital subscriber lines (DSL), radio frequency (RF) channels such as coaxial cable, etc.

[0103] "Input Stream" refers to data or information from a data storage system or source, either local or remote, from which data may be retrieved. Data may be transmitted in real-time. Data may include metadata about the physical source itself or about other content. An input stream may be graphical data intended directly for display on a display system. In some embodiments, an input stream may refer to one or more input streams directed to a subsection of a display system. In some embodiments, an input stream is generated in one subsection of the display by a user's actions and displayed in another subsection.

[0104] Latency is the delay between the moment information starts transmitting along a communication channel and the moment it is received at the other end of the channel. There is usually a trade-off between latency and content bandwidth. For remote sources, data communication latency is a parameter that can be integrated into the design of the software application. Latency in remotely generated content can be built into the weights of ML and linear layers of various neural networks.

[0105] In some embodiments, various AI and ML algorithms may be incorporated into the visual prediction service: Existing learning algorithms, such as generative trained transformers, and bidirectional encoder representations from transformers, as described herein, may be incorporated into the augmented display system to generalize and direct a portion or the entire augmented display with respect to user actions.

[0106] Applications include, but are not limited to, graphical predictive and virtual assistants, quality control, remote operations, flight simulation and defense, medical and diagnostic imaging, e-sports and gaming, and financial trading. In these use cases, multi-dimensional data sets must be displayed in an intuitive way so that users can make informed decisions. In some embodiments, predictive analytics may be calculated. In some embodiments, virtual avatars or AI systems with user-granted authority act on these predictive analytics. Examples of AI-generated content include text-to-image, image-to-text, image or text-to-task, text-to-code, text-to-inference, image or text-to-recommendation, or any other combination. AI functions or modules may be aided by probabilistic analysis to combine different models or training data in content generation.

[0107] The term "user" or "audience" refers to a person who interacts with the system using senses that can be sight, hearing, touch, or smell. In some embodiments, the system is a display system or an augmented display system. The user may be a future user who uses the system at a different time to allow for asynchronous applications. A "user interface" or "UI" corresponds to a set of interactive tools (such as toggle buttons, radio buttons, scroll bars, or drop-down menus) and screens with which a user can interact. Similarly, a "user experience" or "UX" defines the cumulative experience of a user as determined by the UI.

[0108] A "3D image" is an image that triggers any depth cue in a viewer, such that the viewer perceives the displayed content at variable depth, or at various depths relative to one another, or different portions of the displayed content appearing at different depths than the physical display system. In some embodiments, a parallax effect is created. In some embodiments, the 3D effect is triggered stereoscopically by sending a different image to each eye. In some embodiments, the 3D effect is triggered using a "monocular depth" cue, where each eye focuses or adapts to the appropriate focal plane. A virtual image is an image displayed on a virtual display system. A virtual image may be multifocal, variable focus, light field image, holographic, stereoscopic, autostereoscopic, or (auto)multiscopic. The virtual depth of a virtual image may be dynamically adjustable by control of the display system, user or sensor input, or preprogrammed routines.

[0109] The depth at which the content is located is called the "virtual depth" of the "focal plane." A display that produces display content that can be viewed at different virtual depths is called a "multi-layer display system" or "multi-layer display." For example, a multi-layer display system is a display system in which display content is displayed in such a way that a viewer must accommodate his or her eyes to different depths in order to see the different display content. A multi-layer display, in some embodiments, comprises a transparent display. Display content at a particular virtual depth is called a "layer," "depth layer," or "virtual layer."

[0110] An "annotation layer" is display content that provides context, further information, or description of other content in a display system. For example, an annotation layer may be a layer or focal plane in a multi-layer display. An annotation layer provides graphics or text annotations related to content in other layers. Other forms of augmented displays may include annotations. Annotations may be displayed as floating graphics or in an extended FoV display, or may be overlaid on top of related display content in a single image.

[0111] In some embodiments, other characteristics of interest for the displayed content include, but are not limited to, resolution, refresh rate, brightness, FoV, viewable zone, monocular depth, or accommodation, vergence, eyebox, or headbox.

[0112] "Visual template" refers to a predetermined way to computationally structure and display data and information within a virtual display system. An example of a visual template is the set of layers generated by a multi-layer display.

[0113] Typically, a "visual environment" is a collection of display content or virtual images that may be able to interact with each other. Display content may have as sources camera images or computationally rendered images such as computer graphics. A visual environment may be a virtual reality environment, where all content within the virtual reality environment is virtual display content, a visual environment may be an augmented reality or mixed reality environment, where virtual images within the augmented reality or mixed reality environment are superimposed on the physical environment, or a visual environment may be traditional image content from a display panel such as an LCD panel. In some embodiments, a visual environment includes only one virtual image. A visual environment may be used by a single user in an athletic equipment, or a visual environment may be shared or displayed by multiple display systems that communicate with each other, for example, via the Internet or any type of wired or wireless network. A "shared visual environment" is a visual environment that may be used for any collaborative activity, including telework applications, videoconferencing, web conferencing, online classes, or collaborative or multiplayer games. In a visual environment or shared visual environment, different users may view the displayed content from different perspectives, and in some embodiments the shared visual environment is immersive, such that two users each using a display in separate locations but within the same shared visual environment perceive that they are physically adjacent to one another or that they are in a location other than the physical location of the display system, for example by navigating within the visual environment or by the presence of a collaborating user within the surrounding area of ​​a virtual panorama.

[0114] Augmented and virtual display systems are useful in a variety of applications, including video games, game engines, remote control, simulation training, videoconferencing, and computer simulators.

[0115] A video game is an electronic game that involves interaction with one or more players through a user interface, utilizing audio and visual feedback to create an immersive and interactive gaming experience. Video games may be designed for a variety of platforms, including consoles, personal computers, mobile devices, and virtual reality systems, and can encompass a variety of game genres, such as action, adventure, role-playing, simulation, sports, puzzle, and strategy games. Game mechanics and rules may vary from game to game, but games typically include objectives that players must accomplish within the game's environment. A game engine is a platform for generating video games.

[0116] Teleoperation is a method of controlling remote devices or systems, allowing a human operator to perform tasks on the remote devices or systems in real time. Teleoperation systems typically include sensors and actuators that allow the operator to perceive and manipulate the remote environment, as well as a user interface that provides the operator with feedback and control. Remote devices or systems may be in locations that are dangerous or difficult to reach, or may require special skills or expertise to operate, making teleoperation a useful tool in a variety of industries, including manufacturing, construction, surveying, and the use of remote-controlled vehicles. Teleoperation systems may incorporate artificial intelligence and machine learning algorithms to enhance the operator's capabilities and automate certain aspects of teleoperation.

[0117] Videoconferencing is a technology that allows remote participants to communicate and collaborate in real-time conferences over a communication channel such as the Internet. Videoconferencing systems typically include both hardware and software components, such as cameras, microphones, speakers, display screens, and user interfaces, that allow participants to connect to a conference and interact with each other. Systems may incorporate features such as screen sharing, file sharing, virtual whiteboards, and chat messaging to improve the collaborative experience. Videoconferencing generally facilitates remote meetings, presentations, workshops, and consultations, allowing participants to communicate and work together without the need for physical travel.

[0118] Simulation training is the art of recreating the experience of a task in a simulated environment, usually using computer software and specialized hardware. One example is flight simulation technology, which simulates the task of piloting an aircraft. Flight simulation systems usually include a visual display system that provides a realistic representation of the simulated environment, in addition to a cockpit simulator or control interface that mimics the controls and instruments of a real aircraft. The simulator may incorporate motion and sound effects to improve the immersive experience. Flight simulation can be used for a variety of purposes, such as pilot training, aircraft design and testing, and entertainment. Simulations can be based on real-world data and physical models to accurately reproduce the aircraft's behavior and environment, and may also incorporate scenarios and events to simulate various flight conditions and emergency situations. User inputs to flight simulation training applications include control sticks and throttles, physical panels, or touch screens.

[0119] A computer simulation is a digital model of a real-world system or process designed to mimic the behavior and interactions of the system or process under different conditions. A computer simulation typically uses mathematical algorithms, computer programs, and data inputs to create a visual environment within which the behavior of the system can be studied and analyzed. The simulated system may be a physical object or phenomenon, such as a weather system, a chemical reaction, an electromagnetic phenomenon, or a mechanical device, or it may be an abstract concept, such as a market or a social network. Computer simulations may be used for a variety of purposes, such as scientific research, engineering design and testing, and training and education. The accuracy and complexity of computer simulations can vary widely depending on the level of detail and fidelity required for a particular application. Computer simulations often allow a user to interact with the details of the simulated system by changing modeling or computational parameters.

[0120] In any embodiment, any sensor may be used to provide information about the user, the environment, or other external conditions and scenarios to the display system. In some embodiments, a camera is used, for example, to capture information about the user or the user's environment. Multiple cameras or camera arrays or camera systems may be used. In some embodiments, a depth camera captures information about depth or senses gestures and poses, and the depth camera can be any type of camera. In this disclosure, a "depth camera," "depth sensor," or "RBGD camera" is an imaging device that records the distance between the camera and the location of an object and the distance to the location of an object. A depth camera can be actively illuminated or passively illuminated and can include multiple cameras. Light detection and ranging (LIDAR) and time-of-flight cameras are examples of active depth cameras. A depth camera can also use optical coherence tomography detection (i.e., autocorrelation). A depth camera can use infrared (IR) illumination to extract depth from structures or shadows. The depth camera can incorporate gesture or facial recognition features. Depth can also be estimated from a conventional camera or multiple conventional cameras, for example by stereo imaging. A camera array or camera system can include any combination of these cameras.

[0121] A "gesture" is a user movement, facial expression, or postural orientation that is typically interpreted by a human or computer to indicate a particular desired change, emotion, or physical state. Gestures are typically on a time scale observable by humans. Microgestures are movements, expressions, or orientations that occur instantaneously. Microgestures are typically involuntary and exhibit similar characteristics to gestures. Microgestures may include brief shifts of gaze, finger taps, or other involuntary actions. Gestures may be captured by a camera and identified or classified by deep learning algorithms or convolutional neural networks.

[0122] In general, the "shape" of a person, user, object, display image, or other virtual or physical object is a term that includes both the location and orientation of that item. In some embodiments, the shape of an object may correspond to its contour, depending on how the object is distorted, stretched, bent, or generally deformed. For example, cameras and algorithms may be used together to identify the location of a physical object in space.

[0123] In this document, the terms "machine-readable medium", "computer-readable medium", and similar terms are used to refer to a non-transitory medium, volatile or non-volatile, that stores data and / or instructions that cause a machine to operate in a specific manner. Common forms of machine-readable medium include, for example, a hard disk, a solid state drive (SSD), a magnetic tape or any other magnetic data storage medium, an optical disk or any other optical data storage medium, any physical medium that contains a pattern of holes, a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a FLASH-EPROM, a non-volatile random access memory (NVRAM), any other memory chip or cartridge, and networked versions thereof.

[0124] These and various other forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processing device for execution. Such instructions embodied on the medium are generally referred to as "instructions" or "code." The instructions may be grouped in the form of a computer program or other grouping. Such instructions, when executed, may enable a processing device to perform the features or functions of the present application as described herein.

[0125] A "processing device" may be implemented as a single processor that performs processing operations, or a combination of special purpose and / or general purpose processors that perform processing operations. A processing device may include a central processing unit (CPU), a graphics processor unit (GPU), an accelerated processing unit (APU), a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), and / or other processing circuitry.

[0126] AI is any intelligent behavior caused by a machine. Intelligent behavior includes perception, detection, scene understanding, generation, or perception of information, or performing inference. The term "neural network", "artificial neural network", or "neural net" refers to a computational software architecture that is an exemplary implementation of AI and can learn patterns from multiple data sources and types and make predictions about previously unseen data. Types, algorithms, or architectures of neural networks include feedforward neural networks, recurrent neural networks (RNNs), residual neural networks, generative adversarial networks (GANs), modular neural networks, or convolutional neural networks (CNNs) (used for object detection and recognition). Neural networks can include a combination of different types of neural network architectures. Parameters of the neural network may be determined or trained using training data. Neural networks can be supervised or unsupervised. Learning can be completed by optimization of a cost function. In some embodiments, the neural network architecture is a radial basis network, a multi-layer perceptron architecture, a long-short term memory (LSTM), a Hopfield network, or a Boltzmann machine. The neural network architecture can be one-to-one, one-to-many, many-to-one, or many-to-many. In the present disclosure, any of the AI ​​algorithms may be used in the AI-based embodiments. For example, a GAN may use stochastic gradient descent optimization to minimize a loss function. A LSTM or RNN may use a gradient descent algorithm with backpropagation.

[0127] A "Transformer" is a machine learning model in deep learning that relies on self-attention to weight input data in a diverse manner. Transformers are often used in computer vision and natural language processing (NLP). Transformers differ from RNNs in that the input data is processed simultaneously rather than serially. Generative trained Transformers and bidirectional encoder representations from Transformers are examples of Transformer systems. Applications include video or image understanding, document summarization or generation, language translation, etc.

[0128] Learning algorithms may be supervised or unsupervised. Some supervised learning algorithms used to implement the embodiments disclosed herein include decision trees or random forests, support vector machines, Bayesian algorithms, and logistic or linear regression. Unsupervised learning acquires information by understanding patterns and trends in untagged data. Some algorithms include clustering, K-means clustering, and Gaussian mixture models. In some embodiments, non-neural network computational methods are used to generate the display content. In some embodiments, neural networks are combined with other computational methods or algorithms. Other computational methods include optimization algorithms, brute force algorithms, random algorithms, and recursive algorithms. The algorithms can implement any mathematical operation or physical phenomenon.

[0129] An "avatar" is a computer program or program interface that can include a character or representation of a user within a digital or visual environment. An avatar may be a likeness of a person, but may acquire a default morphology. In some embodiments, an avatar has no likeness at all, or communicates with the user using text or voice modes. In some embodiments, an avatar serves as a user interface to make suggestions to the user, make predictions, or assist in the performance of a task. In some embodiments, an avatar has the authority to perform tasks without direct influence from the user. An avatar may be AI-based. An avatar may use neural networks or other deep learning mechanisms.

[0130] "Tandem Computing" is a method for a display system to display display content from multiple sources, at least one of which is a remote source that displays content on an extended portion of the extended display system. The display content may be any of a variety of display contents that may interact with each other.

[0131] In the context of two display contents interacting with each other, "interact" means that the display content of one part of a display system is input to a function having an output that dynamically affects the display content of a second part, and vice versa, i.e., the display content of the second part is input to a function (which may be the same function) having an output that dynamically affects the display content of the first part.

[0132] "Rendering parallelism" refers to the ability to divide rendering tasks so that they can be distributed among different local and non-local computational resources. Graphics may be rendered in a variety of ways, including from computer graphics techniques and radiance equations, leveraging content from volumetric video, neural rendering, or neural radiance fields.

[0133] "Graphical user interface" or "GUI" refers to any interface displayed on a display system that allows a user to interact and interact with the system in a graphical, visual manner. A GUI may include various ways for a user to input information, such as radio buttons, toggle switches, drop-down menus, or scroll bars. A GUI allows a user to interact with or generate software or to interact with an electronic device.

[0134] A "feature" is a mapping that takes a portion of content and generates a different portion of content, or annotates or modifies the original content. A function may be an algorithm for performing a mapping or operation. A function may take multiple portions of content and output multiple portions of content. A function may be low level, for example a mathematical operation or an image processing function. A function may be mid level, for example, take an image and detect features, such as edges, in a scene. A function may be a computer vision assisted function. Or a function may improve a characteristic of the content. A function may be high level, for example, generate content or detect classes of objects or make predictions about possible future actions to be taken by a viewer observing the input content. In some embodiments, the function is predefined. In some embodiments, the function is user defined. A function may be performed by an AI, including neural networks, encoder / decoder systems, transformers, or a combination of these examples. A function may include various methods for optimizing, sorting, or ordering various data or images. A function may be deterministic or probabilistic. A function may receive multiple inputs and produce multiple outputs, which may be time dependent.

[0135] An example of a computational function is a simultaneous localization and mapping (SLAM) function that builds or updates a map of the environment and tracks a user or objects within the map. A SLAM algorithm may involve taking sensory data, such as a camera, as input and calculating the most likely location of an object based on the sensory data. This solution may include an expectation maximization algorithm. A particle filter or a Kalman filter may be used.

[0136] Another feature may be used to track objects or parts of the user's body, such as in head tracking use cases. Tracking may be performed using a constant velocity model.

[0137] The terms "graphics intelligence", "intelligent generative content", or "generative content" refer to a function that has an input that is at least one input stream and outputs content. The input stream may include content configured for a display system. An example of graphics intelligence is an AI module or function that receives as input a set of display images and outputs a second display image that includes various annotations to explain the input and to suggest to the user how to interact with the input. The output content may be visual data. The output content may be used as input for other functions. Graphics intelligence may receive as input sensory data of the user, the user's environment, or another environment, such as a manufacturing warehouse, an automobile's surroundings, or other industrial environment. A "generative function" is a function that receives as input one or more input streams and outputs new content. In some embodiments, the generative function is also influenced or parameterized by the user's input, profile, and history. A user profile includes information about the user (e.g., interests, goals, content desired to see, or demographics). A user history is the past usage by a user of a particular application or set of applications. A user history may be, for example, a search history, a list of email correspondence, a list of media the user viewed within a particular period of time, and so on.

[0138] A "collaborative software application" is a software application in which multiple users interact with one another through the software application. The interaction may be synchronous or asynchronous. Examples include video or web conferencing, online courses, multiplayer games, various applications in control center or remote operation situations, webinars, or other remote learning environments. Collaborative software applications may be used in a shared visual environment.

[0139] In this description, reference to "an embodiment," "one embodiment," or similar words or phrases means that the described feature, function, structure, or characteristic is an example of a technique or invention introduced herein. Occurrences of such phrases herein do not necessarily all refer to the same embodiment. Conversely, embodiments referenced herein are not necessarily mutually exclusive.

[0140] Various embodiments shown herein are described with reference to exemplary block diagrams, flow charts, and other diagrams. As will be apparent to those skilled in the art after reading this document, the illustrated embodiments and their various alternatives may be implemented without being limited to the illustrated examples. For example, the descriptions accompanying the block diagrams should not be construed as mandating a particular architecture or configuration.

[0141] All figures, drawings, and examples in this disclosure describe selected versions of the technology introduced herein and are not intended to limit the scope of the technology introduced herein. Furthermore, throughout this disclosure, "arbitrarily designed" refers to any shape, size, material, feature, type, orientation, position, quantity, components, and arrangement of components, including one or more components, that enables the invention or components of the invention to fulfill the purpose and intent of the invention or that particular component within the invention.

[0142] All such components and software may be optionally designed to deliver a desired profile. As used herein, "variation of any parameter" refers to the variation, modification, adjustment, programming, benchmarking, optimization, and / or control of a parameter, and may include one or more of the following variations: bandwidth, channel capacity, brightness, focal plane depth, parallax, authority level, sensor or camera sensitivity, frequency range, polarization, data rate, shape or orientation, sequence or timing arrangement, execution time, or other physical or computational characteristics.

[0143] Some capabilities described herein, such as functions, visual templates, graphical user interfaces, receiving input streams, and generating input streams, may be implemented in one or more modules. A module includes hardware and / or software for implementing a capability. For example, such capabilities may be implemented by a module including one or more processors executing computer code stored in one or more non-transitory computer-readable storage media. In some embodiments, a capability is implemented at least in part by a module including dedicated hardware (e.g., ASIC, FPGA). In some embodiments, modules may share components. For example, a first functional module and a second functional module may both utilize a common processor (e.g., by time sharing or multithreading) or include computer executable code stored in a common computer storage medium (e.g., in different memory locations).

[0144] In some cases, a module may be identified as a hardware module or a software module. A hardware module includes or shares hardware for implementing the capabilities of the module. A hardware module may include software, i.e., a hardware module may include a software module. A software module includes information that may be stored, for example, on a non-transitory computer-readable storage medium. In some embodiments, the information may include instructions executable by one or more processors. In some embodiments, the information may be used at least in part to configure hardware such as an FPGA. In some embodiments, information for implementing capabilities such as functions, visual templates, graphical user interfaces, receiving input streams, and generating input streams may be recorded as a software module. The capabilities may be implemented, for example, by reading the software module from a storage medium and executing the software module using one or more processors, or by reading the software module from a storage medium and configuring hardware using the information.

[0145] Each of the processes, methods, and algorithms described in the preceding sections may be embodied in code components executed by one or more computer systems or computer processors comprising computer hardware, and may be fully or partially automated. The processes and algorithms may be partially or entirely implemented in application specific circuits. The various features and processes described above may be used independently of each other, or may be combined in various ways. Various combinations and subcombinations are intended to be within the scope of the present disclosure, and in some implementations, certain method or process blocks may be omitted. Furthermore, unless otherwise indicated by the context, the methods and processes described herein are not limited to any sequence, and the blocks or states associated therewith may be performed in other suitable sequences, or may be performed in parallel, or in some other manner. Blocks or states may be added to or deleted from the disclosed example embodiments. The execution of a particular operation or process may reside not only in a single machine, but may be spread across multiple machines and distributed among computer systems or computer processors.

[0146] As used herein, the term "or" may be interpreted in either an inclusive or exclusive sense. Moreover, descriptions of resources, acts, or structures in the singular should not be read to exclude the plural. In particular, conditional language such as "can," "could," "might," or "may" is typically intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not, unless otherwise specifically stated in the context or otherwise understood in the context when used.

[0147] Terms and phrases used in this document, and variations thereof, unless expressly stated otherwise, should be construed as open-ended, as opposed to limiting. Adjectives such as "conventional," "traditional," "usual," "standard," "known," and terms of similar meaning should not be construed to limit the described items to items available for use during a particular period or at a particular time, but should instead be read to encompass conventional, traditional, ordinary, or standard technology that may be available or known at any time now or in the future. Words and phrases of breadth, such as "one or more," "at least," "not limited to," or other similar words and phrases, should not be read to imply that a narrower case is intended or required in the absence of such a broad phrase, as the case may be.

[0148] FIG. 1 shows icons representing elements used throughout all figures of this disclosure and serve as dictionary elements or glossary elements. Icon 1 in FIG. 1 represents a typical user of the display system described in this disclosure. The term "user" is defined above. Icon 2 represents a user engaged in collaboration with other users. Collaborative software applications include videoconferencing, online educational platforms, multi-user games or entertainment, and simultaneous streaming. Collaborative users may interact with each other in a shared visual environment. Collaborative users may interact with the visual environment asynchronously at different times.

[0149] Icon 3 illustrates an input stream drawn from a source. The input stream may be any content, such as visual content, metadata, programming code, textual data, database information, mathematical quantities, audio data, or numerical data. Furthermore, the format of the data stream may be any format, including, for example, compressed or compressed formats, vector or bitmap formats.

[0150] Icon 4 illustrates a generic source that can be remote or local. A source can provide data or metadata for display. A source can also operate on the data or metadata. A generic, local, or remote source may operate on the data before sending it to the display system. Icon 5 illustrates a local source. Local sources include workstations, laptops, and desktop computers, as well as microcontrollers and microcontroller arrays, that are physically connected to generate content for the main portion of the augmented display. Icon 6 illustrates a remote source. Remote sources include the Internet, IoT, remote servers, other computers on the extended network, distributed networks, or edge devices. Remote sources may also be referred to as "indirect sources," i.e., they provide information not directly related to or augmented information or display content to the extended portion on the augmented display. Remote sources also include computing modules that are not directly connected to the local source, that receive display content for the main portion of the augmented display system as input, use functions to operate on that display content, and output the results of the functions such that the output affects or becomes part of the display content for the extended portion of the augmented display system. That is, the remote source uses the display content of the main portion of the extended display to affect the display content of the extended portion without having any information about how the display content of the main portion was generated by the local source.

[0151] Icon 7 illustrates a generic display system. In the embodiment described herein, the display system is an augmented display system, but one skilled in the art can adapt and implement this description for use with any display system. In some embodiments, the display system simply receives data for display as content. In some embodiments, the display system may process the data. The display system may include an audio system, such as a microphone or speaker, that are synchronized to affect the displayed content. The audio system may be integrated into the display system. Icon 8 illustrates a local source combined with the display system. One example is a workstation with a computer monitor.

[0152] Icon 9 shows a general image or display content being displayed. Icon 10 shows a general image or display content being generated from a remote source. The image can be a separate display content or the image can be a subsection of a larger display content, with other portions of the display content being drawn from another source. Icon 11 shows a set of layers or multiple layers of graphical information, where at least a portion of one display content overlaps with at least a portion of a second display content. The number of layers can be any number, for example, 2 layers, 3 layers, 6 layers, 8 layers, etc. In some embodiments, the characteristics of the layers, such as the depth of focus, are adjustable.

[0153] Icon 12 represents a generic input device. Icon 13 represents a generic sensor that captures and communicates information about a person, user, or environment. Generic sensors may include cameras. Icon 14 represents a generic camera or camera system.

[0154] Icon 15 illustrates a block diagram icon representing a function that operates on at least one data stream. Icon 16 illustrates a series of connected functions or widget blocks that generate a desired output based on a specified input. Icon 17 illustrates a general annotation. Icon 17 may be used as a specific function to include, for example, text or graphics that appear in a multi-layered display or to generate annotations. Icon 18 illustrates a general AI module. Exemplary AI modules may include neural networks, transformers, or other deep learning or ML algorithms. An AI module may include multiple AI modules that communicate with each other, for example, by each AI module feeding its own output content into the input of the other AI module. In some embodiments, an AI module includes multiple AI modules that perform interrelated tasks, for example, creating a movie, where one module generates audio content, another generates visual content, the audio content influences the video content, and the video content influences the audio content. In some embodiments, multiple AI modules are configured to process separate tasks in parallel. In general, a "computation module" is a device configured to process inputs in a specified manner. A computing module tends to have a specific functionality and is usually different from a general processor, for example in a computer.

[0155] Icon 19 shows a general geometric transformation function. An example of a geometric transformation algorithm is a pose warp algorithm. Pose warping or motion warping may involve comparing time series of positions of points on an object and optimizing their distance using a dynamic time series (dynamic time series may be used for example for speech recognition) algorithm. The transformation function may be based on splines to transform various parameter curves. Such transformation functions or transformation algorithms may be used for stride warping, perspective warping, orientation warping, deformation warping, or motion warping. The geometric transformation function may operate on synthetic data, such as data about a character in a video game, or the geometric transformation function may operate on real data, such as images of a user captured by a camera and segmented from the environment based on machine learning algorithms.

[0156] In this disclosure, a geometric transformation is any type of geometric transformation, including shifts, rotations, affine transformations, and homography transformations. Geometric transformations also include computational remapping. For example, depth remapping is one example where the user's distance to the camera is processed to render a virtual image that maintains the correct physical or geometric proportions. Depth remapping may evaluate the remapping using an isomorphism or homography. Geometric transformations also include dewarping, which is used to remove distortions that may be caused by optical systems, including fish-eye or barrel / pincushion distortions.

[0157] Icons 20 indicate user-defined actions or user-defined models / templates. Any component of the software technology herein may be user-defined.

[0158] 2A-2D show embodiments of software applications configured for use in a virtual display system, including software generation, predictive applications, single user applications and collaborative applications, as well as software applications incorporating both local and remote sources.

[0159] FIG. 2A illustrates a software-generated application, referred to herein as a “stream weaver” (STW), for generating visual content configured for use in an augmented display system. FIG. 2A includes a sequence of steps involved in collecting and consolidating data from various sources 4, operating on said data using functions 15, and displaying said data to a user according to a template 21. Step t1 represents a set of N sources 4 from which data is drawn. The sources 4 may be remote, local, or any suitable combination of the two types. The sources may be video input streams, camera input streams, game input streams, applications, or any cord or device connection. Step t2 represents a set of functions 15 that process the data drawn from the sources. These functions may act on the input streams from the sources, including metadata of the sources. Step t3 represents the process of displaying the display content generated by the functions in step t2, configured for a selected visual template 21. The exported visual template 21 may be a built-in selection or user-defined. Various visual template options and features include display type or graphics specifications, focal plane or virtual image placement, resolution, brightness, and depth resolution. In some embodiments, functionality 15 is selected after or simultaneously with the selection of the visual template. An error correction block may be added to correct, modify, or improve the information created in step t2.

[0160] The functions and sources do not have to be arranged in order, and the number of sources does not have to equal the number of functions used. In some embodiments, a function receives multiple sources as input. For example, function "F4" may receive input streams from "Source 1", "Source 2", and "Source 3" as input. Functions may operate in a composite manner. For example, function "F8" may receive the output of function "F7" as input. Some input streams may be integrated into an export template, where no function operates on the input streams at all. In some embodiments, there are no functions, and all sources are integrated directly into the visual template. In some embodiments, a function includes a feedback loop, so that the output of the function may be fed as input to the function. This is the case, for example, when feedback is desired for stability, recursive functions, oscillations, or nonlinear dynamics.

[0161] The functions themselves include basic or extended mathematical and computational or graphic operations. Other functions include ML architectures such as self-attention transformers or neural networks. In some embodiments, the neural network includes a dictionary and training data. Functions are also typically time-dependent, depending on user input at the time of operation or on the history of the user's actions on the display system.

[0162] In some embodiments, the complete set of features may be determined by a generative neural network based on prompts entered into the system. This allows the computer to choose how things can be reshaped and visually displayed to the user based on those prompts. For example, one prompt may be "Show me a full view of search results of 1000 videos related to my search, highlighting the most popular videos." With such a prompt, the computer would define N=1000 and start sending N collectively, cohesively through all the features to display annotations at different depth layers.

[0163] In another very simple example, the user may have only a primary content source (e.g., a game stream) and navigate through the UI to choose how he wants to select (or create) other streams to interact with this primary content source. For example, the user may choose that for every frame on the primary central monitor, the two side monitors display the central image, the central color, the average color, a duplicate with a 2 second time delay, or an outpainting frame of the primary game stream. In this case, as mentioned, the two other monitors depend on the content displayed on the central monitor. The streams may be interactive interfaces, not necessarily video streams. This is a notable difference between video mixing performed in video editing software and multiple interactive streams mixed together here. More categories and trees of these functions are explained in Figs. 4A-4B, 5A-5J, and 6A-6J.

[0164] It should be understood that the functionality, visual templates, graphical user interfaces, AI, and other algorithms described throughout this specification and referenced in the drawings may be implemented in software, hardware, or any suitable combination thereof. Software may be comprised of machine-readable code stored in memory (e.g., a non-transitory computer-readable storage medium) that, when executed by a processing device, produces results as described, either in the processing device itself or in hardware operatively connected to the processing device (e.g., memory, augmented display system).

[0165] 2B shows a predictive software application "funnel expander" or "events" where past and present events or actions by a user 1 looking at a central display 9 are input into a function 15 which causes local sources including a display system 8 to work to display content to the user corresponding to possible actions the user can take. In some embodiments, multiple remote or local sources are used. Past actions may be displayed in a narrower or less prominent display 9A, and possible actions may be shown in a wider display 9B, such that past and present actions help to display an expanding or funnel shape of future possibilities.

[0166] Current input and feedback by the user captured by the general input device 12, camera 13, or sensor 14 is processed. The display content may include some infographics 22 that show the user history in a meaningful way. The user history includes which applications were used, which features of the applications were used, how long the applications were used, which applications were used in what order, the actions taken, the displayed display content, the duration and timestamp of the displayed display content, and the importance of the displayed display content as measured against some metric such as productivity. The function 15 may generate as output a set of predicted actions that the user is most likely to engage in. In some embodiments, the suggested content is formed by a method other than probabilistic analysis. This method may be event-based, priority-based, time-of-day-based, based on settings preselected by the user, or any other suitable method.

[0167] In some embodiments, the user interacts with an avatar 23, which can assist with user input or can be empowered to perform predicted actions. In this way, the user can perform multiple tasks in multiple parallel processes. An avatar can be a visualization, a set of text instructions, or a subroutine that is invisible to the user.

[0168] In some embodiments, the functionality is probabilistic, such that actions that occur most frequently or that are most correlated with the current action or the displayed content are weighted more heavily than other actions. In some embodiments, the functionality is based on a temporal factor, such that actions from the recent past are weighted more heavily than actions from the distant past. In some embodiments, neural networks or transformers are used to help determine or refine the predictive behavior of the software.

[0169] In some embodiments, the predictive features include estimates regarding the success of a user's current action, or how long it will take for the user to complete the current action, and how the user's schedule or calendar may be affected. The predictive features may use the calendar as an input to suggest alternative times for completing various tasks.

[0170] This embodiment allows for fourth-dimensional scrolling in both time and space, using the extended display screen as an infinite scroll with cursor or user input. In some embodiments, the user may be able to see parallel possibilities in multiple parts or depths of the extended display system and simply select the desired option using a gamified mechanic. Which parallel possibilities are displayed depends on the current user's actions and can therefore change dynamically in real time. This embodiment helps the user see as many computer-generated sets of possibilities as possible while achieving near real-time interaction with the computer (back and forth "ping-pong" feedback) when creating a computer data stream. For example, currently to write a word document one must write line by line, or if the text is generated by a computer the user must read one variant at a time and edit line by line or request different variants. In the embodiments described herein, the expanded set of variants is displayed in different parts of the extended display, allowing the user to also make real-time selections of which variants will be woven into the text as they read.

[0171] Another example is a rotating screen embodiment. Currently, users are limited by the vertical resolution of the screen when scrolling through websites, computer code, or vertically long data. In the case of a three monitor setup, this arrangement does not lend itself to seeing more such vertical data. In a funnel expander, the user has a side monitor or a front depth layer as a continuation of such vertical data. The funnel expander may offer different possibilities or parallel possibilities within the monitor, in other depth layers, or in the peripheral FoV. For example, in a VR headset, the user can see multiple other parallel articles appearing beside the main article, which can be seen in the periphery when reading a vertical article. Further details of the funnel expander are shown in Figures 7A-7G.

[0172] Figure 2C illustrates the use of a software application in an environment where one or more users interact with the software generated by the embodiment of Figure 2A or 2B. The users may interact with the same content in different ways, i.e. the content may appear in a different format (Format A) or visualization to a first user 1A compared to a format (Format B) to a second user 1B.

[0173] A first user 1A uses a display system that generates multi-layered display images 11, floating graphics 24, and 2D augmentations 25 in addition to a central display 9. The user inputs information by any means, such as a general input 12 or a sensor 13. Based on the user input, or based on a function that determines the display content, the display content in each of the multi-layered display images 11 may be pushed forward or backward relative to the front of the user's viewable area via function 15. The displays 7 may be connected to a local source 5.

[0174] In some embodiments, multiple display systems are connected via a remote source 6 (e.g., the Internet). A second user 1B interacts with a local source and a display system 8 that displays similar content as the first user 1A. Display content of type B may be presented using different templates for different users. For example, in some embodiments, the visual template may consist of multiple sets of a first image 9 and multi-layered images 11A and 11B, each configured to interact via different functions.

[0175] For example, user 1B may use a generic input 12, such as a mouse, to scroll through a video game environment, and as the video game character moves around the environment, different layers, each corresponding to a different depth within the environment, move closer to or further away from user 1B. A first user 1A may be a teammate in the game, and the floating graphics 24 may be used as annotations regarding the health of first user 1A's teammate.

[0176] In another example, a video conference call application may depict users on one layer and various calling features, virtual whiteboards, shared environments, or notes on other layers. The various display content and display layers may interact with each other via functionality. For example, a floating graphic 24 for a user 1A may present information based on a set of images that includes a video of another user 1B in a multi-layered display configuration.

[0177] FIG. 2D highlights an embodiment in which multiple users interact with the display content via remote and local sources. In some embodiments, there is only one user, but there may be multiple users. In FIG. 2D, a first user 1A is viewing a pair of display images in a set of multi-layered images 11. A back layer of the set of multi-layered images 11 may be generated by a remote source 6 and correspond to a shared visual area, where multiple users can access the same display content of the visual area. Input streams from the remote sources may be manipulated by functions 15 before being displayed. The front layer may be generated by a local source 5 connected to the display system 7, or may be generated by a local source in conjunction with the display system 8. In some embodiments, subsections 26 of a particular layer or image are generated based on user input, history, or settings. In some embodiments, the sections are not contiguous. In some embodiments, the subsections are individual pixels or sets of pixels. These shared visual areas can be, for example, a shared visual environment corresponding to a common space on the back layer, and a perspective or window into that common space generated in the front layer.

[0178] In some embodiments, the input from the user is motion tracking, SLAM input, or orientation input to dynamically change the scene based on the user's position or orientation relative to the display system. In some embodiments, a subsection of the displayed image is input to a back layer affecting function 15. In some embodiments, the division of the data source depends on the bandwidth, image mode analysis, which is content dependent. The user may be an active user and may manipulate windows, or the user may be a passive user and may simply experience determined content, as may be the case in an advertising use case, where the displayed content is intended to showcase a product or service.

[0179] In some embodiments of FIG. 2D, multiple users may be in different environments, for example, viewing a high-level virtual "cloud" image from different workstations, with local workstations providing different windows into the display content to the cloud image as sharable viewing zones. In some embodiments, a single user may be viewing content from multiple input sources. In some embodiments, the display system comprises a main workstation that is influenced by mobile devices, tablets, or distributed networks.

[0180] 3A and 3B show a flow chart of the software generation program of FIG. 2A.

[0181] The flow chart of FIG. 3A shows that in step 27 a set of source information is first selected or described. Then in step 28A a function is described or selected that operates on those sources. In some embodiments a function takes one or more sources as input and produces another source as output. Then in step 29 a visual template is described or selected. In some embodiments the selection of a visual template allows for the selection of further functions in step 28B, such as the orientation of the visual template, which sources appear in which templates, etc. In some embodiments, error feedback action steps 30A and 30B between the description or selection check for errors, inconsistencies, or incompatibility of the selection. In some embodiments, the software generator makes suggestions to optimize, modify, or improve the resulting software by comparing the selection of sources, functions, or templates in these feedback loops. This may occur due to prompts given to the software, or may occur dynamically based on user or other sensory input. Finally, an export interface stream 31 is selected by the user or an algorithm to define the form of the final interface or stream. The selection can be, for example, a particular format type, compression rate, or file name.

[0182] The flowchart of Figure 3B represents an alternative flowchart in which source description and template description occur simultaneously in step 27A, then all possible functions are selected in a separate function description block in step 28. The stream is then exported to a final export stream for the end user in step 31. As with the flowchart of Figure 3A, in some embodiments, the feedback step 30 utilizes an error checking module between the selections to check for errors, inconsistencies, or incompatibility of the selections. In some embodiments, the software generator makes suggestions to optimize, modify, or improve the resulting software by comparing the source, function, or template selections.

[0183] 4A and 4B show the core pipeline of the Stream Weaver (STW) process. This pipeline may be a series of drop-down menus in a GUI for a user to generate a software application. The GUI may be of any configuration, but may be configured to clearly show information about input streams, functions, sources, and visual templates. The STW process shown in FIG. 4A starts with a source extraction step 32 to determine the sources from which data or input streams are to be retrieved. The process continues with a function placement step 33, where the data and information retrieved in the previous step are processed using various functions; that is, in this step, functions are selected, inputs to the functions are selected, and outputs are selected. Depending on the selection of functions and sources, the output may be automatically determined. The next step is a template selection step 34, where a visual template is selected to assemble the information processed in the previous step. The final step is an export step 35 to export the information to a user or other application.

[0184] 4A also shows exemplary input streams, functions, visual templates, and export modes. The source set 36 includes, but is not limited to, a camera 14, a video or clip 37 (the video or camera source may be any, e.g., not limited to a camera capturing a video of a user), music or voice recording 38, a UX environment 39, GPS or other mapping data 40, a text document 41 with or without annotations 17, a website 42, a gaming application 43, metadata or hyperlinks 44, a general data stream 3, a remote source 6 (e.g., cloud-based data), an output of a function 15, or a library 45, or general sensor data 13. The functions may be individual functions, or the functions may be grouped into functional blocks.

[0185] The function block set 46 includes, but is not limited to, a camera source function block 47, a UX or UI source function block 48, a text / annotation source function block 49, a general source function block 50 (functions within this function block may be arbitrary or user defined), an engine source function block 51, and an AI generation function block 52. In these function blocks, the functions themselves are AI generated, for example, based on understanding or classification of an input stream. For example, the input stream may be a video, and an AI function first classifies the type of the video as a training video or an entertainment video. Then, another AI function may generate an action based on an expected user desired application.

[0186] The visual template set 160 includes, but is not limited to, templates for displaying information such as floating graphics 24, multi-layer screen 11, expander mode in edge mode 53, horizontal 2D desktop extension 25, tandem extension or virtual bandwidth display 54 (a display in which at least a portion of the image is generated by a remote source), user-defined templates 55, and AI-generated templates 56. The templates may be automatically generated based on the output of the functions in the previous steps. For example, the output of a clickable training video containing annotations may be a display including multiple floating graphics that contain annotations and that automatically shift based on the movement of the annotated object.

[0187] The floating graphics 24 can display display content such that the viewer's eyes accommodate to closer distances than the physical display system. In this way, the floating graphics appear closer to the user than the display system itself. The floating graphics can be generated, for example, using phase conjugate, retroreflective, or retroreflective (retroreflective in transmission) elements that cause a point source of light from the display system to be focused between the user and the display system.

[0188] The multi-layered image 11 displays multiple layers of display content such that the viewer's eyes accommodate to different depths, resulting in the viewer seeing different display content coming into focus. The multi-layered image 11 may be generated, for example, using multiple display panels or a field development cavity that includes switchable elements that can change the path length that the light travels, by cycling the light back and forth one or more times depending on the polarization of the light.

[0189] The edge mode expander 53 and the 2D expansion template 25 generate a virtual image that expands the viewer's FoV. This can be achieved by starting with multiple displayed images and moving light along paths that move in different directions before exiting the system. To form a cohesive image across the entire depth plane, multiple images are tiled together such that the separation is smaller than what the human eye can see, for example, a person with 20 / 20 or 20 / 40 vision can see when viewing the displayed content. In some embodiments, gaps may be desirable. In some embodiments, tiling occurs in multiple directions, for example vertically and horizontally. In some embodiments, images or data are spatially separated in the extended FoV using any template. The position of the tiles or spatially separated images may change dynamically according to user or sensor input or various computational routines.

[0190] In some embodiments, the edge expander or extended FoV template uses multiple physical monitors in an extended display system. In some embodiments, the edge expander or extended FoV template may be a virtual image generated by a virtual display system.

[0191] A tandem extension or virtual bandwidth display template 54 is a display where information about a portion of the display content has been received by a remote source. This information can be the display content itself (e.g., remotely rendered display content), metadata about the display content, information about graphics settings, or data about the environment. This information can be specific to a particular application and can affect multiple applications. In some embodiments, the division of the display content affected by the remote source changes dynamically depending on user settings, application characteristics, or bandwidth constraints.

[0192] The result of the export step 35 is a software application set 57 including, but not limited to, new applications which may be predictive applications, interactive videos 57A (which may be clickable), metadata, databases, new UX 57B, new games 57C including interactive features or dynamic game engines, and / or interactive media.

[0193] The resulting applications generated by STW may be displayed on an augmented display system. These applications may be displayed on a virtual display system.

[0194] Figure 4B shows a detailed view of the user-defined or AI-generated template selection of Figure 4A. In this process, the template drop-down menu 34 focuses only on user-defined templates 55 or AI-generated templates 56. If either of these templates are selected, a new property drop-down menu 58 appears. The user defines the new template by selecting from among a set of properties 59.

[0195] The characteristics include the shape, orientation, and location of the displayed content, the resolution of the center, and the allocation of different sections to different sources or studies. For example, in some embodiments, the user selects the shape of the displayed image, which can be a square, a rectangle, any rectangle, a triangle, a circle, or a bubble, or any combination. The resolution can be any setting of resolution, such as high resolution, full high resolution, wide ultra extended graphics array, quad high resolution, wide quad high resolution, or ultra high resolution. The user-defined visual template can be a combination of the visual templates shown in FIG. 4A.

[0196] The properties dropdown menu 58 may include a set of AI parameters 60 for the AI-generated template. For example, a user may select various AI analyses to perform on the output of the function. A user may want the AI-generated template to first analyze the bandwidth of the output and then generate a 2D augmentation with a size that can display all the information. Or a user may set the AI-generated template to first perceive or estimate the depth range of the image and then generate a multi-layered image that includes a depth layer that optimizes the viewer's depth perception, for example, by matching the depth layer to the horopter of the human visual system.

[0197] The user-defined template may include a permissions drop-down menu 61 for selecting, through user interaction, various permission settings, including whether the resulting software can integrate an app, multiple apps, span the entire computer's operating system, include Internet access, or generate active or passive media.

[0198] In some embodiments, the template may be a generic and dynamic 2D geometric outline or an arbitrary mask, and may be displayed on the same 2D display. For example, the display may be divided into triangles to display a video, while another triangle is a camera video stream for a game in a more attractive format. In some embodiments, when a user is reading a text file on the screen, input from an eye-tracking device may understand where the user is looking, and therefore automatically dim other parts of the display content except for the highlighted area based on the user's gaze position. In some embodiments, the gaze area may be rendered in any other different way or with different characteristics. For example, the gaze area may be rendered with higher graphics fidelity, or the gaze area may track a set of tool options as the user looks around, so that the toolset becomes more accessible no matter where the user looks within the FoV of the screen.

[0199] In some embodiments, the mask may be dynamically changed based on internal or AI algorithms that are propositional and generate contours or masks based on an analysis of the displayed content.

[0200] In some embodiments where there are multiple depth layers, there may be a set of tools displayed in a first layer that follows the user's head and eye position and shows the user the most likely choices to make based on other pieces of information displayed on the screen. However, in this case the user does not need to move the mouse and click buttons in the native underlying app. Instead, an arrow key or other modifier key may simply be clicked to proceed, with suggestions displayed, which helps reduce repeated mouse movements.

[0201] In some embodiments, templates may be defined within a 3D environment such that the display content undergoes affine transformations such that the display content is displayed as if it were mapped to or draped over different facets of the 3D environment, e.g., an advertisement is transformed into a perspective view for display within the 3D environment.

[0202] In some embodiments, the applied geometric template may dynamically change based on events or performed action items in the primary or secondary streams, for example, when an event occurs in the game, such as a character shooting or jumping, the display content on the side may glow a certain color or show a certain image, or may expand or shrink.

[0203] In some embodiments, the templates include templates configured for display on multiple parallel devices. For example, a mobile phone screen or a tablet screen may share a visual template with a laptop. As a non-limiting example herein, if a game character is jumping up and down in the game, the particular display content may be displayed on the laptop, secondly on the mobile phone, and thirdly on the tablet.

[0204] In another example, a user is conducting a financial trading transaction using a desktop screen and selects a mobile phone or tablet screen as part of an application generated by STW: when a particular news item comes along or a particular stock is updated, the relevant content of that stream is sent to the mobile phone or tablet.

[0205] In some embodiments, STW is used to create simulation, training, or educational applications. A user acting as a trainer or educator may share a depth layer, auxiliary view, or part of the extended FoV with the trainee and provide geometrically manifested training instructions related to the training medium and training materials. In some embodiments, the trainer may be a chatbot or an AI-based algorithm generating instructions by predicting the user's intent. In some embodiments, the AI ​​may have the authority to access the primary input stream as opposed to simply displaying what the user can do. In some embodiments, training content may be played step-by-step as a video stream in front of the user.

[0206] Training and simulation experiences may include multiple users. For example, an instructor or trainer may be observing the users training on a display system. The instructor may be using his or her own display system, or an image of the instructor may be captured by a camera and displayed to the user on an augmented portion of the user's augmented display system. The instructor may provide live feedback to the user based on voice, keyboard or mouse input, or sensory input, which may be presented on the user's display system as visual or textual content, as annotations or modifications to existing annotations.

[0207] In some embodiments, multiple users may each be using the display system, but to mimic the experience of being next to each other, an image of a first user captured by a camera is displayed on an augmented portion of a second user's display system, and an image of the second user captured by a camera is displayed on an augmented portion of the first user's display system, each image may be distorted or undistorted to provide a realistic image of the surroundings.

[0208] In some embodiments, a display system includes multiple display devices, such as a communicatively coupled free-standing monitor and a headset. For example, the free-standing monitor may display a wide-field image of a simulation or training exercise, and a user wears a headset that displays annotations based on the displayed content of the monitor or the user's geometry or line of sight. Communication between the headset and the monitor may be hardwired, for example, via a connecting cable, or wireless, for example, via a wi-fi network or a remote central source.

[0209] In some embodiments, the STW application may be configured to assist with editing the video depending on permission settings and output templates. The AI ​​program may show the user how a task is performed in a video stream that appears as part of the augmented display, or the AI ​​or trainer controls the program and performs the task step by step in front of them. At any of the steps, the trainee may interject and / or collaboratively change what the trainer is doing based on user or sensory input.

[0210] 5A-5J show several examples of functional blocks for various sources. These various functions are selected by a user using the STW to operate on selected input streams for a final display structured into a visual template. The functional blocks may be a way for the STW to structure and select available functions. The final software product, which is the software-based display content generated by the user using the STW, includes general apps, videos, clickable videos, metadata, predictive apps, databases, games, and interactive media. The following embodiments describe both the functional blocks of the STW and some of the resulting software applications.

[0211] In some embodiments, such as "Media annotator using user input" in FIG. 5A, the sources include camera sources that may be used for VR or AR applications, video see-through, remote operation, other remote control devices, video conferencing, or video content creation, some of the possible functions appear in the camera source function block 16A. The resulting software is used for interactive video applications. For example, the camera source function block may include annotator function 17 so that the display system generates digital content to overlay on top of the camera source content to highlight features of the video. The digital content may be located in the same focal plane as the camera content, or the digital content may be located in another focal plane, such as in floating graphics or floating text or in another layer of a multi-layer template. The annotations may be pre-programmed according to a user profile or actions, or the annotations may be dynamically generated by an AI module. The camera source function block may include a comment function 63 so that the user can provide feedback or ask questions regarding the content. User feedback may be a text-based feedback mechanism integrated into the AI ​​module, such as a chatbot that can respond to feedback. More generally, there may be a user input function 12, which allows the user to provide input to the software in any manner, including keyboard input, mouse clicks, gestures or facial expressions, or voice commands. In some embodiments, the user input function is configured to request input of a specified frame of camera source information, for example, for use in an online quiz or training video. Another function is an avatar assistance function 23, which allows the user to interact with a virtual avatar or assistant. The avatar may provide suggestions based on the user's content or camera source information to guide the user through the experience. At the end of this functional block is a graphics function 64.Graphics include warning labels, images of the user's celebration, or graphics to highlight features of the camera content. Graphics functions may be implemented as standard graphics functions, e.g., processing video frames, or the graphics functions may be based on sensor input or user input of the user. For example, a graphics function may capture the user's gaze and illuminate the area of ​​the display on which the user is focusing.

[0212] 5A-5B, specific functionality is associated with specific sources, but those functionality may be used in other embodiments. For example, avatar assistant 23 may be used in any embodiment to assist or affect the functionality of the resulting software.

[0213] In embodiments, the avatar assistant may be programmed to output information based on the relative importance of objects in the training video, and can take user input such as voiced questions and answer those questions based on the video content. The facility may be connected to dictionaries, training data, or search engines related to the content of the video to provide additional information upon request, or to provide clues that link concepts from one part of the video to another.

[0214] In an embodiment, the graphics function may highlight features on the video based on the user's progress through the video, the user's gaze captured from a sensor, or the user's SLAM input. For example, the video may be a training video for proper posture when performing a physical task, and the indicator function takes the user's posture as input and compares it to the character's posture. The function outputs highlights in the video to indicate how the viewer should change their posture relative to the character in the video, for example, by highlighting the back posture or shoulder posture of the character in the video compared to the user. A flow chart of this example is shown in FIG. 6A.

[0215] 5B shows an "e-commerce smart recommender / advertiser" embodiment in which the camera source functional block 16B is configured for use as an online shopping platform or source of advertisements. For example, multi-layered display content may include a video layer that is a commercial advertisement and a second layer that includes annotations of items based on the user's gaze captured by the camera. The annotations may highlight items available for purchase or display further information within the augmented display image.

[0216] This functional block may use live video or video recording. One of the functions includes a purchase function 65 configured to highlight purchasable items in the video content and include links to online shopping platforms. Purchasable content may be identified by object detection algorithms and a search engine that determines saleability, and the software may determine which objects to highlight based on user input or user profile. A flow chart of this example is shown in FIG. 6B. For example, a user who browsed and purchased scientific equipment will see different highlighted objects than a user whose browsing history focuses on interior decoration. The input of this function is the video frames and user input / profile, and the output may be an annotation layer that includes saleability information, purchase options, and various alternatives. In some embodiments, the details are influenced by the user's history with the display system, previous purchases, search history, or details specific to other users. A share function 66 allows the user to share content from the camera source with other users or potential users in the network. This function may, for example, create a quick response (QR) code for an object detected in the video, which is shared with other users so that the QR code appears in the floating graphics of other users watching the video. In this example, the QR code is generated based on where the user clicks using mouse input. Another function is a comment function 63 configured for a user to click on an object in the video and provide feedback, which may be text generated by the user or selected graphics, to the creator or various vendors of the video. Input to this function includes an annotation layer generated from a purchase function 65. In some embodiments, the purchase function or feedback function is configured to receive as input the user's body type based on a camera system connected to the display, for example, to determine whether a wearable item is a good fit.In these embodiments, the geometric transformation sub-function must be used to align the shape of the user to the shape of the vendable item on the screen to create a virtual reality image of the user wearing the item.

[0217] Similarly, the referral feature 67 allows the user to obtain more information about an object in a video by viewing customer testimonials about previous purchases or to connect to online forums to review products. For example, in some embodiments, when a user places the cursor over a particular object, a list of user experiences with that product is displayed in a floating graphic or edge expansion display.

[0218] Another function in this block is a synchronization function 68, configured so that information regarding the navigation of the user's software experience in the current case is automatically entered, for example, into the software accounts of multiple users on various shopping platforms, into separate software accounts of an individual user, or into the memory bank of a future user of a referral function 67. For example, a user may synchronize a shopping platform application stored on a mobile device, and the shopping cart or browsing history is entered into the multi-layer display device, such that various annotations and QR codes are highlighted or de-highlighted.

[0219] In another embodiment, “Teleoperation / Collaborative Experience Facilitator,” shown in FIG. 5C, the camera source function block 16C includes functions for various collaborative visual or monitoring environments, for example, for use in online classrooms, webinars, quality control monitoring, control centers, or teleoperations. In some embodiments, the camera source function block includes a general sensor integration function 13. This function allows integration of any sensor connected on a network, or other camera (e.g., from other users, remote or autonomous vehicles, security cameras, or cameras observing robotic or machined parts). These inputs may be synchronized so that their content is overlaid in real time in a multi-layered display. In other embodiments, images are tiled in an extended FoV, for example, for use in a multi-camera vehicle navigation system to generate a bird's-eye view of the vehicle's environment. Another function is an image processing function 69. This function may include separate subfunctions that operate the camera source, or any sensor source configured to generate visual content. For example, the inputs to this function may be the video itself and sensory inputs, and the image processing function is programmed to output brighter pixel areas or display content on which the viewer needs to focus their attention, based on a user-defined monitoring task.

[0220] Additionally, the whiteboard feature 70 allows users to share separate applications or merge separate applications with a camera source, such as in an online lesson for an online course. The shared content may be a traditional sharing mechanism, or the shared content may be a dynamic mechanism where the content is dynamically transformed to adjust to the needs of the viewer. For example, the input to the whiteboard feature may be a data set of flight trajectories, and the feature is configured to plot those data into a visual trajectory that is overlaid on a multi-layered flight simulator.

[0221] For example, the augmented display system may include an area where multiple users can interact with each other via virtual images of themselves captured by a camera. This area is generated by the whiteboard functionality 70. The second area, which may be a second layer in the multi-layered display or augmented field of view, may be a virtual whiteboard space that is manipulated by users via gaze or gesture sensing. For example, the sensor integration functionality 13 may receive as input gestures captured by a gesture sensor or camera system and then determine actions, such as handwritten text, to display in the virtual whiteboard space. An example of this is further described in FIG. 6C.

[0222] For displays whose content includes images of a user or a part of a user's body, projection mapping or geometric transformation can be a type of image processing function used to affect the displayed image. Geometric transformations may include removing distortions of the optical system. In general, geometric transformations may be removed or corrected in any manner. For example, polynomial transformation algorithms may be used to remove lens or fish-eye distortions. Camera calibration may be used to remove distortions from the camera.

[0223] Image processing functions 69 also include brightness adjustment, foveated display, edge enhancement, blurring functions, video or image filters, background blurring, computational remapping, etc. This function may operate on the entire source, or it may operate on a segmentation of the source determined by the user or based on sensor input. This function may require other routines to assist with image processing. In an autonomous or remotely operated vehicle, a full view is displayed and one of these image processing functions is configured to identify an object, estimate the speed of the object, and then highlight the object if its speed exceeds a threshold. Another function is an AI module 18 configured to analyze all visual content together and propose generative methods to act on the content.

[0224] Additionally, the audio function 71 is a function for modifying sounds, music, and other audio effects. The audio source can be a microphone connected to the display system, or it can be a remote source. This function can also be configured to output audio via any speaker or other audio transducer. For example, audio signals can be configured to sound as if coming from a first or second layer in a multi-layer display, by holographic or beamforming methods, so that when a user hears the audio, they perceive a distance associated with the source. This can be, for example, a whiteboard space, or an audio effect associated with speech made by multiple users in a virtual classroom. Beamforming is caused by using an array of speakers that each radiate individual sound waves such that the sum of the sound waves creates a wavefront that approximates the sound source from the desired depth. The individual sound waves are determined by an optimization algorithm that outputs the relative phase of the individual sound waves based on how accurate the approximation is.

[0225] FIG. 5D shows an embodiment of the camera source function block 16D "multi-source / content generator and merger" where the camera source is a generic image source. This function block includes image processing function 69 and annotation layer function 17 so that various descriptions or visualizations of the camera content can be overlaid on or displayed near the camera content. The embodiment also includes a merge function 72 that allows the user to merge other video or camera sources with the original content. For example, this function block may be used in an embodiment involving remote operation or research methods, where a camera or photodetector is recording optical information about the setup and other sensors are monitoring the equipment used, and the merge function merges the overlaid sensor data with the camera. The merge function may be first automatically analyzed by a user-selected or user-defined function, and a side window of the augmented display displays alternative results that could have occurred using equipment settings shifted by incremental amounts. The alternative results may be calculated by numerical simulation of the underlying physical laws or by deep learning algorithms.

[0226] In some embodiments, the merging function may be based on an AI neural network that compares data for various correlations and trends. In this example, the original image may be merged with AI-generated image content based on user specifications, which may include touch-up functions, automatic encryption of visual data, or content generation of video media.

[0227] In an embodiment, the video may be live footage of a workplace, such as a construction site or warehouse, for monitoring personnel. In this example, the central display may display the live footage, and the augmented display image may display snapshots or frames of the live footage. In this case, the merge function 72 is programmed to merge past frames of the video with the live frames in the augmented display. A subroutine of the merge function may first analyze the frames to identify important or correlated personnel actions, such as incorrect procedural actions, productivity levels, or interactions with coworkers. This subroutine may use a CNN to detect similar objects or poses. Another subroutine may add annotations for the user to note when these frames are displayed. For example, the output of the CNN may detect and display all frames in which personnel are lifting heavy boxes in a warehouse, identify frames in which too few people are present, and add annotations to alert the user to intervene. This embodiment is further described in FIG. 6D.

[0228] In some embodiments, the video source is used within a video editing environment. In some embodiments, the merged content is not visual content, but some other type of information, typically to affect or improve the camera content. The merge function may depend on the particular layer or subsection of the layer of interest in the multi-layer display. The audio function 71 allows the user to edit, add, or emit an audio signal. Finally, the upload function 73 allows the user to send the content or a portion of the content to another device or network. The upload function may include its own merge or sync subroutines that collect content from multiple users and add the content to a database or training library for machine learning algorithms.

[0229] Another embodiment "Benchmark and Logic Analyzer" is shown in FIG. 5E, where the source is a text-based source and the set of functions is the Text Source function block 16E. The text source may be a document, a spreadsheet, an online address book, a magazine or publication, an e-book, computer code, or a presentation. This function block includes a merge function 72 that allows multiple text sources to be merged. For example, a user may wish to combine two versions of computer code. This function may be configured using multiple options. The first option is to update existing lines of code with updated code written in a separate file. The second option is to compare the two versions of the computer code and generate an optimized updated version based on the two inputs. In some embodiments, the user's original code may be automatically compiled, executed, and evaluated, and a set of adjustments or alternative algorithms are proposed within different display content, arranged in a way that allows the user to compare the various runs. This example is further explained in FIG. 6E. The annotation function 17 can add explanatory, graphical, or other visualizations to the original text in the floating graphics or multi-layered display. In some embodiments, the text or annotations can be made event dependent or object anchored, for example, by clicking on a QR code. In some embodiments, SLAM input or eye gaze input influences the annotation. In some embodiments, the annotation function may include subroutines configured to read the text for tone, coherence, logical soundness, or sentiment, and annotate locations in the text that require correction, or suggest alternative paragraphs or images in the augmented display image.

[0230] Another function in this block is a logic analyzer function 74, which is generated by logical programming, for example, by mapping trivial statements to programming commands. The user may specify the method of proof and configure the function to prove by induction, prove by contradiction, or another suitable method of proof. Alternatively, the function may use AI generative techniques to collect various proofs and theorems available online and generate new proofs. The function parses text or code into statements with truth values ​​that are analyzed based on the structure of the document. The output of the logic analyzer function 74 may be a classifier that ranks the strength of the argument or may point out logical flaws. In some embodiments, the output may include suggestions for correcting logical errors. The logic may be a formal linguistic logic based on the rules of Aristotelian logic, or the logic may be formalized as a mathematical logic, for example, as used in axiomatic set theory or geometric proofs.

[0231] User input functions 12 allow the user to interact with the text, for example using gestures. In some embodiments, the input is the same as the input at the source, for example a new input next in an existing document. The user input can also be a new method or mode of input, such as a speech-to-text function or a speech-to-computation function. Finally, in this embodiment, there is a comment function 63, which allows the user to annotate or view metadata or other characteristics of the document without directly editing or changing the text.

[0232] FIG. 5F shows an embodiment of a user-defined source function block 16F configured as a way to design a software engine, i.e., this embodiment is a "software engine / data assembler." The source that this function block operates on can be any data type. For example, the source can be a database, a point cloud, a lookup table or dictionary, an online repository, the Internet, or a library of code blocks. The type of engine that is generated is arbitrary. The engine can be a database engine, a graphics engine, a physics engine, a search engine, a plotting engine, a web browsing engine, or a game engine. The STW can include multiple function blocks to create multiple engines. The engine can assist with content processing, scene understanding, or image processing. In some embodiments, the engine is a recommendation engine configured as a real-time engine or an app-specific recommendation engine.

[0233] In this functional block, library functions 45 may be used to classify libraries of various engines or to design or implement new libraries. In some embodiments, the library may have a user query or desired task as input, and the library is generated based on the AI ​​module. For example, a user may input the library function "provide all subroutines for plotting three-dimensional data in a graph," and the library function may search the source data or generate the data itself to output a method of graphical display of the data. Or the library function may take the input data and identify a library based on the structure or size of the input data. For example, the input data may correspond to a gene sequence or a series of proteins, and the library function is an AI-based function that first identifies the data as a gene sequence or a series of proteins, searches the internet for all such similar datasets, and builds a library of datasets in the same format as the input data.

[0234] The graphics function 39 may allow customized graphics settings, such as changes in resolution, frame rate, or brightness, for use in visual applications, physics-based graphics rendering or engines. In some embodiments, the graphics function may include sub-functions that implement various physical or dynamic laws to render graphics. The input data for this function may be point clouds used in video games or scientific images for research purposes. This function may also be a subroutine of a specific game engine function block.

[0235] The UI / UX function 75 acts on the source and displays it in a helpful or attractive way. For example, the UI / UX function 75 may include a sub-function that (1) takes in numerical data and classifies the data set based on an AI module, (2) optimizes the best mode of presentation based on the classification and data size, and (3) graphically arranges and generates annotations / labels for user interaction. This embodiment is further described in FIG. 6F. Another sub-function includes a prediction function, which can be a probabilistic function, a time-dependent function, or a neural network or other deep learning function, that takes both the source and the history of user input as input and generates new graphics that suggest possible future actions for the user. For example, an AI-based UI / UX function may classify the data as weather data in a certain area over a certain period of time, and a toolbar is generated that allows the user to average the data, extrapolate the data into the future, or search the Internet for data from different time intervals.

[0236] In some embodiments, for example, the desired engine is a database engine and the display panel is configured as a multi-layer display, with the depth layer corresponding to another dimension of the data to generate, for example, a three-dimensional database, which may be used to manipulate volumetric information such as a point cloud of images. The UX functionality pulls the data from the database, analyzes the structure of the data, and in some cases compares the data against a library of data types, and presents the data in a visually appealing manner, such as an infographic or multi-dimensional graph.

[0237] Code block 76 allows a user of the generated engine to generate new code and modify or improve the engine. Neural network functionality 77 allows the engine to incorporate neural networks for any application. For example, in a game engine, a CNN may be used to detect objects in a scene captured by a video camera and incorporate them into the video game environment. In some embodiments, the API functionality further allows the user to configure source information to interact with local hardware or hardware distributed over a network. For example, data may be pulled in real time from a set of camera images or from details of the use of an appliance or machine.

[0238] In the embodiment "Game and world warping engines" shown in Figure 5G, the source is an existing game or game engine and the function block is the game function block 16G. This function block includes graphics functions (such as resolution effects or refinements) 39, audio functions 38, a comment function 63 for adding comments (e.g., in a multiplayer game), a compute remapping function 78 for 3D remapping effects and mesh creation, and a geometric transformation function 19 for various warping effects of game characters or graphics in the game. In some embodiments, annotation functions are included.

[0239] In some embodiments, the existing game is a first-person game, and various items in the scene are displayed at different depths on a multi-layered display. In some embodiments, one of the layers may be an annotation layer to provide hints based on the user's gaze or character movement. In another embodiment, a user may be playing a game, the character is an image of the user captured by a camera system, and geometric transformations are used in the geometric transformation function 19 to dynamically optimize the shape and size of the character in the game. In some embodiments, the game is a beta version of the game, and the AI ​​component suggests different viewpoints or interactions within an extended delay window as the user evaluates the game. An example of this is further described in FIG. 6G.

[0240] In some embodiments, such as the "Dynamic UI Creator" shown in FIG. 5H, the UI function block 16H includes any UI for the source. The UI can be a landing page with specific features, buttons, links, icons, visual elements, or audio elements of a website. The UI function block includes a graphics function 39 and the ability to set various graphics qualities, receive user input via an input function 12, detect or upload information from local or remote sources via an upload function 73, or receive instructions via a download function 80. In some embodiments, a user-defined function 20 allows the user to manipulate the input source in any way. The user-defined function can be an image processing sub-function block, a terminal window for writing, compiling, and executing code, or any function described in this disclosure. For example, in some embodiments, this function block is used to test a website, where the user tests the website using various input prompts such as checkboxes or radio buttons. As the user navigates the website, gazes and interactions are recorded and then annotation overlays highlight or light up unused areas to allow the designer to get graphical feedback about the website, or website features may dynamically adapt based on the tester's past usage. An example embodiment of this is illustrated in FIG. 6H.

[0241] In FIG. 5I, the illustrated embodiment "Media Feature Recognizer and Annotator" includes display content as a source, which may include generic display content output from a previous function block. For example, here the input may be a clickable training video after being manipulated by the functions of FIG. 5A. The composite function block 16I includes a detection function 81 for detecting features in a source image. The feature detection may be low level (e.g., edge detection), mid level (e.g., gaze or face tracking), or high level (emotion detection in display content that includes people). In some embodiments, the detection may be object detection, or the detection may be feature detection related to the environment rather than the user. Another function is a user-defined function 94, which is any function determined by the user. In some embodiments, a user-defined function 20 is included, which may be a mathematical operation. In some embodiments, this function inputs the source (as training data or input data or data to encode) into other selected functions or machine learning pipelines. Further functions include an annotation function 17 for adding annotations and annotation layers to the source, a code blocks function 76 for generating and compiling custom code that operates on the source, and an image processing function 69 for processing the source images or videos using existing image processing functions.

[0242] The code blocks function 76 may be aided by generative AI such that code blocks are automatically generated and merged with source data based on training data. In some embodiments, the code blocks function may display a terminal in a side window or display, allowing the user to modify or influence the AI-generated code in real time via feedback.

[0243] For example, in a remote survey of an environment or a search and rescue operation, a camera may capture images of a display for a user to survey the scene. A primary display layer displays the scene, and in an emphasis of the multi-layer display, a second layer is programmed by a user-defined function to detect people or faces. Furthermore, the user-defined function or parallel function subroutines allow a higher level of scene understanding to quantify the danger of people for the rescue team to prioritize. In some embodiments, the video is a simulation-based training video, and the user is asked to determine the danger and rescue operation. An example of this is further explained in FIG. 6I.

[0244] In some embodiments, the various ML / AI engines are separate functions to operate on the input. For example, in a clickable training video, the user may be asked to select components of an image based on various other data in the displayed content. The AI ​​engine predicts likely outcomes based on possible selections or based on the user's gaze. Training difficulty, time response, and future developments dynamically adapt based on user actions and AI training.

[0245] In FIG. 5J "Visual environment / UX immerser", the input sources can be sources configured for the generation of a visual environment. Such a visual environment can be for immersive video conferencing or online classrooms, for which a virtual immersion functional block 16J is used. Video conferencing is an example of a collaborative software application. This functional block may be used in some of the embodiments described in FIG. 5C. Here, the functional blocks include a whiteboard function 70 for sharing a virtual whiteboard space, which can be overlaid on other video sources, for example, using a multi-layer function 82 configured to make see-through changes in multi-layer applications or to optimize 2D or 3D content for display on a multi-layer display. This function may capture visual content and optimize virtual depth to present data. This optimization minimizes focal depth information compared to the viewer's depth perception.

[0246] Additionally, annotation function 17 overlays annotations, geometric transformation function 19 aligns and maps the various captured images to the visual environment, and image processing function 69 performs image processing on the various layers of the displayed content. For example, one of the image processing functions may be a distortion correction function programmed to perform a geometric transformation on the user's image to correct for barrel or pincushion distortion, depth remapping, or automatic scene blurring / deblurring. In another example, a shared whiteboard space may be projected into a first focal plane and the user may be projected into a second focal plane to create a realistic virtual classroom. The geometric transformation function 19 automatically resizes objects based on which focal plane the content is in and based on the user's physical location relative to the webcam.

[0247] In some embodiments, the webcam may be part of a video camera system that captures the environment such that the captured content is displayed on a display system as part of the visual environment, such as a virtual classroom or virtual workspace. An object detection function may recognize and segment important objects in the scene, such as physical objects or physical whiteboards that are merged into the visual environment. Image processing function 69 and geometric transformation function 19 may operate on the environmental scene to geometrically warp objects in the scene to be superimposed on the visual environment. Based on the line of sight detected by another camera pointing towards the user, the display system may use neural radiance fields (NeRFs) to adjust the viewpoint of see-through components in the visual environment. An example of this is further described in FIG. 6J.

[0248] As another example, the whiteboard feature 70 allows a user to share a separate application or merge a separate application with a camera source, such as in an online lesson for an online course. The shared content may be a traditional sharing mechanism, or the shared content may be a dynamic mechanism where the content is dynamically transformed to adjust to the needs of the viewer. For example, the input to the whiteboard feature may be a data set of flight trajectories, and the feature is configured to plot those data into a visual trajectory that is overlaid on a multi-layered flight simulator.

[0249] In these embodiments, specific input sources are described, but any digital content can be input as a source. In some embodiments, the sources include other existing apps, existing websites, groups of websites. For example, an input to the virtual environment / UX immerser function block 16J can be a video conference call from an existing commercial software. Another example is that the game and world distortion engine function block 16G or the software engine / data assembler function block 16F can receive an existing game engine environment as input.

[0250] 6A-6J show flow charts of the example embodiments of FIGS. 5A-5J, respectively.

[0251] FIG. 6A shows a flow chart 601 corresponding to the functional blocks of FIG. 5A configured to generate interactive training videos. In step 83, the user's SLAM data is input to a pose estimation function 15A, which may also have a dictionary of poses 86 as input. The output is a classification of the user's pose. The software determines whether the user's pose matches the character's pose sufficiently in step 84. If the pose matches, the system outputs a first display content showing the training video as complete or allows the training video to continue in step 85A. If the pose does not match, the difference is calculated in a calculation block in step 87, and a second display content showing the highlighted part of the video is output in step 85B for the user to correct himself. A pose estimator may be generated by a feedforward neural network, and the difference between the user and the character may be calculated by classifying the poses in vector space using an encoder and the difference between them.

[0252] Figure 6B shows a flowchart 602 of an interactive video as described in Figure 5B. In this flowchart, the user's gaze is an input 83 to a visual focus estimation function 15B to estimate visual focus. The output is the position of the gaze on the display system. The software then makes a decision at step 84. If the focal object is available for sale, additional information about the object is displayed as first display content 85A. If the focal object is not available for sale, the display displays second display content 85B, which maintains the same video until the gaze changes.

[0253] Figure 6C shows a flow chart 603 highlighting the application of Figure 5C. A user gesture is a first input 83A to a gesture estimation function 15C1, which outputs the identified gesture to a first computation block 87A that can show a displayable gesture. At the same time, the camera system inputs a captured image as a second input 83B to a shape estimation function 15C2, which outputs information to a second computation block 87B that may combine the outputs of the two functions into a display content 86 that includes the displayable gesture. The estimated shape and the displayable gesture are combined to transform or distort the gesture before it is displayed.

[0254] FIG. 6D shows an example of scene analysis or action reporter flow chart 604 of FIG. 5D. Real-time video is input 83 to scene understanding analysis function 15D. This function compares and correlates frames of the video. This correlation may be feature based. The output in computation block 87 identifies which frames are related to a specified activity that may be predetermined by the user. This output is displayed as display content 85 consisting of a set of frames along with the original real-time video. Scene understanding analysis may be completed by CNN or region-based CNN (R-CNN), or by tree search.

[0255] FIG. 6E shows a flow chart 605 of the example embodiment described in FIG. 5E. The user code is input 83 to a first computation block 87A, which compiles the code and inputs the results to a merge function 15E. The compiled code is also analyzed independently for functionality or benchmark tests in a second computation block 87B, the results of which are also input to the merge function 15E. The merge function may compare the user code and benchmarks to existing code blocks, which may be stored in a library, or the merge function may generate new code using a generative trained transformer in an AI module. Another new code is then output as a merged code, which is then analyzed for functionality in a third computation block 87C. The resulting merged code and analysis are displayed as display content 85 along with the original code for the user to compare.

[0256] Figure 6F shows a flow chart 606 representing the functionality of the embodiment of Figure 5F. A database is input 83 to a UX analysis function 15F, which may have a data type library, dictionary 86. The data type library may contain information about the best mode of presentation, as well as various forms of data, file formats, and applications. The UX analysis outputs a proposed visualization of the database data to a calculation block 87, which is then displayed in the display system as display content 85.

[0257] FIG 6G shows a flow chart 607 associated with the embodiment described in FIG 5G configured for game engine or testing. A user inputs information or user gaze as input 83 to AI content generator function 15G. The AI ​​content generator may have a game engine dictionary 86 containing information about the game style, genre, characters, or game environment. The AI ​​content generator outputs new game modes or graphics to a calculation block 87 which is then visually displayed on the display system as display content 85.

[0258] FIG. 6H shows a flow chart 608 of the embodiment of FIG. 5H configured for use as website testing software. User input 83 is input to feature usage tracking function 15H1, which automatically updates the website features based on past usage and displays the results of calculation block 87 in the expanded display as first display content 85A. For example, the function may simply track the position of the cursor coordinates as a time series and then identify the location where the cursor spent the most time. The updated feature of the website may then be a feature from the set of features that is placed at the most likely cursor location. The expanded display content of the modified website may be next to the original website for the user to compare the changes. The output of the tracking function may be input to AI function 15H2, which suggests changes that are displayed as second display content 85B as an annotation layer over the original website display.

[0259] FIG. 6I shows a flow chart 609 of the example embodiment of FIG. 5I for a search and rescue operation. Real-time video is input to an object identification function 15I1 (83). This function may be a CNN or an R-CNN. The output of this function is to identify personnel in danger in a first computation block 87A and display that information on the display system as a first display content 85A. The method of display can be, for example, brightening the image of the personnel or annotating the personnel's location. The output of the object identification function is input to a scene understanding function 15I2, which, for example, analyzes the scene for specific hazards, and fire or electrical hazards may be the most dangerous. The output is an identification of those hazards in a second computation block 87B, which is then displayed on the display system as a second display content 85B. This output can include a procedure or ordering of which personnel should be rescued.

[0260] FIG. 6J shows a flow chart 610 of the example embodiment of FIG. 5J for a videoconferencing or ARVR application. A camera system captures an environment that may include a user. This information is input as a first input 83A to an object identification function 15J1, which identifies important or relevant objects in the environment. (The object identification function 15J1 may be a different object identification function in that it uses a CNN with a different set of weights compared to the object identification function of FIG. 6I). The identification may be related to a dictionary or lookup table, or the important objects may be pre-specified. The output is a superposition of an image of the environment onto the visual environment, as shown in computation block 87. The user's input, gaze, or SLAM data, along with the virtual environment input 83B, may be input as a second input to a NeRF function 15J2, which may be implemented as a fully connected deep neural network to calculate various perspectives of the visual environment. The results are then displayed on the display system as display content 85.

[0261] Figures 7A-7G depict different embodiments of Figure 2B showing various software applications that use predictive features to assist or affect a user experience. In some embodiments, the presently described software applications are created using the STW interface described in Figures 4A and 4B in addition to the methods of Figures 2A and 5A-5J.

[0262] FIG. 7A depicts a general “Funnel Expander” embodiment in which past actions and events are processed along with user input to generate predictions of various future actions and possibilities. In this embodiment, a user 1 is viewing a central display image 9, which further includes a past content display image 9A depicting content or information related to past usage, and a future content display image 9B depicting content or information related to future usage. The user may be viewing content via a local source combined with the display system 8. In addition to the general functions 15A, 15B, the AI ​​module 18 may receive user input 12 as input, in addition to past and current content, to generate an expanded visualization of future action possibilities 89. An infographic 22 may display past content in a useful manner.

[0263] Input to the feature can be current and past usage of any duration. In some embodiments, the feature is a recommendation engine, where the user or the user's history or profile determines the set action. Other features are probabilistic or time-dependent. Features including neural networks take as input user input to the system or sensor input. In some embodiments, the history of past actions is displayed as an infographic. In some embodiments, the infographic is an expandable tree graph where each branch is an aggregation of a set of common actions performed by the user. The trunk of the tree graph indicates the timestamp of those sets of actions, and the extent of each branch may correlate with the amount of time spent on each type of action.

[0264] In an embodiment using time delay as a function, the user is using a database, performing data entry, or analyzing numerical results of a simulation. The primary display content is a spreadsheet into which the user is entering data. The primary predicted activity displayed in a second layer or extended FoV adjacent to the primary image is ongoing data entry, since the most recent activity was the most recent data being entered. The software may predict what data should be entered, or the software may display an extended area of ​​the database or spreadsheet. The second most recent activity was opening a document, so the software predicts an instruction on the secondary display layer to save the database or spreadsheet, and anticipates opening a new document or closing the current document. The oldest action was using a different application, e.g., a simulation, to generate data. The third predicted action is to rerun the simulation to change parameters.

[0265] FIG. 7B illustrates an embodiment of a “probabilistic predictor” where predictions for various actions and probabilities are displayed according to weighted time decay or probabilistic factors. In this embodiment, a user is looking at a display image 9, which further includes past content display content 9A and future content display content 9B. Past and current usage is input to a function 15, which uses a probability distribution 93, for example, to calculate the most likely next actions and displays them accordingly. The most likely future action 91 is displayed centrally and most prominently in the most prominent extension of the display content. A moderately likely future action 92 is displayed less prominently in a moderately prominent extension of the display content, and the least likely future action 93 is displayed furthest away and least prominently. In some embodiments, the display content is displayed as a multi-layered display or floating graphics, with the most likely content being the brightest or closest to the user.

[0266] A user can directly input information via input devices or sensors 13, and data from this information may rearrange or dynamically modify actions. In some embodiments, sensors capture information about the user or the environment and relay that information to the display system to aid in predictive capabilities.

[0267] A probabilistic method may be formulated as follows: Encode all user actions into a vector space x. This may be for a particular application or for a set of applications. In some embodiments, the non-zero vectors are sparse in the basis so that new actions can be added. Then define a probability density function. In some embodiments, the probability density function is a bell curve (Gaussian function), a Lorentzian function, or a Cauchy function. These functions may be discretized for a distinct set of actions. In some embodiments, the probability density function is defined by certain constraints, such as maintaining a certain standard deviation, skew, kurtosis, or set of moments or central moments. Or alternatively, a characteristic function, moment generating function, cumulative function is given. In some embodiments, a probability characteristic is given for the various actions x belonging to the vector space x. i or by the relative frequency of the user's actions during the period the system is being calibrated.

[0268] In some embodiments, the sequence of actions is static in some sense, e.g., broadly static, strictly static, or gradually static, hi some embodiments, the system is not static, e.g., dependent on the time of day or other external factors.

[0269] A second set of actions is encoded in a second vector space y. In some embodiments, there are more than two sets of actions, for example three or four or nine sets. If a user selects a particular action x i If a display system is used for j Calculate all conditional probabilities for each p ij = P(y j |x i ) The conditional probability P(A|B) of two events A and B is the probability that A occurs with the condition or constraint that B has occurred. Conditional probability can be thought of as the ratio of the probability P(A and B) that both A and B occur to the probability P(B) that B occurs: P(A|B) = P(A and B) / P(B) The above value p ij determines the action with the highest probability, the second highest probability, or some other indicator. The display system then displays those possible actions in a set of secondary virtual displays or display layers. In some embodiments, the method for predicting the user's actions uses over-prediction, time series analysis, or other sequence analysis.

[0270] In some embodiments, as shown in FIG. 7B, a user 1 is interacting with a social media platform. A central display 9 displays a landing page. Based on the user's history and the history of multiple users with the display system or application itself, a probabilistic function determines that the user is most likely to scroll through a series of updates. The most likely future action 91 is displayed centrally and most prominently in the most prominent extension of the display content. This content then displays an extended update or scrolling feed. In an adjacent window, a moderately likely future action 92 is displayed less prominently in a moderately prominent extension of the display content. Finally, a least likely future action 93 is displayed furthest away and least prominently. This content may include clicking on a marketing campaign. As the user interacts with the social media platform, the probability distribution is updated and the display content is repositioned. Various sensors 13 may capture information about the user. The user may bring any proposed content into the window using any input means. In some embodiments, the predicted action corresponds to switching to a different application.

[0271] Predictive algorithms use data about various possible user actions, events that contain metadata about productivity, success / failure, and user satisfaction. For example, a user who first begins navigating a social media site is most likely to click on an ad and purchase an item, while the second most likely event is responding to a message. 11 = 0.8 and p 21 Let x1 be a navigation to a social media site, y1 be a click on an ad, and y2 be the event of responding to a message, such that y = 0.5. In this scenario, the center secondary display displays content related to the ad, and the second secondary display displays content related to responding to the message. However, 11 Metadata about the app indicates that clicking on an ad led to excessive withdrawal fees in a budget monitoring app. Therefore, the display system 11 may be reduced to less than 0.5, for example to 0.4, or the display system may include a warning message in the display content.

[0272] FIG. 7C depicts a “dynamic prioritizer” embodiment in which different options and possibilities are displayed in different tiers based on priority criteria P1, P2, P3 based on user focus, time of day, productivity style, metadata, or environmental factors. Content deemed to have the highest priority (P1) is displayed in the central primary screen 9 for the user 1 to see, while content of second priority (P2) is automatically pulled to the FoV 2D extension 25 as a reminder. Some of the priorities may be organized in the multi-tiered display 11 such that the highest priority is closer to the user. Suggestions that would normally be made by the user are displayed as a third priority (P3), for example as an edge display 53. The distance of the content relative to the central display is an indication of the priority and / or importance of the content. In some embodiments, this distance is time-dependent and depends on the user history. In some embodiments, sensors 13 capture information about the user 1 or the environment and relay that information to the display system to aid in predictive capabilities. The user has the ability to ignore certain priorities, indicate reminders, or perform recommended tasks by directly inputting information into the control features. Priority-based embodiments may be generated by identifying or comparing items listed on a calendar or digital list. Or, if the embodiment includes user input and time, the embodiment may track the duration a task requires at different times according to different users' requirements, for example, and suggest a task when it was completed most quickly in the past.

[0273] In some embodiments, the content of the display system is configured for productivity. A user 1 is interacting with the display system at a particular time, and the primary priority action displayed in the central display 9 is to answer an email. Based on the time of day, the software senses that a second action P2 is high priority due to the user's productivity level at that time with that second action. In some embodiments, the next priority P2 is displayed as a FoV 2D extension 25 based on deadlines listed in the calendar. A third priority P3 is to monitor personal finances such as bills, investment accounts, taxes, etc., all of which appear in the edge display 53 as possible actions. In some embodiments, priority P3 is a secondary layer in the multi-layer display 11 so that the user can be aware of priority P3 without having to focus their eyes directly on priority P3, i.e., they can maintain priority P3 in a peripheral location.

[0274] In some embodiments, the different priorities may all relate to a single task: for example, a central priority may include making important financial transactions, a second priority may monitor cash flow for the results of those transactions so that the software program suggests changes or other transactions, and a third priority may represent a set of long-term financial goals, such as increasing savings for a down payment on a house, retirement activities, or travel plans.

[0275] The display system may arrange non-pertinent activities into different dimensions. For example, financial priorities may all be displayed in a horizontal extension. A displayed image containing a mortgage payment may include multiple depth layers, including annotations regarding home improvements, needed repairs, or important weather alerts. The arrangement may change dynamically based on user input or sensory data.

[0276] In some embodiments, the priorities P1, P2, ... are recommendations based on a recommendation engine that takes a user profile as input and outputs various recommended activities. The recommended actions may be within a single software application (e.g., displaying all possible digital library books related to the user's reading history) or the recommended actions may span multiple apps (e.g., the engine recommends multiple chat streams across different social media platforms based on the user's history of using chat features within a particular social media app).

[0277] FIG. 7D depicts an embodiment of a “parallel search recommender” in which predictions and recommendations may be made within an application or across multiple applications. In some embodiments, predictions and recommendations may be based on a vertical search engine function. A user 1 is looking at a central display image 9, and multiple vertical search engines are generated within multiple display images based on the current action or based on the user's query. For example, the user inputs a query into the vertical search engine function 94. In some embodiments, the display images are arranged in a multi-layered display 11 or arranged as columns in a vertically expanded FoV. As the search progresses within a column, the search dynamically updates based on the user's current action or query, but also pays attention to other possible searches that may be useful and presents those results in another set of display images. The data retrieved in the first search is then input into the function 15, which pays attention to keywords, for example by using a self-attention mechanism, and then uses that information as a new query for a second search, which may be displayed in the second multi-layered display 11. The functional relationship that one search engine uses to generate another may be a transformer that looks at various sections or correlations in the first search.

[0278] In some embodiments, a user is performing a literature search on a research topic. A primary search is initiated by the user using keywords A, B, and C. A vertical search appears in a first set of virtual display images. A software mechanism actively scans the search results to discover a new keyword D. A second set of virtual display images then reports search results for D alone or for a combination of A-D. In some embodiments, the user has limited the search parameters to scientific sources and journals, but the software detects phrases that indicate the patent history of the first keyword and displays the prior art in a second search. After analysis of the diagrams of the first two vertical searches, a third search may display various downloadable executable files that can aid in numerical simulation or quantitative analysis of the desired research topic.

[0279] Vertical search engines may use standard vertical search algorithms (e.g., crawling, indexing, and ranking), and object identification algorithms may be used to identify keywords or phrases to initiate the next search.

[0280] FIG. 7E depicts an embodiment of an "avatar-assisted predictor" in which a virtual assistant avatar 23 is displayed in the FoV 2D extension 25 when interacting with a user 1 looking at a central display image 9 to respond to the user's commands and perform various task functions 15, such as answering emails, drawing sketches or designs, chatting, and taking notes. Various tasks, predictions, and recommendations available to the user are displayed as commands issued to the virtual assistant. In some embodiments, the virtual avatar is not always directly visible to the user, or the virtual avatar is invoked by the user from a voice command.

[0281] In an embodiment, the virtual avatar 23 is assisting the user with a secondary task to help the user complete a primary goal. For example, the user is generating a document that requires text, figures, and references. User 1 has generated the primary text content and input the basic parameters of the figures (figure size, resolution, format) into the avatar system. In response, the avatar starts to edit a set of image files and then has the authority to incorporate these files into the image using the API. The avatar also analyzes the image content itself and extracts words that describe the image based on the transformer mechanism. These words become keywords in a web search that are presented to the user as alternatives or improved figures to help improve the final product.

[0282] In some embodiments, user permissions are defined by avatar-controlled subsections 23A of the display content, where the avatar automatically monitors the content in certain windows of the display and allows the user to interact with it by dragging elements into or out of those subsections. This helps to grant or revoke permissions to the avatar in real time, with certain content dynamically asserting which functions the avatar should prioritize. In an embodiment, a user may drag an image into a subsection, thereby indicating that the avatar should engage in image processing techniques, while if a folder of text documents is dragged into the subsection, the avatar interprets this as performing a literature search and building a bibliography.

[0283] In embodiments, a user is analyzing the results of a simulation, with the avatar functionality assisting in the analysis by comparing the results to known results, dynamic search results, or original input parameters. For example, the results of the simulation may include graphs or images that the avatar functionality processes for non-obvious correlations with the input data, and the avatar may suggest that the results are physically valid or that a technical error occurred in the simulation.

[0284] In some embodiments, the avatar assistant may be a terminal for the user to input text or graphics, and the avatar assistant may continually trigger subsequent questions based on the input. For example, the user may input an image of a chair, and the avatar assistant may first generate and display the question "What is this?". The avatar assistant may then provide a series of possible answers below this content: "This is a piece of furniture," "This is brown," "This is an object made of wood." Then, below this series of answers, there is a tree of further questions that depend on the initial response. The user may interrupt, direct, or guide the questions and answers generated by the avatar at any time. The progression of questions and answers may depend on the user's history or the user's settings.

[0285] In embodiments, multiple avatar assistants may be influencing derived content simultaneously, for example, an avatar assistant may be a chatbot in a help center, and users may monitor the messaging of the avatar assistant and influence the results in real time.

[0286] 7F depicts an embodiment of an "event-triggered predictor" in which different predictions and recommendations are displayed in different event layers E1, E2, E3 based on the user clicking on events or clicking during an interaction with a user input 12, such as a mouse click, which dynamically derives different predictions and probabilities. Events can be generated automatically by triggers in the video or in another software application, or events may be triggered when a particular combination of software applications is used in a particular way.

[0287] For example, in some embodiments, user 1 is watching a video tutorial on painting techniques in the central display image 9, and after a while, starts performing image processing of the video. During the tutorial, when the user plays that part of the video, a particular brush stroke is detected by the multiple output function 15, and the user's click is input to this function, a similar tutorial on that brush stroke is found in another tutorial E1, the user may click on the image of the brush so that an advertisement for a similar graphic design product is displayed in E2, and while the user pauses the video to see the final result of the tutorial, the next venue for showcasing the finished product is displayed with contact information or an online form for follow-up questions to the instructor is displayed in E3. Events may be displayed in the FoV 2D extension 25. Or events may be displayed in a multi-layered display. In some embodiments, machine learning algorithms may display various alternative techniques or methods for achieving a similar effect in other display images.

[0288] In another embodiment, a user is playing a video game. The user navigates the game and reaches a certain milestone, and a first event E1 may be a selection of which task to complete in the next step of the game. A second event may be the user scrolling over a particular area in the video game environment, which triggers a display event E2 that is a hidden feature of the game. Finally, a third event may be triggered when the user pauses the game or clicks a link, and the display content of E3 is a marketing add for a bonus feature, a sequel to the game, or other entertainment option. In any embodiment, the event-based display content may be influenced by user history.

[0289] In various embodiments, the display content may be arranged in any manner. In embodiments, the display content may be arranged horizontally, for example, to create a visual scroll or a visual belt. The user may provide input via gaze or gestures such that the visual scroll may be dynamically rotated, and as the user focuses on a display content of interest, that display content is moved to a central display position, and other display content is shifted continuously. For example, an event-based predictive display may display three augmented displays of events E1, E2, and E3, such that E1 is located on the left, E2 is located in the center, and E3 is located on the right. When the user focuses the gaze on E1, E1 is shifted rightward toward the center, E2 is shifted rightward toward the right, and E3 is moved to a left position. The visual scroll may be configured to display a single event or action at various past or future times. This is a "temporal scroll." For example, the visual scroll may include a series of possible time-dependent actions. The visual scroll may be spatially separated such that various aspects of an action or multiple different actions related to a particular application are displayed separately. The visual scrolling may be spatiotemporally separated such that the possible content is a combination of temporally scrolled actions or spatially separated content.

[0290] Fig. 7G represents an embodiment of a "Parametric visualizer", which takes into account the mechanism of a parametric visualizer, so that a virtual continuum of possibilities can be simultaneously or easily seen. An example of this embodiment is the following: A user 1 is watching a movie in a central display image 9. The content of the movie is fed to a neural network 77 and / or an AI module 18, which generates annotations or alternative results for the current scene and displays them in the augmented part of the augmented display system.

[0291] A user may also use generic input device 12 to input information to parameterizer function 15, which may receive library 45 as input. The parameterizer allows the user to input preferences, user history or profile, number and scope of annotations, or other constraints to the AI ​​and ML functions. The output P is a set of parameters for tuning the AI / ML functions.

[0292] In this embodiment, for example, one of the parameterizations results in Profile A, which generates a set 11 of multi-layered display content for a movie, where a first set of the multi-layered display content set 11 represents annotations on the visual content along with more detailed annotations and visual content. A second set is muted and quieter and includes less information on the associated soundtrack. A second form, Profile B, may have the relative importance of visual information versus sound reversed. Soundtrack information is prominently displayed along with annotations as floating graphics 24, and some basic information on the visual content is displayed as edge display 53.

[0293] In another example, a first user may be interested in the scientific details of a movie and has set the display content's "light" setting parameters so that all possible annotations show the scientific and technical details of some of the objects or movements in the movie. A second user is an interior designer and sets the display parameters to "strong" so that whenever a movie scene is an interior room scene, all annotations of furniture, household items, and other items in the scene include saleability, price, availability, or vendor location. This may be described as a "display equalizer" function, where the output display is balanced according to various settings.

[0294] 8A-8D illustrate different processes for predicting future user actions in some of the embodiments described in FIGS. 7A-7G.

[0295] FIG. 8A illustrates a process associated with the embodiment of FIG. 7A. User input 95 includes a user history, and the user's current action 97 is input to a predictor function 96, which generates a display of a predicted or possible action 99. The user then makes a decision 98 regarding which action to perform, which triggers a next action 100. The current action 97 is then incorporated into the user history for the next prediction. FIG. 8B illustrates a process associated with the embodiment of FIG. 7B. User input 95, user history, and current action 97 are input to a probabilistic correlation predictor 96, which generates a display of a possible or predicted action 99. The user makes a decision 98 and performs a next action 100. The current action 97 is then incorporated into the user history for the next prediction.

[0296] Figure 8C illustrates a process related to the embodiment of Figure 7C. In Figure 8C, the user input 95 is a user history, which is fed to a priority ranking function 101 and then to a priority correlator 102. Based on the priority, the possible actions are ranked and a display result 99 displays the preferred action. The user makes a decision 98 as to which should be the next action 100 of the current action. In some embodiments, the priority correlator is any type of neural network, including a feedforward network, an RNN, an LSTM, an attention-based transformer, or a combination thereof.

[0297] Figure 8D illustrates a process related to the embodiment of Figure 7D. User input 95 is a search query that is fed into a first vertical search engine 103A. The results are displayed in a first display 99A. Data from the first search is also input into a transformer 104, which notes and identifies keywords for input into a second vertical search engine 103B. The output of this second search is displayed as second display results 99B. In some embodiments, more than two vertical search engines are used. In some embodiments, the output of a later search may be used to modify an earlier search.

[0298] Figure 8E depicts the process of the embodiment of Figure 7E configured to include an avatar assistant. An avatar with authority 105A generatively displays display results 106 corresponding to possible or predicted actions. The given authority allows the avatar to perform an action when the user moves from a current action 97 to a next action 100. The next action may change the authority 105B of the avatar assistant, affecting the next iteration of the task completed by the avatar assistant.

[0299] Figure 8F illustrates the process of the embodiment of Figure 7F. A user's current action 97 is detected by an event-based trigger 107, which generates display results 99 corresponding to various actions or other content. The user makes a decision 98 as to what the next action 100 should be.

[0300] Finally, Fig. 8G illustrates the process of the embodiment of Fig. 7G. User inputs 95 (parameter settings) are input to a parameterizer 108, which generates parameters P that determine the strength and content of the AI ​​module 18 or neural network 77. The generative output provides a display result 98 that corresponds to the annotation layer.

[0301] 9A and 9B illustrate various machine learning algorithms and methods that support the predictive and generative software of the present disclosure.

[0302] 9A shows a general neural network pipeline, where input 109 is the user history and input, plus a bag of possible actions, serve as input to a machine learning architecture such as a neural network. The neural network 110 outputs a set of possible actions 111. The neural network can include training data derived from the long-term history of a single user, multiple users on a display system, or a combination of these.

[0303] In some embodiments, the neural network uses a dictionary that is learned against training data, which may come from a local display system and work environment and a unique set of users. In some embodiments, the dictionary and learning occurs based on training data from distributed users.

[0304] FIG. 9B shows a more specific example of a recurrent neural network configured as a long short-term memory (LSTM) neural network. In this figure, a bag of user input, history, and actions are input to the LSTM (109). This input is fed to an LSTM 110 that includes activation functions g1, g2, g3, g4, and g5. This input and the output value from the previous cell are sent through a neural network layer that includes activation functions g1, g2, and g3. These values ​​are then combined with the state of the previous cell through multiplication and addition operations. A possible action is generated (111A). After the neural network includes activation function g4, the state of the cell is acted upon by the neural network and activation layer g5 and combined with the current action and the user input. The activation function can be any activation function. In some embodiments, the activation function is a sigmoid function or a hyperbolic tangent function. The result is fed to the next cell iteration 113A of the LSTM along with the hidden layer information 114A, which generates an output for the second possible action in the sequence 111B, which is fed to the next cell iteration 113B along with the updated hidden layer information 114B. A third action 111C is generated, and so on. In some embodiments, the user's avatar has the authority to perform the predicted actions or a subset of them 112A, 112B, 112C. The activation function can be a standard sigmoid or tangent function. In some embodiments, the activation function is user defined.

[0305] In some embodiments, various neural networks are implemented, including traditional neural networks, simplified RNNs, GRUs, CNNs, especially for image / object detection recommendations that also use user input in various applications. In some embodiments, the architectures are one-to-one, one-to-many, many-to-one (as in a classifier), or many-to-many.

[0306] 10A and 10B show the use of attention in a transformer architecture to derive predicted actions, classify past actions, or transform a set of actions into a new set of actions, configured for other applications. In FIG. 10A, the user's actions and history are input into the pipeline (115), transformed by positional (continuous) embedding 115A, and input into the encoder block 116. The input data is operated on by a linear layer to generate a query Q, a key K, and a value V. The encoding block combines Q, K, and V and normalizes them, for example via softmax. A feedforward layer operates on the data to generate an attention matrix A. In some embodiments, residual data is routed around the elements in the encoding block. In some embodiments, multiple encoding blocks operate in parallel. The data is then sent to a decoding block 117, which includes a multi-head attention block, which combines and normalizes the data matrices, and operates on the data using a feedforward layer. In some embodiments, there is a residual element or a masking block. The output 118 is a search result set of generative actions / avatar responses / potential actions. In some embodiments, a classification block 118A identifies the type of action currently being used. In some embodiments, the action is automatically performed by the user's avatar.

[0307] In some embodiments there are multiple transformer heads, or multiple stages of attention, or multiple stacks of decoders and encoders. In any embodiment, feedback mechanisms, masks, and position encoders may all be included.

[0308] An example of an attention matrix 119 is shown in FIG. 10B. Each row corresponds to an input (action from the user history) 115, and each column corresponds to a possible output action 118. The grayscale values ​​correspond to the correlation between the input and output actions. For example, input action 1 is very strongly correlated with output action 1 (shaded in white), is moderately strongly correlated with output actions 2 and 3 (shaded in gray), and is very weakly correlated with output action N (shaded in black). In this way, the set of output actions is determined by both the set of input actions and the order in which those actions occur.

[0309] 11A-11G depict multiple embodiments of the new single user use case.

[0310] FIG. 11A illustrates an exemplary display system "Intelligent Expander," which includes an extended FoV and floating graphics configured for use in active content generation. In this embodiment, the user experiences dynamic browsing of the text using predictive features. At time t1, the central display 9A displays the text. An analyzer in the object detector function 81 detects keywords and phrases in the text and identifies equations and diagrams, which are displayed in a separate display image in the 2D extension 25A. The separate display image may be part of a multi-layer display, or the separate display image may be an extended field of view image. At time t2, the separate display content is automatically updated such that different secondary content is identified in the secondary 2D extension 25B as different content is displayed in the primary display image 9B. In either case, the floating graphics 24 displays content from the previous time. For example, the primary display may include the text "As shown in FIG. 1," and the floating graphics may automatically display the portion of the primary text that includes the image of "FIG. 1." A separate annotation function 17 may annotate or add further information about the content displayed in the expanded window, for example, the separate annotation function 17 may display relevant figures or mathematical inferences made from the displayed figures and equations.

[0311] In embodiments, functionality may highlight portions of the central display or annotate the extended display content to highlight relationships between their various contents.

[0312] FIG. 11B shows a variation of the display of FIG. 11A, a "logical inference expander." A primary display layer 9 displays primary content, such as text. Within the text are various statements that are automatically detected by software as logical statements. In some embodiments, a secondary display image, which is a FoV 2D extension 25 or multi-layer display, displays the logical consequences of the detected statements as generated by logic function 74. For example, if the primary text states, "Substitute Equation 1 into Equation 2," then both Equations 1 and 2 are displayed in the secondary image, and the generated result of the substitution of Equation 1 into Equation 2 is also displayed in the secondary image. The logic function that controls the secondary display panel has a pre-programmed mathematical logic structure for calculating the result.

[0313] In some embodiments, the logical conclusions are dictated by the user. The user may query the text using voice input, various commands, or questions, including, for example, "Can you prove Equation 10?" or "Are Equations 11 and 12 simultaneously true, i.e., mutually consistent?" or "What are the differentiable properties of the expression on the left hand side of Equation 9?" The AI ​​program may answer the questions based on various mathematical libraries stored in the AI ​​program. For example, the AI ​​program may then parse Equation 9 to identify the desired expression on the left hand side, analyze the expression for connectivity, smoothness, differentiability, or other geometric or topological characteristics, and output the results in secondary floating graphics or as an annotation overlay.

[0314] The embodiment shown in FIG. 11C is a "smart format integrator" and functions as an application merger configured to edit or create content. In some embodiments, a user generates content that is a text document. At time t1, the user generates some text information in the primary display 9A, and some source information, for example used for referencing, is displayed in the secondary display as a FoV 2D extension 25A. In some embodiments, the secondary display is a floating graphic or part of a multi-layered display image. At time t1, the user performs an action to merge the content from the two windows using the merge function 72. Based on the learned understanding of the two contents, the software automatically formats the source information in the secondary image and generates a formatted bibliography in the primary display image. In some embodiments, the user's action includes clicking and dragging a mouse, a keystroke, a voice command, or a gesture. In some embodiments, the merge is suggested by a predictive model and the user confirms or rejects the suggestion. In some embodiments, the merge is performed automatically based on the user's permissions of the predictive avatar. As a result, at time t2, the primary display image 9B is modified with the source information integrated. The FoV 2D extension 25B may not be changed at time t2.

[0315] In some embodiments, other suggestions about the source material are made based on the library and on an analysis of what text is being written. In embodiments, the suggestions may be a thesis, a hypothesis, or a set of open problems based on the input text.

[0316] FIG. 11D shows an embodiment of an "intelligent programming recommender" that reacts to user context. In some embodiments, the user generates content, which is a computer program. In some embodiments, the content is a multimedia product or an artistic or entertainment product. The central display 9 is the user's primary workspace, and the display system generates two virtual side images as FoV 2D extensions 54. In some embodiments, the side images are floating graphics or edge display images. The left display image displays the user's actions. The right image displays the actions that are suggested based on the user history and are performed by the AI ​​module. The camera 14 can be optionally used to function as a gesture sensor. In some embodiments, the suggested display content is an alternative way to generate the same result that the user is trying to generate. In some embodiments, the suggested content is an optimized version of the content generated by the user. The user inputs information about the desired result by gesture recognition via the camera. In some embodiments, the user uses a keyboard or mouse or voice commands. This software application may be used in a variety of ways, including programming, art, a / v or multimedia production, architecture, 3D design and engineering, and game design.

[0317] The mechanisms of the AI ​​software may display other alternatives. For example, in a game design module, a user creates a character for a game that is generated by speaking or entering text into prompts. The AI ​​software generates the character and suggests a story for the character, other characteristics or traits that the character may need to carry out the story, and supporting characters that may interact with the character.

[0318] FIG. 11E shows an embodiment of a "pose encoder with AI feedback" in which a user uses a display system with a camera. In some embodiments, the software application is a chatbot, natural language processing, predictive text, or chat prompts using generative trained transformers. User 1 inputs data into a workstation, and the virtual system displays the resulting content in the central display image 9A at time t1. The camera 9 captures gestures, microgestures, facial expressions, and postures related to the user, and the resulting display 9A incorporates those physical features into the result. In some embodiments, the results may appear as annotation information in the floating graphics 24A. Using the same requested interaction, the software uses learned data on posture or facial expressions to generate a different result (floating graphics 2B and main display image content 9B) at t2 compared to the result at t1. The annotations explain how the person's physical features were used and can predict alternative results. In the context of videoconferencing software, the display may have other features to automatically, computationally remap poor posture, eye averting, or highlight other negative social cues for more positive display content.

[0319] FIG. 11F shows an embodiment of the display system "Global graphics intelligence profiler" configured as a multi-layer display 11 with floating graphics 25 objects. In some embodiments, the use case includes medical imaging. On the multi-layer display 11, the user views a set of images of the object derived from different modalities. For example, in some embodiments including medical imaging, the different layers are CT scans, MRIs, PET scans, X-rays, or photographs. These images are input to the neural network 77 (along with dictionary data for all modalities) to generate a final layer, which is an annotation layer indicating regions of interest or confirmation of the target question. A secondary floating image 24 layer takes the annotation layer as input and generates, via the AI ​​module 18, an annotation description, a diagnosis or prognosis, or other features of the annotation not detailed in the annotation layer.

[0320] In some embodiments, the source data is AI-generated and configured for the training module. In some embodiments, the display content is geometrically transformed using neural radiance fields, and the AI ​​software suggests different displays for interactive training and suggested lessons. In some embodiments, the AI ​​mechanism is controlled by a second user, who acts as a trainer or educator and dictates which images or annotations are highlighted based on the program goals.

[0321] In some embodiments, as shown in FIG. 11G "Multi-Layer Geometric Warper," different layers may correspond to a set of pose assets for a character in a video game that share a common anchor or target point, and the floating graphics are warped versions of the assets based on the common anchor. In this embodiment, the function is a geometric transformation function that can warp, for example, the character's pose or stride. One of the layers of the multi-layer image 11, for example the back layer, may contain target graphics against which the character's appearance must be warped. The target graphics may be a scene or environment that contains certain landmarks or anchor points, so that the warping is dynamically adjusted based on the anchor points. In some embodiments, function 15 analyzes the scene and displays a subset of the character's pose assets. These pose assets are then input to a geometric transformation function 19 to generate the warped poses in the floating graphics 24. The type of warping is arbitrary. The distortion and geometric transformation can be implemented using Generative-Adversarial-Networks (GANs), where the anchors can act as "seeds" for the GAN.

[0322] FIG. 12 shows a flow chart of the software mechanism of tandem computing. The multiple input streams include the Internet 6A, local sources 5, and general remote sources 6. Examples include cloud servers, local workstations, daisy-chained workstations, distributed networks, and edge devices. The data is optionally manipulated by various functions 15 and then merged in a merging block 120. The resulting merged data may be manipulated by another function 15 and is analyzed by a display system, which may include a neural network, in a content analysis block 121. A second content analysis block 122 understands the content in terms of the user's actions or tasks, or the current context of the application being used. In an assembly block 123, the content is assembled and then displayed on a display system 124. In some embodiments, predictive actions or suggestions are included in the content. In a monitoring block 125, the user's inputs, actions, and history are monitored and fed back to the analysis unit. This feedback may be present in an updated learning dictionary 126 for the machine learning algorithm before being directed to the analysis block.

[0323] In some embodiments, the merge is a non-linear or multi-dimensional function of the input streams.

[0324] User actions and feedback include time delays for execution of actions, choice of decisions. Suggested content can be automatic, depending on the permissions given to the software. In some embodiments, suggestions invoke sub-applications or forms of auto-completion or online data entry requests. In some embodiments, suggestions affect the user's health, for example, by suggesting a break, switching tasks, or maintaining focus based on the user's health data.

[0325] 13A-13I illustrate various embodiments using the tandem computing method, including edge computing devices and distributed networks.

[0326] FIG. 13A shows a general tandem computing environment "Tandem Expanded Display System." This display system generates N display images, including a general central display 9 and a FoV 2D extension 25. In some embodiments, there are more than four panels or less than two. In some embodiments, the configuration is a multi-layer display panel. In FIG. 13A, there are multiple sources of display content. One of the sources is a local source 5 connected to the display system 7, which generates display content for the entire central display image 9. The other source is a remote source 6, which generates content for the side windows. The right window is fully remotely sourced (10), i.e., its entire display image is attributable to a remote source. In the left window, a portion 10A is generated by the remote source, and the original local source generates the remaining portion. In any embodiment, functions 15, e.g., F1, F2, ..., FN, may be used to manipulate the display content. All sources may communicate with each other directly or via various daisy-chained configurations.

[0327] The embodiment of FIG. 13B illustrates an exemplary tandem computer “AI Sensory Network Integrator” configured for use in remote operation, robotic control, or quality control. One display area of ​​the multi-layer display 11 displays video from a remote source 6, for example, of a remote controlled robot used in manufacturing. An array of distributed remote sensors 13 at a manufacturing facility, vendor site, or any other location remotely generates front layer content that highlights the robot's environment. In some embodiments, the remote sensors show information about the robot, such as operating temperature or range limits. In some embodiments, the remote sensors show information about the product the robot manipulates, such as quality control sensing, random changes, stress and strain, or thermal or mechanical stability of the product or robot. Both of the dual layer multi-layer display images are input to a geometric transformation function 19, which is used to overlay the sensory data onto the video. For example, the sensory data may be a set of temperature sensors, and the geometric transformation function uses a back propagation algorithm to map the temperature profile of the device. The second function is an AI module that takes sensory data and video as inputs and outputs annotations 17 to floating graphics 24. The annotations provide descriptive content about the robot, or predict part failures, or suggest changes to operation, or suggest contacting the vendor for support. For example, the AI ​​module may generate content based on a layered temperature profile over the video that indicates that a part of the robot is overheating, or is likely to overheat soon unless intervention is performed.

[0328] FIG. 13C shows a tandem computing embodiment "Multilayer smart teleconferencer" for videoconferencing in a virtual reality environment. A multilayer display 11 displays, in each layer, a particular user in a shared virtual reality environment. One of the layers is a remotely provided image 10 of a person. A geometric transformation function 19 operates on objects and people in the scene, which are assigned different depths and positions of the users. For example, an object detection subroutine detects the size of a person in one layer, which depends on the person's position relative to the camera, and then scales the image of a second person so that they are both similarly sized.

[0329] In some embodiments, a secondary floating graphics layer 55 provides annotations and feedback to each user based on their facial expression, gaze, tone, or head position so that the user can modify their actions based on the suggested feedback. In some embodiments, the AI ​​module 19 evaluates the conversation and the multiple users in the conversation to influence the conversation. For example, a facial expression analyzer function may evaluate the mood of the collaborative users and indicate whether the tone of the conversation should be serious, formal, informal, or light-hearted. The embodiments may be combined together. For example, the "multi-tier smart teleconferencing" may include the "posture encoder with AI feedback" of FIG. 11E as part of its operation.

[0330] The embodiment of FIG. 13D illustrates the use case of a virtual extended FoV display using multi-layer tandem computing, configured for use as a flight simulator, gaming experience, training experience, or weather / climate monitor. In this "multi-focal intelligent simulator," images 10 are remotely fed into multiple layers of a multi-layer display 11. These images may be flight simulator images. The input stream passes through a multi-layer optimizer to optimize the content for the display to maximize depth perception. An AI module 18 takes the simulation images and provides annotations 17 on the front layer for the user (trainee) to view. Annotations may be suggestions for next actions, hazards in the simulated environment, warnings, predicted alternatives, predicted movements, or future dynamics of the environment. Annotation layers may include images of metrology clusters and measuring instruments. In some embodiments, a central display area may display the simulation, and based on the content of the central, the extended display may display AI-generated predicted outcomes of choices the user can make.

[0331] In some embodiments, the environment is a real-time image, such as generated by a camera located on an existing aircraft, which is then used for flight simulation or observation. Or the environment may be a real-time image of a remotely controlled vehicle controlled by a user in a teleoperated environment. In some embodiments, the annotation layer displays a predicted scene or predicted motion at a future time point based on a delay that encompasses the latency.

[0332] In some embodiments, the augmented view of FIG. 13D is configured for use as a virtual tour of an environment, such as a museum, with the annotation layer providing annotations of items within the environment.

[0333] In some embodiments, a sensor array in communication with the display system collects SLAM information about the user to affect or display individual parts of the visual environment. For example, in a remote operation center, the SLAM information is input to a function to geometrically modify the perspective of the virtual content to obtain a perspective of a precise angle, which is a true perspective without deformations that arise from the sensor, camera, or communication channel. Or, for example, head tracking and gaze may be used to detect where the user is looking and modify that part of the displayed content or expand that area. In some embodiments, the AI ​​module is replaced by or influenced by a trainer or instructor who can provide instructions as annotations. The instructor may be visible in the user's surroundings so that the visual environment becomes immersive and the instructor and the user have the feeling of being in the same place. In this embodiment, this immersion allows the user to experience a more realistic visual environment. In some embodiments, head tracking or gaze may be input to a geometric transformation function that modifies the simulated environment to mimic a shift in viewpoint.

[0334] FIG. 13E shows an embodiment of a "tandem intelligent content generator" in which a tandem display system is configured. The multi-layered display 11 includes a primary display image 9 generated by a local source, for example showing scientific data or graphics. If the local source, including the display system 6, is a low-bandwidth source, the graphics or data will be low resolution or limited in other ways, such as FoV, temporal resolution, feature depiction, or spatial resolution. The user provides input, including cursor input, which moves a cursor position 127 on the display image. The position of the cursor is detected and a nearby portion of the display content is transmitted to the remote source 6, which relays further information about the nearby environment. The remote source may generate desired annotations in the remotely supplied image 10 using annotation functionality 17. In some embodiments, no cursor is used, and instead the content of interest is determined by the gaze position generated by an eye-tracking input device. The additional information is displayed in a secondary display image in the multi-layered display 11. This display may be a FoV 2D augmented display or floating graphics. The additional information can be a high-resolution or otherwise enhanced image of the environment. In some embodiments, the additional information is explanatory text, additional drawings or schematics, images of similar objects (e.g., as used in image web searches). In some embodiments, the additional information is graphs or simple text that appear in the edge-display image.

[0335] In some embodiments, a graphics function 39 generates dynamic image refinement or upscaling, drawn from high performance computational sources. In some embodiments, different functional blocks allow the user to select different classes of modifications, annotations, explanations, or suggestions. Annotation and graphics functions may be parameterized by the user's profile or history.

[0336] The embodiment of FIG. 13F illustrates the use of a tandem computer “time-delay AI predictor / differentiator” in which a multi-layer display 11 generates multiple frames of video or a time-lapse sequence of images of an object. In this embodiment, an image 10A is generated that is fed to a first remote and an image 10B is generated that is fed to a second remote, the second image being a time-delayed version of the first image. The time delay 128 may be controlled by a user setting or may include the latency of the video of the object's movement. This delay may be adjusted to follow different time scales. For example, the delay may be adjusted to be as small as possible so that the user can account for very fast changes, or the delay may be adjusted to be larger to account for slow changes.

[0337] In some embodiments, this application is used for weather forecasting, and the object of interest is a storm or other local weather effect. Both layers are then input to an AI module 18, which outputs the predicted evolution of the object to a third layer. Time delays can be included, and the predicted images can show multiple possible trajectories, e.g., 135A and 135B, with different probabilities highlighted, or the predicted images can show various outcomes based on different time scales, e.g., local weather patterns of long term trends in climate history.

[0338] In some embodiments, the two images are nearly identical and the predicted image provides information about the edges or differences between the two images or two frames of a video. In this way, this embodiment differentiates the visual content in time. In some embodiments, the different images come from different input streams and the time difference is adjustable to contrast content on different time scales. The AI ​​module may incorporate any physical laws that describe the motion of the object under study.

[0339] In FIG. 13G, a tandem computer is configured for use in building automated financial trading programs with a "real-time programmable update predictor." A user views a central display 9 that displays computer code for high-frequency trading operations. A number of virtual edges 53 are displayed around the central image, all generated by a remote source 6. The edge images include stock market values ​​S1 and trends at various times, and a local workstation automatically compiles the code in real time. The code and remote source data are input to a function 15 to generate and display predicted changes in stock prices along with true values ​​on the edge display S2. In some embodiments, latency is subsumed in a time delay 128 to aid the function in making predictive suggestions on how market conditions will change and which algorithms may be advantageous in future revisions. The function may be an AI module, or the function may be a statistical model in traditional econometrics. In some embodiments, a predictive tool may be used to reduce latency for high-frequency trading that is physically located far from the stock exchange. The feature relies on a dynamic time warp algorithm to compare time series data and optimize the agreement between the time series data.

[0340] 13H and 13I show an embodiment in which a user is viewing a multi-layered display. In FIG. 13H, a multi-layered display 11 is viewed by a user 1. Different content is displayed on each layer. Each layer corresponds to a different focal depth to which the user's eyes accommodate individually. The sensor 13 may detect information about the user, such as gaze or shape. The sensor may be a gesture camera that identifies a specific gesture made by the viewer. The sensor may detect information about the environment. Data from the sensor and the content on each layer are input to a set of AI modules 18A and 18B, which may be configured differently, for example using different dictionaries, training mechanisms, or architectures. The output of the AI ​​modules affects the display content. For example, the user's gaze may be at the center of the display and the content of each layer moves radially inwards or outwards to change the position where the content is displayed. In this way, the set of AI modules creates a feedback and communication network between the viewer and the various focal planes. Figure 13I shows a similar setup, including four layers in a multi-layer display 11, and different AI modules 18A, 18B, and 18C. User 1 focuses on a point 130 in the back layer, causing the lenses of both eyes to accommodate to that depth. When the user looks at content on a different layer, the lens accommodation changes. In either case, the layers may be provided locally or remotely.

[0341] 14A-14E illustrate various embodiments of novel multi-user and collaborative use cases, including but not limited to editing / creating / annotating content of applications, exploring content / data, control room applications, performing processes such as calculations / simulations, drawing, mapping, and analyzing trends / patterns or visualizing multi-dimensional data.

[0342] FIG. 14A shows an embodiment of a “Content-aware content sharer” in which two users 2 are involved in a video conference. One, shown in a central display image 9, is sending information to explain and send content to a receiving collaborative user 2. In this embodiment, the sending user controls some aspects of the receiving user's display system. For example, the sending user may select visual templates for optimal presentation of or interaction with the content. The sending user may decide to send content to the receiving user's multi-layer display 11, which is a set of images for a particular lesson, including annotations made by the user. The sending user directs the content and display system configuration by various means, including gestures, keyboard or mouse input, or voice activation. The collaborative users 2 can interact with the displayed content using a generic input device 12, including a camera 14. In some embodiments, a sensor or sensor array is used to receive input from the users. The roles of explainer and receiver can be dynamically switched based on the configuration of the software that determines who is the host and who is the guest.

[0343] FIG. 14B shows an embodiment of a “collaborative task manager” in which two collaborative users 2 collaborate on a scenario involving a complex task such as air traffic control. Each user has their own display system and content that displays different aspects of the scenario (e.g., air traffic control). In this embodiment, both display systems display a set of multi-layered images 11. Information processed by each user can be passed between the users and sent to different streams of the other user using any input device or sensor. For example, in air traffic control, one user monitors the route of the aircraft 131 while the other user calculates a different optimized route or interacts with the expected trajectory 132. Traffic information monitored by the first user, such as altitude, speed, and heading, can be sent numerically to the second user, who can generate a route that is optimized using code function 76 and passed to the first user as an alternative route in graphical mode. A function used by the software here is the merge function 72, which merges the information of one user into a form that can be used by the second user. The information display for a user monitoring traffic may be more graphical in nature, while the information display for a user performing calculations may be more numerical and tabular in nature. The per-user streams can adapt to the nature of the information being processed. The software may automatically adjust the virtual display content or template based on the information being sent.

[0344] FIG. 14C depicts an embodiment of the "Multiple User Dynamic Content Translator" including an online teaching scenario. A teacher sends content to an audience (e.g., students who are Collaborative User 2), and the audience receives a customized version of the content. This embodiment is similar to the application of FIG. 14A, except that each student may receive the content differently based on their learning preferences and display settings. For example, a first student may be a visual learner, discovered through various calibrations, tests, or interviews with the student, so the content generated for each display system is more graphical. A second student may be good at quantitative reasoning, such that each display is more text-based, including mathematical equations. A software function then converts the original content into multiple display content, for example, by machine learning algorithms. For example, an instructor may be listing information about physical principles. A speech-to-text program transcribes the statement, and an AI generator generates visual content based on the text and a web search using the text as input. All this information is transmitted to the student's workstation, where a local analyzer then determines which technique (audio, text, or image, or any combination) is best based on the student's history and input. Each of the users receives the content shared by the teacher via a unique whiteboard function 70A, 70B, 70C that incorporates such conversion functionality.

[0345] In some embodiments, the dynamic transformation uses the data or metadata and the AI ​​module provides an annotation layer that helps the student formulate questions. The annotation layer may be displayed for the instructor or for the student.

[0346] FIG. 14D depicts an embodiment of a collaborative scenario, such as movie-making or the creation of entertainment media between two users at different locations. In this "generative content multi-user mixer" embodiment, two users 2 share a multi-layered display 11, which includes a common work in progress displayed on each of the users' display systems. Each user has a FoV 2D extension 25 or multi-layered display 11 that can be sent to the other user while discussing different aspects of the work being performed. For example, a first user can work on editing the frames and soundtrack 133, while a second user can work on improving and adding artificial effects 134 to the clips and soundtrack being edited by the first user. Each user can use the peripheral display to display different suggestions to the other user to edit and improve the final product. In some embodiments, this configuration is used for web conferencing, multiplayer games, or collaborative remote operation.

[0347] In some embodiments, an annotation layer may be AI-generated: An annotation layer for one user may take another user's details as input and output various app suggestions, such as hyperlinks, advertisements, or chat interfaces, to assist in completing the collaborative task.

[0348] FIG. 14E represents an embodiment of the collaborative scenario "collaborative content merger" in which two users 1, not necessarily in the same location, are generating multi-dimensional content, for example in the creation of a research paper including text and graphics. The first user may be focused on analyzing and reporting medical images in a display image 9, while the second user may be focused on analyzing and reporting the effects of a drug in a second display 9. The results of both reports are sent to a remote source 6, which then merges and updates the collaborative results live, after which the collaborative results are displayed by the display system, for example as layers in the remotely supplied image 10. The images may together form a multi-layered display that may include various superimposed annotations 17. As the work progresses, both users see the same collaborative project. In some embodiments, the two users are in the same physical location and the final result is displayed at once for both users to see simultaneously in a common display. In some embodiments, various annotations may be made on any layer by either user for the other user to see. In this embodiment, the users may be using different software applications. For example, both users may contribute text, with one user using what you see is what you get (WYSIWYG) software while the other user uses plain text software. As the individual content is generated, it is converted locally into a common format. Both sets of information are uploaded to the remote source 6, which analyzes and consolidates or merges these sets of information together. The remote source can then send the merged document back to the user's display system for display.

[0349] 15A and 15B show different flow charts illustrating different text and graphic editing techniques in tandem.

[0350] FIG. 15A shows a flow chart of a tandem text editing scenario, where two users are working on the same document. Each user is working with a different text editing tool. For example, one user can work with a plain text editor 135A such as TeX, and the other user can work with a WYSIWYG editor 135B such as Microsoft Word. The local processing unit 136 includes a conversion block 136A and a conversion block 136B for converting the input data into a common format. The common format can be ASCII type data. This information is sent to the remote source 6, which analyzes the data in an analysis block 137. This block also captures a dictionary 138 configured for use with a neural network. The data is then merged in a merge block 139 and compiled in a compilation block 140. The remote source then directs the results to a display system for display 85. The compiled data is fed to a conversion to dictionary data block 141 to update the analysis dictionary.

[0351] FIG. 15B shows a flow chart in which a 3D image is generated by a remote source 6. In some embodiments, the original data is a 2D image 142 or a set of 2D images. The remote software includes an estimation block 143 to estimate the depth in the images based on various cues, such as stereo information from multiple cameras, time-of-flight information, depth from shading or contour, projective geometry, or monocular depth estimation, and merge those depths into a 3D image 144. The remote source 6 then transmits this information to a local source, which includes a threshold block 145, which thresholds or bins the depth information into a discrete set of depth planes corresponding to layers of the multi-layered display 85. The thresholding optimization may use a neural network that includes inputs 146 from the user or SLAM information, in addition to an HVS dictionary 147 that includes information about the human visual system (HVS). The HVS dictionary may include information about visual acuity or depth perception. The depth perception information may include data on the human horopter and Panum's fusion zone, measures of vergence or accommodation in human populations that vary with age, or information on the brain, eyes, and connecting nervous systems. In some embodiments, the depth perception information is weighted highly relative to other information such that the algorithm optimizes the viewer's perception of depth of the image content. For example, the focal plane of the image may be mapped to a contour associated with Panum's fusion zone.

[0352] 16A-16C show a set of pipelines in which both remote and local sources are used to generate display content. In some embodiments, the pipelines are application independent and configured by the local workstation.

[0353] In FIG. 16A, the local computer 148 divides the pixels into two sets s j ' and s j Divide into pixels s j Is there any behavior R ij The pixel s is sent to the remote source 6 to be rendered by thej ' is the operation L ij The resulting pixels are added together for a sum of 149 to produce pixel p i =L ij s j +R ij s j These pixels are then displayed as display 85. In some embodiments, user input 95, based in part on the current display content, is fed back to the local computer to modify the set of remotely transmitted pixels and locally transmitted pixels. In some embodiments, set s j and s j In some embodiments, these sets overlap such that the intersection represents the set of pixels that receive contributions from both remote and local sources as a weighted superposition.

[0354] FIG. 16B shows a similar pipeline where content is split by bandwidth. A content controller unit 150 analyzes the desired display content. This information is sent to a basis optimizer unit 151, which breaks down the content into appropriate bases. For example, the bases can be standard Fourier bases, or the bases can be sparse bases, or wavelet bases, or content-adaptive bases. The basis modes in which most of the energy resides (referred to herein as high bandwidth content) are sent to the remote source 6, which then captures or renders that content. The low bandwidth content is then captured or rendered by the local source 5. The two sets of content are then added together as a sum 149, and this sum is sent as the display 85. In some embodiments, user input 95 can adjust the type of content. For example, a user may want to process an image or use a particular function. The user can select a technique, which in turn corresponds to a particular image basis. The basis can be a standard Fourier basis, a point-sparse basis, an edge basis, or a higher-level basis for high-level object detection. An image I is computed using weights w m The fundamental mode B accompanied by m As a superposition of I(r m )=Σ m' w m' B m' (r m ) Next, w m' Threshold w m' Find the range of m' where m' exceeds a threshold. This range corresponds to the bandwidth portion of the display content. Send that range to a high-bandwidth processor to process and produce pixel values. Use a low-bandwidth processor to combine the remaining portion, add the results, and send them to the display system.

[0355] In some embodiments, the content is separated based on feature type. For example, a remote source generates display content that includes sharp edges and a local source generates display content that includes broad features. Or in some embodiments, a remote source generates information about a subject and a remote source generates information about a scene. The basis selected may depend on the particular software application, or the basis selected may be created dynamically. In this way, the separation is in the form of a foveated rendering.

[0356] In Figure 16C, the content of the multi-layered graphical display is analyzed by a local computer 148. A portion of the layer information is sent to a local source 5 for display generation. The results are sent to a remote source 6. The results of the local and remote display content generation are added together as a sum 149 and displayed as a display 85. User input 95 allows modification of the local computer's analysis of the desired content. In some embodiments, information regarding the latency of the remote source is integrated into a time delay block 152 with predictive modeling capabilities to affect the remote display content.

[0357] 17A-17D show some auxiliary embodiments including infographic displays of various events spread across time and space.

[0358] 17A shows a "time-spanning" embodiment, in which the central display 9 displays information and events related to the current time, while a display as a bar below the central display, past content graphics 155, displays past events in a sequence leading to the present. Meanwhile, the screen above the central screen displays future content graphics 154, which display a stack of possible future options and events in such a way that more likely events / options in the near future are emphasized over possible events / options compared to the far future graphics 153. In some embodiments, the width of the bar indicates the likelihood of the activity, and the position indicates a recommendation based on other factors (such as productivity or time of day). Bubbles indicate minimum certainty in the far future.

[0359] 17B shows a "spread across space" embodiment in which past indicated events 156A, 156B, 156C, and 156D are displayed in such a way that the size of possible events in the near future is related to the likelihood that a user will act on such events. In some embodiments, the shading or color of different elements indicates future recommended actions.

[0360] 17C shows a "tree-spreading" embodiment in which a target event 157A is connected to previous events 157C displayed in the screen below the center screen, and possible derived events 157B from the target event are displayed in the screen above the center screen. Nodes in the graph correspond to various past, present, or future actions. Correlations between actions determine connections. In some embodiments, there are multiple components where each component is a graph and the nodes of one component are not connected to the nodes of any other component.

[0361] FIG. 17D shows an embodiment of an infographic in which the tone of suggested words in a writing, e.g., email, changes dynamically depending on the user's current input. The tone can change between a positive tone, a negative tone, or a neutral tone. The current display 9 may display an image of an email environment for the user to compose an email. At the start, the user is presented with the first suggested words, organized vertically based on tone. The first distribution 158A is centered on a neutral tone. As the user proceeds with the word selection 159, the user selects a slightly more negative tone, so the next distribution 158B displays a different set of words based on the tone selected. The third distribution 158C follows the user changing the tone from negative to positive.

Claims

1. an input stream module for generating or receiving a plurality of input streams; a functional module performing a plurality of functions, each function of the plurality of functions configured to operate on at least one input stream of the plurality of input streams; a plurality of visual templates, each template of the plurality of visual templates configured to format an output from the plurality of functions; a graphical user interface configured to generate a graphical representation of the plurality of functions, the plurality of input streams, and the plurality of visual templates; 1. An augmented display generator comprising: user selection of at least one of the plurality of functions, the plurality of input streams, and the plurality of visual templates via the graphical user interface generates display content; an augmented display generator, wherein the display content includes an output corresponding to a presented selected feature from the plurality of features using a visual template from the plurality of visual templates;

2. The augmented display generator of claim 1 , further comprising an augmented display system for receiving and displaying the display content.

3. the plurality of input streams includes a first input stream and a second input stream; the extended display system comprises at least one main section and at least one extended section; 3. The extended display generator of claim 2, wherein the extended display system displays a first display content based on the first input stream in the main section and a second display content in the extended section, the second display content being output from one of the plurality of functions configured to operate on the first display content and the second input stream.

4. 4. The augmented display generator of claim 3, wherein the augmented display system is a virtual display system, the main section is configured to be displayed at a first depth on the virtual display system, and the at least one extended section is one of a plurality of extended sections configured to be displayed at a plurality of different depths on the virtual display system.

5. The augmented display generator of claim 3 , further comprising an input device from the set consisting of an eye tracker, a head tracker, and a localization device.

6. The augmented display generator of claim 3 , wherein multiple augmented parts of the augmented display system communicate wirelessly.

7. the augmented display system is a virtual display system; 3. The augmented display generator of claim 2, wherein a template from the plurality of visual templates formats a plurality of virtual images at each depth, each depth including at least two or more unique depths.

8. 3. The augmented display generator of claim 2, wherein the augmented display system is configured to display multiple objects within the display content in multiple focal planes, and the depth of one of the multiple focal planes is calculated by an algorithm that optimizes a user's depth perception.

9. a function of the plurality of functions configured to operate on a first input stream of the plurality of input streams and to output an annotation based on detection of a plurality of features of the first input stream; 3. The augmented display generator of claim 2, wherein the augmented display system is a virtual display system configured to display the annotations in a first focal plane and to display content of the first input stream in a second focal plane different from the first focal plane.

10. The augmented display generator of claim 2 , wherein the augmented display system is integrated into a phone, a watch, a tablet, a headset, a viewer, a viewfinder, a television, or a vehicle instrument cluster.

11. 2. The augmented display generator of claim 1, wherein a visual template from the plurality of templates is configured to augment a field of view using a plurality of virtual images tiled along a direction substantially perpendicular to a user's line of sight.

12. 2. The augmented display generator of claim 1, wherein the input stream module generates or receives at least one of the plurality of input streams using or from a set including the internet, a game, a game engine, an existing application, a website, a simulation, a training video, a camera video, a camera image, a user input, a sensor input, and a data set.

13. 2. The augmented display generator of claim 1, wherein at least one function from the plurality of functions is an artificial intelligence (AI) function configured to generate at least a portion of the display content based on user input.

14. An experience station, comprising: an input stream module that generates or receives a number of input streams; a functional module implementing a plurality of functions, each of said functions operating on at least one of said plurality of input streams; a visual template that formats output from the plurality of functions for display on an augmented portion of an augmented display system; Equipped with Any of the functions is a generating function configured to: (i) receive the multiple input streams, at least one of the input streams providing display content to a main portion of the augmented display system; (ii) capture a sequence of actions by a user; and (iii) cause the augmented display system to display as the output a plurality of possible future actions by the user on the multiple input streams. Experience station.

15. The experience station of claim 14 , wherein the generation functionality is configured to use a time factor to determine the plurality of possible future actions for the augmented portion of the augmented display system.

16. The experience station of claim 14 , wherein the generation functionality includes an event-based action trigger that, when triggered, causes generation of content in the augmented portion of the augmented display system.

17. The experience station of claim 14 , wherein the generation functionality is further configured to automatically suggest and execute predicted tasks in parallel with the user's actions based on an authority level granted by the user.

18. 15. The experience station of claim 14, wherein the multiple input streams include at least one of the following: internet, games, existing applications, websites, videos, images, and data visualizers.

19. The experience station of claim 14, further comprising a sensor for detecting the user's gaze and outputting a gaze position as one of the multiple input streams, and wherein the generation function is configured to dynamically change the display of the multiple possible future actions based on the gaze position.

20. An input stream module for generating or receiving a plurality of input streams; a functional module implementing a plurality of functions, each of said functions operating on at least one of said plurality of input streams; a visual template that formats output from the plurality of functions for display on an augmented portion of an augmented display system; Equipped with a first display content from one of a plurality of input streams is displayed on a main portion of the augmented display system, and the output is displayed as a second display content on the augmented portion of the augmented display system. Generative augmented display station.

21. 21. The generative augmented display station of claim 20, wherein the augmented display system is a virtual display system such that a multi-layer display displays images from the multiple input streams in a first layer and the functional module generates an annotation layer.

22. 21. The generative augmented display station of claim 20, further comprising a sensor for capturing information from a user, the information being one of the multiple input streams input to the functional module, the functional module further configured to modify the first or second display content based on the information.

23. 21. The generative augmentation display station of claim 20, further comprising an artificial intelligence (AI) module configured to receive as input a user query, such that the second display content is recommendations of a plurality of vertical search engines, a recommendation of a first vertical search engine being determined by the query, and a recommendation of a second vertical search engine being determined at least in part by a result of the recommendation of the first vertical search engine.

24. the functional module is an artificial intelligence (AI) module comprising a neural network or a transformer; 21. The generative augmented display station of claim 20, wherein the AI ​​module is configured to compare at least two of the multiple input streams and merge the input streams into a single display content on the augmented display system.

25. The generative augmented display station of claim 20, further comprising the augmented display system.

26. The generative augmented display station of claim 20, further comprising a graphical user interface that generates a graphical representation of the plurality of input streams, the plurality of functions, and the visual template.

27. The experience station of claim 14, further comprising the augmented display system.

28. The experience station of claim 14, further comprising a graphical user interface that generates a graphical representation of the plurality of input streams, the plurality of functions, and the visual template.

Citation Information

Patent Citations

  • Transactions in virtual property

    US20050021472A1

  • System and Method for Display Power Management for Dual Screen Display Device

    US20150116364A1

  • Methods and systems for synchronizing media stream presentations

    US20150215497A1

  • Coordinating multiple virtual environments

    US20170021273A1

  • Magic wand methods, apparatuses and systems for defining, initiating, and conducting quests

    US20170087453A1

Cited By

  • Information processing device, information processing system, control method for information processing device, and storage medium

    EP4787881A1