Systems and methods for augmented reality

Abstract

The augmented reality viewer includes components, assemblies, and executable logic to provide a user with a perception of a rich augmented reality experience, including aspects of an aquatic world.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 879,408, filed on July 26, 2019; U.S. Provisional Patent Application No. 62 / 881,355, filed on July 31, 2019; and U.S. Provisional Patent Application No. 62 / 899,678, filed on September 12, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to augmented reality viewers and augmented reality visualization methods. Background Art

[0004] Modern computing technology has advanced to the level of compact and continuously connected wearable computing systems and components, which can be used to provide users with the perception of a rich augmented reality experience.

[0005] An augmented reality viewer typically has multiple sensors, including a camera positioned to sense the positions of real - world objects. A storage device stores a data set including virtual objects. A display module on the storage device can be executed by a processor to determine the desired display of virtual objects relative to the positions of real - world objects. A data - stream generator on the storage device can be executed by the processor to generate a data stream based on the data and the desired display. A light generator (such as a laser generator) is connected to the processor to receive the data stream and generate light based on the data stream. A display device is positioned to receive the generated light and display the light to the user. The light creates a rendering of the virtual object, which is visible to the user and is rendered according to the desired display. Summary of the Invention

[0006] The present invention provides an augmented reality viewer, comprising: at least one sensor positioned to sense the position of at least one real - world object among a plurality of real - world objects; a storage device; a data set on the storage device, the data set including virtual objects; a processor connected to the storage device; a display module located on the storage device and executable by the processor to determine the desired display of virtual objects relative to the position of at least one real - world object among the real - world objects; a data - stream generator located on the storage device and executable by the processor to generate a data stream based on the data and the desired display; a light generator connected to the processor to receive the data stream and generate light based on the data stream; and a display device positioned to receive the generated light and display the light to the user, wherein the light creates a rendering of the virtual object, which is visible to the user and is rendered according to the desired display.

[0007] The augmented reality viewer may further include: the virtual object is a swimming life object, and further includes a swimming subroutine, which is located on a storage device and executable by a processor to move the swimming life object relative to a real-world object.

[0008] The augmented reality viewer may further include data that includes a plurality of swimming life objects, wherein the swimming subroutine is executable by a processor to move the plurality of swimming life objects relative to a real-world object and relative to each other.

[0009] The augmented reality viewer may further include at least one sensor that senses a wave motion initiated by a user, and the augmented reality viewer further includes a wave motion routine, which is located on a storage device and executed by a processor to move the swimming life object in response to the wave motion sensed by the at least one sensor.

[0010] The augmented reality viewer may further include initiating a wave motion in a target area, and the swimming object is moved out of the target area.

[0011] The augmented reality viewer may further include at least one sensor that senses wave motion, and the sensor is a camera that detects an image of the user's hand.

[0012] The augmented reality viewer may further include a handheld controller, wherein the sensor is mounted to the handheld controller held by the user's hand.

[0013] The augmented reality viewer may further include a retrieval agent, which is located on a storage device and executable with a processor to retrieve incoming information from a resource, associate the incoming information with the position of the swimming life object relative to the real-world object, and transmit the incoming information from the position of the swimming life object to the user.

[0014] The augmented reality viewer may further include a retrieval agent that activates a light source to display the incoming information through a display.

[0015] The augmented reality viewer may further include a speaker, wherein the retrieval agent activates the speaker to enable the user to hear the incoming information.

[0016] The augmented reality viewer may further include a transmission agent, which is located on a storage device and executable with a processor to sense an instruction of the user with at least one sensor, sense an instruction of the user pointing to the swimming life object at the position of the swimming life object relative to the real-world object with at least one sensor, determine an outgoing communication and a resource based on the instruction, and transmit the outgoing communication to the resource.

[0017] The augmented reality viewer may further include at least one sensor that senses communication, and the sensor is a microphone suitable for receiving a voice instruction from the user.

[0018] The augmented reality viewer may further include a transmission agent that is located on a storage device and executable by a processor to sense a user's instruction with at least one sensor, sense an instruction to point to a roaming living object by the user at a position of the roaming living object relative to a real-world object with at least one sensor, determine an outgoing communication and an IOT device based on the instruction, and transmit the outgoing communication to the IOT device to operate the IOT device.

[0019] The augmented reality viewer may further include at least one sensor for sensing communication, and the sensor is a microphone suitable for receiving voice instructions from the user.

[0020] The augmented reality viewer may further include an artificial intelligence system that is located on a storage device and executable by a processor to sense a user's action with at least one sensor, execute routines of virtual objects in response to the user's action, associate the routines with the action as an artificial intelligence cluster, determine parameters existing at a first time when the action is sensed, associate the parameters existing at the first time with the artificial intelligence cluster, sense the parameters at a second time, determine whether the parameters at the second time are the same as the parameters at the first time, and if it is determined that the parameters at the second time are the same as the parameters at the first time, execute the routines.

[0021] The augmented reality viewer may further include: at least one sensor including an eye-tracking camera, wherein the action is a wave motion initiated by the user, and the parameter is the gaze direction determined by using the eye-tracking camera to track the user's eyes.

[0022] The augmented reality viewer may further include an action that is a wave motion initiated in a target area, and the roaming object is moved out of the target area.

[0023] The augmented reality viewer may further include: at least one sensor including an eye-tracking camera that is positioned to sense the user's gaze direction, and further include a gaze direction movement routine that is located on a storage device and executable by a processor to move the roaming living object in response to the gaze direction sensed by the eye-tracking camera.

[0024] The augmented reality viewer may further include that there are multiple roaming living objects and a roaming subroutine is executable by a processor to move the multiple roaming living objects relative to real-world objects, and further include a personal assistant module that is located on a storage device and executable by a processor to select a personal assistant roaming living object from among the multiple roaming living objects, and move at least one roaming living object other than the personal assistant roaming living object among the roaming living objects together with the personal assistant roaming living object.

[0025] The augmented reality viewer may further include: The swimming living object is a fish of the first type, and further includes a motion module, which is located on a storage device and can be executed by a processor to articulate the body of the fish in a first back-and-forth manner.

[0026] The augmented reality viewer may further include: a motion module, which can be executed by a processor to articulate the body of a fish of the second type in a second back-and-forth manner different from the first back-and-forth manner.

[0027] The augmented reality viewer may further include: a motion module, which can be executed by a processor to sense a slow hand movement of a user's hand with at least one sensor, in response to the slow speed of the hand movement, articulate the body of the fish in the first back-and-forth manner at a low speed, sense a fast hand movement of the user's hand with at least one sensor, and in response to the fast speed of the hand movement, articulate the body of the fish in the first back-and-forth manner at a high speed.

[0028] The augmented reality viewer may further include: a motion module, which can be executed by a processor to move the first fish closer to the hand when the hand moves at a slow speed, and move the first fish away from the hand when the hand moves at a fast speed.

[0029] The augmented reality viewer may further include a surface extraction routine, which is located on a storage device and can be executed by a processor to identify a surface in a real-world object.

[0030] The augmented reality viewer may further include that the surface is a two-dimensional surface of a wall or a ceiling.

[0031] The augmented reality viewer may further include a depth creation module, which is located on a storage device and can be executed by a processor to display a virtual object to the user on the opposite side of the surface from the user in three-dimensional space according to a desired display, and the surface is between the user and the virtual object.

[0032] The augmented reality viewer may further include: a depth creation module, which can be executed by a processor to display a porthole in the surface to the user, through which the virtual object is visible to the user.

[0033] The augmented reality viewer may further include that the virtual object is a three-dimensional virtual object.

[0034] The augmented reality viewer may also include a vista placement routine that is located on a storage device and executable by a processor to capture a space including real-world objects, represent the space as a real-world grid, collect vertical and horizontal planes from the real-world grid, filter the planes by position, orientation, size, and height from the floor relative to the real-world grid, generate a blueprint that includes all content and portal frames in the vista at a selected location, and cut holes in occlusion materials of the real-world grid material so that the user can see the vista through the portal.

[0035] The augmented reality viewer may also include a vertex animation routine that is located on a storage device and executable by a processor to store a virtual object grid representing a virtual object, associate a texture with the virtual object grid, and manipulate the virtual object grid to cause movement of the texture and the virtual object in the user's view.

[0036] The augmented reality viewer may also include manipulating the virtual object grid to connect virtual objects.

[0037] The augmented reality viewer may also include the same virtual object grid being used multiple times to cause movement of the texture and the virtual object.

[0038] The virtual object of the augmented reality viewer may be a coral cluster.

[0039] The augmented reality viewer may also include a coral cluster generator that is located on a storage device and executable by a processor to determine a volume, perform line tracing at random points from the maximum height of the volume to the floor of the volume within the volume, determine whether a valid position is identified by the line tracing, if a valid position is identified, perform box tracing to test whether a random cluster fits without overlapping the world grid when trying different scales and rotations and generating a score for each placement, determine a selected placement having the highest score among the scores, and generate a coral cluster at the placement having the highest score.

[0040] The augmented reality viewer may also include a vista placement routine that is located on a storage device and executable by a processor to place a vista, wherein the volume is constrained by the vista.

[0041] The augmented reality viewer may also include a coral generator system that is located on a storage device and executable by a processor to store at least a first type of first coral element on the storage device, and build a coral cluster following a plurality of coral elements including the first coral element.

[0042] The augmented reality viewer may also include: a coral generator system that is executable by a processor to build a coral cluster from a plurality of first coral elements.

[0043] The augmented reality viewer may further include: a coral generator system capable of being executed by a processor to store at least a second type of second coral element on a storage device, wherein the plurality of coral elements includes the second coral element.

[0044] The augmented reality viewer may further include: a coral generator system capable of being executed by a processor to determine a coral cluster setting, wherein the processor constructs a coral cluster according to the setting.

[0045] The augmented reality viewer may further include: the setting is the detected available space, and a plurality of coral elements are selected based on the available space.

[0046] The augmented reality viewer may further include: a coral generator system capable of being executed by a processor to simulate ambient light, wherein the setting is ambient light, wherein a plurality of coral elements are selected based on the ambient light, and wherein the orientation of the coral elements is selected based on the ambient light.

[0047] The augmented reality viewer may further include: a coral generator system including a data table having a plurality of coral cluster settings on a storage device, wherein the coral generator system constructs a coral cluster according to the plurality of coral cluster settings.

[0048] The augmented reality viewer may further include: the coral cluster setting includes at least one of the following: population; species maximum count; generation type; and height-based percentage.

[0049] The augmented reality viewer may further include: a coral generator system including a vertex crawl and ray casting algorithm for checking placement feasibility, growth, and caching valid points to a file.

[0050] The augmented reality viewer may further include: a coral generator system including a runtime coral static mesh loop that first calculates the shadow channel and then creates an instantiated static mesh of all coral clusters.

[0051] The augmented reality viewer may further include: a coral generator system including a collision-based exclusion configuration that places box colliders where certain species should not grow.

[0052] The augmented reality viewer may further include: when the user views at least one real-world object among real-world objects, the display device displays the generated light to the user.

[0053] The augmented reality viewer may further include: the display device is a perspective display device that allows light from at least one real-world object to reach the user's eyes.

[0054] The augmented reality viewer may further include: a head-mounted structure adapted to be worn on a user's head, wherein a display device is mounted to the head-mounted structure, and at least one sensor is a sensor adapted to sense movement of the display device caused by movement of the user's head; and a positioning adjustment module executable by a processor to adjust the position of a virtual object such that, within the user's field of view, the virtual object remains stationary relative to at least one real-world object.

[0055] The present invention also provides an augmented reality visualization method, including: sensing the position of at least one real-world object among a plurality of real-world objects with at least one sensor, storing data including virtual objects on a storage device, determining, with a processor, a desired display of the virtual objects relative to the position of at least one real-world object among the real-world objects, generating, by the processor, a data stream based on the data and the desired display, generating light based on the data stream with a light generator, and displaying the generated light to a user with a display device, wherein the light creates a rendering of the virtual objects, the rendering being visible to the user and being rendered according to the desired display.

[0056] The method may further include: the virtual object being a swimming life object, and further including moving the swimming life object relative to the real-world object with the processor.

[0057] The method may further include: the data including a plurality of swimming life objects, and further including moving the plurality of swimming life objects relative to the real-world object and relative to each other with the processor.

[0058] The method may further include sensing a wave motion initiated by the user with at least one sensor, and moving the swimming life object in response to the wave motion sensed by at least one sensor with the processor.

[0059] The method may further include initiating a wave motion in a target area, and the swimming object being moved out of the target area.

[0060] The method may further include: at least one sensor for sensing the wave motion being a camera that detects an image of the user's hand.

[0061] The method may further include: mounting the sensor to a handheld controller held by the user's hand.

[0062] The method may further include: retrieving incoming information from a resource with the processor, associating the incoming information with the position of the swimming life object relative to the real-world object with the processor, and transmitting the incoming information from the position of the swimming life object to the user with the processor.

[0063] The method may further include displaying the incoming information to the user.

[0064] The method may further include the user hearing the incoming information.

[0065] The method may further include: sensing a user instruction with at least one sensor, sensing an instruction for the user to point to a virtual living object at a position of the virtual living object relative to a real-world object with at least one sensor, determining an outgoing communication and a resource with a processor based on the instruction, and transmitting the outgoing communication to the resource with the processor.

[0066] The method may further include the user speaking an instruction.

[0067] The method may further include: sensing a user instruction with at least one sensor, sensing an instruction for the user to point to a virtual living object at a position of the virtual living object relative to a real-world object with at least one sensor, determining an outgoing communication and an IOT device with a processor based on the instruction, and transmitting the outgoing communication to the IOT device to operate the IOT device with the processor.

[0068] The method may further include the user speaking an instruction.

[0069] The method may further include: sensing a user action with at least one sensor, executing a routine involving a virtual object with a processor, the routine being responsive to the user action, associating the routine with the action as an artificial intelligence cluster with a processor, determining parameters existing at a first time when the action is sensed with at least one processor, associating the parameters existing at the first time with the artificial intelligence cluster with a processor, sensing the parameters at a second time with at least one sensor, determining whether the parameters at the second time are the same as the parameters at the first time with a processor, and if it is determined that the parameters at the second time are the same as the parameters at the first time, executing the routine.

[0070] The method may further include: the action is a wave motion initiated by the user, and the parameter is the user's gaze direction.

[0071] The method may further include: the action is a wave motion initiated in a target area, and the virtual object is moved out of the target area.

[0072] The method may further include: sensing the user's gaze direction with at least one sensor, and moving the virtual living object in response to the gaze direction with a processor.

[0073] The method may further include: moving a plurality of virtual living objects relative to a real-world object with a processor, selecting a personal assistant virtual living object from among the plurality of virtual living objects with a processor, and moving at least one virtual living object other than the personal assistant virtual living object among the virtual living objects together with the personal assistant virtual living object with a processor.

[0074] The method may further include: the virtual living object is a first type of fish, and further includes connecting the body of the fish in a first back-and-forth manner with a processor.

[0075] The method may further include: using a processor to connect the body of a second type of fish in a second back-and-forth manner different from the first back-and-forth manner.

[0076] The method may further include: using at least one sensor to sense a slow hand movement of a user's hand, using a processor to connect the body of the fish in the first back-and-forth manner at a low speed in response to the slow speed of the hand movement, using at least one sensor to sense a fast hand movement of the user's hand, and using a processor to connect the body of the fish in the first back-and-forth manner at a high speed in response to the fast speed of the hand movement.

[0077] The method may further include: when the hand moves at a slow speed, using a processor to move the first fish closer to the hand; and when the hand moves at a fast speed, using a processor to move the first fish away from the hand.

[0078] The method may further include: using a processor to identify a surface in the real-world object.

[0079] The method may further include: the surface is a two-dimensional surface of a wall or a ceiling.

[0080] The method may further include: the processor displays a virtual object to the user on the opposite side of the surface from the user in three-dimensional space according to the desired display, and the surface is between the user and the virtual object.

[0081] The method may further include: the processor displays a porthole in the surface through which the virtual object is visible to the user.

[0082] The method may further include: the virtual object is a three-dimensional virtual object.

[0083] The method may further include: using a processor to capture a space including the real-world object, using a processor to represent the space as a real-world grid, using a processor to collect vertical and horizontal planes from the real-world grid, using a processor to filter the planes by position, orientation, size, and height from the floor relative to the real-world grid, using a processor to generate a blueprint that includes everything in the vista at a selected location and a portal framework, and using a processor to cut a hole in an occluding material of the real-world grid material so that the user can see the vista through the portal.

[0084] The method may further include: using a processor to store a virtual object grid representing the virtual object, using a processor to associate a texture with the virtual object grid, and using a processor to manipulate the virtual object grid to cause movement of the texture and the virtual object in the user's field of view.

[0085] The method may further include manipulating the virtual object grid to connect virtual objects.

[0086] The method may further include that the same virtual object mesh is used multiple times to cause the movement of textures and virtual objects.

[0087] The method may further include that the virtual object is a coral cluster.

[0088] The method may further include placing the coral cluster with a processor, including: determining a volume, performing a line trace at a random point from the maximum height of the volume to the floor of the volume within the volume, determining whether a valid position is identified by the line trace, if a valid position is identified, performing a box trace to test whether the random cluster fits without overlapping the world mesh when trying different scales and rotations and generating a score for each placement, determining a selected placement with the highest score among the scores, and generating the coral cluster into the placement with the highest score.

[0089] The method may further include placing a vista, wherein the volume is constrained by the vista.

[0090] The method may further include storing at least a first type of first coral element on a storage device; and building the coral cluster with a processor from a plurality of coral elements including the first coral element.

[0091] The method may further include that the processor builds the coral cluster from a plurality of first coral elements.

[0092] The method may further include storing at least a second type of second coral element on the storage device, wherein the plurality of coral elements includes the second coral element.

[0093] The method may further include determining coral cluster settings with a processor, wherein the processor builds the coral cluster according to the settings.

[0094] The method may further include: setting to the detected available space, and selecting a plurality of coral elements based on the available space.

[0095] The method may further include simulating ambient light with a processor, wherein the setting is ambient light, wherein a plurality of coral elements are selected based on the ambient light, and wherein the orientation of the coral elements is selected based on the ambient light.

[0096] The method may further include storing a data table having a plurality of coral cluster settings on the storage device, wherein the processor builds the coral cluster according to the plurality of coral cluster settings.

[0097] The method may further include that the coral cluster settings include at least one of the following: population; species maximum count; generation type; and height-based percentage.

[0098] The method may further include: executing, by a processor, vertex crawling and ray casting algorithms for checking placement feasibility, growth, and caching valid points to a file.

[0099] The method may further include: executing, by a processor, a runtime coral static mesh loop that first calculates the shadow pass and then creates an instantiated static mesh for all coral clusters.

[0100] The method may further include: executing, by a processor, a collision-based exclusion configuration for placing box colliders where certain species should not grow.

[0101] The method: when the user views at least one real-world object in the real world, the display device displays the generated light to the user.

[0102] The method: the display device is a perspective display device that allows light from at least one real-world object to reach the user's eyes.

[0103] The method may further include: sensing, by at least one sensor, movement of the display device caused by movement of the user's head; and adjusting the position of the virtual object such that the virtual object remains stationary relative to at least one real-world object within the user's field of view. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] The present invention is further described by way of example in the following drawings, in which:

[0105] Figure 1 is a perspective view of an augmented reality system according to an embodiment of the present invention;

[0106] Figure 2 is a flowchart showing the basic operation of the augmented reality system;

[0107] Figure 3 is a depth map showing the creation of such a mesh;

[0108] Figure 4 is a depth map showing the mapping of a "vista" volume;

[0109] Figures 5A to 5E is a depth map showing a living room scene where the user can experience a dynamic virtual aquatic environment;

[0110] Figure 6 is a view configured to assist the user in selecting the size and position for framing / portal viewing into a "vista";

[0111] Figure 7 is a view showing the various aspects of the "vista" that can be selected;

[0112] Figures 8A to 8C is a depth map with a created virtual framework or portal that enables him to virtually "see" outside into the ocean "vista" volume;

[0113] Figures 9A to 9D is a depth map showing the interaction between the user and various swimming objects;

[0114] Figure 10A and 10B is a view of a mathematical model of the cyclic behavior of various virtual characters;

[0115] Figure 11A is a screenshot of a configuration that can be used to move fish relative to the user;

[0116] Figure 11B is a flowchart of vertex animation that helps convert animated skeletal assets to animated vertex meshes in clownfish;

[0117] Figures 12A to 12D is a depth map of vertex animation that helps convert animated skeletal assets to animated vertex meshes in other virtual objects;

[0118] Figure 13 is a slide showing the unpredictability in an unknown environment;

[0119] Figure 14 is a screenshot of a "code blueprint";

[0120] Figure 15 is a screenshot showing aspects of an interface for arranging and texturing various elements;

[0121] Figure 16 is a depth map of a coral cluster of components including smaller coral elements;

[0122] Figure 17A is a depth map of a high-fidelity model of a diver's helmet;

[0123] Figure 17B and 17C is a depth map showing a more efficient reproducible output of a diver's helmet;

[0124] Figure 18 is a screenshot of a core unlit shader network configuration;

[0125] Figure 19 and 20 is a depth map showing that the system can be configured to utilize the geometry of a room or environment;

[0126] Figure 21Is a depth map showing the position of a location within a grid of a room, which position can be a suitable anchor for an element such as a coral cluster;

[0127] Figures 22A to 24 Is a screenshot showing the creation of an interesting and believable interaction between a user and an augmented reality element;

[0128] Figure 25 Is a screenshot of a tool for visualizing the utilization of various computational threads over time;

[0129] Figure 26 Is a slide on various aspects of learning;

[0130] Figure 27 Is a sequence diagram of an animation pipeline;

[0131] Figure 28 Is a screenshot of an artificial intelligence (“AI”) pathfinding solution;

[0132] Figure 29 Is a depth map showing the baking of animation data into textures;

[0133] Figures 30A to 30H Is a depth map of various coral cluster configurations;

[0134] Figure 31A and 31B Is a depth map of a coral generator object system; and

[0135] Figure 32 Is a block diagram of various instruction routines and modules. Detailed Description

[0136] Figure 1An augmented reality system is shown that includes an augmented reality viewer 2, a handheld controller 4, and an interconnected auxiliary computing or controller component 6 that can be configured to be worn on a user as a fanny pack or the like. Each of these components can be operably coupled to each other 10, 12, 14, 16, 17, 18 via a wired or wireless communication configuration and coupled to other connected resources 8 (such as cloud computing or cloud storage resources). The connected resources 8 can be those specified by IEEE 802.11, Bluetooth RTM, and other connection standards and configurations. For example, as described in U.S. Patent Application Serial Numbers 14 / 555,585, 14 / 690,401, 14 / 331,218, 15 / 481,255, 62 / 627,155, 62 / 518,539, 16 / 229,532, 16 / 155,564, 15 / 413,284, 16 / 020,541, 62,702,322, 62 / 206,765, 15,597,694, 16 / 221,065, 15 / 968,673, and 62 / 682,788, each of which is incorporated herein by reference, various aspects of such components are described, such as various embodiments of the two depicted optical elements 20 through which a user can see the world around them along with visual components that can be generated by associated system components to obtain an enhanced reality experience. The system can also include various sensors configured to provide information related to the user's surrounding environment, including camera type sensors such as monochrome, color / RGB, and / or thermal imaging components 22, 24, 26, depth camera sensors 28, and sound sensors 30 (such as microphones).

[0137] The augmented reality viewer 2 includes a head-mounted structure 25 that can be worn on a user's head. The augmented reality viewer 2 and the controller component 6 each have a processor and a storage device connected to the processor. Data and executable code are stored on the storage device and executable by the processor. The augmented reality viewer 2 includes a projector that serves as a light generator. The processor of the augmented reality viewer 2 sends instructions to the projector and the projector generates light, typically a laser, that is transmitted through the display to the user's eyes.

[0138] Figure 1Accordingly, an augmented reality viewer is described, the augmented reality viewer including: at least one sensor positioned to sense the position of at least one real-world object among a plurality of real-world objects; a storage device; a data set on the storage device, the data set including virtual objects; a processor connected to the storage device; a display module located on the storage device and executable by the processor to determine a desired display of the virtual objects relative to the position of at least one real-world object among the real-world objects; a data stream generator located on the storage device and executable by the processor to generate a data stream based on the data and the desired display; a light generator connected to the processor to receive the data stream and generate light based on the data stream; and a display device positioned to receive the generated light and display the light to a user, wherein the light creates a rendering of the virtual objects, the rendering being visible to the user and being rendered according to the desired display.

[0139] When a user views at least one real-world object among the real-world objects, the display device may display the generated light to the user. The display device may be a perspective display device that allows light from at least one real-world object to reach the user's eyes.

[0140] The augmented reality viewer may include: a head-mounted structure adapted to be worn on a user's head, wherein the display device is mounted to the head-mounted structure and at least one sensor is a type of sensor adapted to sense movement of the display device caused by movement of the user's head; and a positioning adjustment module executable by the processor to adjust the position of the virtual objects such that, within the user's field of view, the virtual objects remain stationary relative to at least one real-world object.

[0141] The following definitions will assist in understanding the various terms used herein:

[0142] 1. Virtual object

[0143] 1.1. Lifelike object (an object that swims and migrates, displays connected features, or has a moving surface texture)

[0144] 1.1.1. Swimming classification (an object that moves from one position to another)

[0145] 1.1.1.1. Swimming life

[0146] 1.1.1.1.1. Sea otter

[0147] 1.1.1.1.2. Fish

[0148] 1.1.1.1.3. Turtle

[0149] 1.1.1.2. Non-swimming life

[0150] 1.1.1.2.1. Sea anemone

[0151] 1.1.1.2.2. Coral

[0152] 1.1.2. Connection classification

[0153] 1.1.2.1. Connected life (objects with a curved body)

[0154] 1.1.2.1.1. Sea otter

[0155] 1.1.2.1.2. Fish

[0156] 1.1.2.1.3. Sea turtle

[0157] 1.1.2.1.4. Sea anemone

[0158] 1.1.2.2. Non - connected life (objects with an unbending body)

[0159] 1.1.2.2.1. Coral

[0160] 1.1.3. Texture classification (surface features)

[0161] 1.1.3.1. Fluid - textured life (objects with a moving surface)

[0162] 1.1.3.1.1. Fish

[0163] 1.1.3.1.2. Sea otter

[0164] 1.1.3.1.3. Sea anemone

[0165] 1.1.3.2. Fixed - textured life (objects with a static surface)

[0166] 1.1.3.2.1. Sea turtle

[0167] 1.1.3.2.2. Coral

[0168] 1.2. Inanimate objects (objects that do not migrate by swimming, do not show connected features, and do not have a moving surface texture)

[0169] 1.2.1. Rock

[0170] 1.2.2. Sand

[0171] 2. Real - life objects

[0172] 2.1. Wall

[0173] 2.2. Ceiling

[0174] 2.3. Furniture

[0175] Reference Figure 2 In various embodiments, a user may wear an augmented reality system such as that depicted in Figure 1 . With respect to the interaction of such a system with the three-dimensional world around the user when operating at 32, such an augmented reality system may also be referred to as a "spatial computing" system. Such a system may include, for example, a head-mounted display component 2 and may be characterized by the environmental sensing capabilities described above and incorporated by reference in the foregoing, such as various types of cameras that may be configured to map the environment around the user or create a "mesh" 56 of such an environment. Figure 3 Illustrates the creation of such a mesh. A point cloud forming a mesh of a given room is depicted, where walls or perimeters 42, 52 and chairs 46, 48, 50, 54 and a central desk 44 are shown. As Figure 2 further shown at 34, the spatial computing system may be configured to map or mesh the environment around the user, as referenced above Figure 3 described.

[0176] At 36, the spatial computing system may be configured to run or operate software, such as software available from Magic Leap, Inc., Plantation, Florida, under the trade name Undersea(TM). The software may be configured to utilize the mapping or mesh of the room to assist the user in selecting (such as using Figure 1 a handheld controller 4, or via gestures or other available control options available to the system and software) the location of virtual image elements, such as aquatic coral growth, aquatic scene features (i.e., components such as sunken shipwrecks), and virtual connection framing or porting to one or more virtual "vista" elements, where a virtual extension of the presented virtual aquatic environment is perceived to extend beyond the actual geometry of the room (i.e., a relatively large, substantially spherical extended volume of the presented aquatic environment extends beyond the actual room, a roughly rectangular prism occupied by the user). In various embodiments, the system may be configured to automatically locate possible locations, such as vertical or horizontal planar locations within the user's actual room, for the possible placement of corals, seaweed forests, etc., for selection or placement confirmation by the user. In various embodiments, the system may be configured to create (or "generate") groups, or schools, or fish that may be configured to swim around in the virtual aquatic environment. In various embodiments, such fish may be configured to be attracted by certain gestures or inputs of the user, while in other embodiments, they may be configured to flee from the user's movement, as if they were afraid of the user's movement. The fish may be configured to grow and / or reproduce over time and may be configured to spontaneously or at given time intervals change into different species, such that the aquatic environment remains timely changing for the user.

[0177] At 38, the system can be configured to operate software to present an aquatic environment to the user. The aquatic environment is preferably presented to the user in full color and three dimensions such that the user perceives the environment around the user as an aquatic environment, such as an aquarium, which, in the case of a "vista", not only extends around the user's immediate environment / room but also extends through virtual connections framed or portals and into an extended vista beyond the user's immediate environment / room. Temporal features (i.e., features that change position and / or geometry over time) can be configured such that elements simulate natural movement (i.e., such as the slow aquatic movement of a virtual presented aquatic plant, the slow growth of a virtual presented aquatic coral element, and / or the creation, movement, or growth of a virtual presented fish or fish school or group). The system can be configured to suggest appropriate locations and sizes for possible portal or frame positions for the "vista", such as the location and size of the center of a vertical wall within the actual room occupied by the user, which would give the user a broad view of the virtual "vista" expansion of the virtual aquatic environment. In Figure 4 , the grid 56 of the user's room is shown mapped to a somewhat spherical "vista" volume 58 that is operatively and virtually fluid-coupled to the actual room presented to the user. Also shown are the predetermined paths 60 for the various fish presented to the user in the "vista" presentation.

[0178] In Figure 2 , at 40, the spatial computing system can be configured such that the user can interact with certain presented features. For example, when the user attempts to view an actual real-world computer monitor on an actual desktop, the user can use a user gesture or a hand-held controller 4 to wave away the presented fish school, pushing aside virtual plants and / or coral growth that might obstruct a clear view of the computer monitor or other actual elements (such as the actual desktop). The system can be configured with artificial intelligence capabilities that are configured to learn, for example, through the use of such gestures or controller instructions, that the user generally does not like virtual elements that obstruct his clear view of such elements (i.e., such as a computer monitor or an actual desktop), or virtual fish schools near the actual doorway of the room, for example, and generally "learns" to model the virtual elements according to such specific needs of a particular user. In other words, the system can be configured to keep fish, plants, coral, and other presented virtual features out of places where the user does not seem to want them. Additionally, by considering the geometry of the grid 56, the system is preferably configured to prevent virtual elements from colliding with actual elements of the room.

[0179] Figure 2Thus, it is described that the virtual object can be a swimming life object, and the augmented reality viewer can further include a swimming subroutine that is located on a storage device and executable by a processor to move the swimming life object relative to real-world objects. The data can include multiple swimming life objects, wherein the swimming subroutine is executable by the processor to move the multiple swimming life objects relative to real-world objects and relative to each other.

[0180] The at least one sensor can sense a wave motion initiated by a user, and the augmented reality viewer can further include a wave motion routine that is located on a storage device and executed by the processor to move the swimming life object in response to the wave motion sensed by the at least one sensor. The wave motion can be initiated in a target area, and the swimming object can be moved out of the target area. The at least one sensor that senses the wave motion can be a camera that detects an image of the user's hand.

[0181] The augmented reality viewer can further include a handheld controller, wherein the sensor is mounted to the handheld controller held by the user's hand.

[0182] Reference Figures 5A - 5E , shows various embodiments of a user's visual scene, characterized by actual room elements 62 and various virtual elements, such as coral groups or clusters 64, 68, 74, 78, 82, 83, seaweed plants or forests 72, fish or other animals or schools of fish or groups thereof 66, 70, 76, 77, 80, etc. Reference Figure 5A , shows a living room scene in which a user 1 wearing a spatial computing system 2 can experience a dynamic virtual aquatic environment characterized by a central virtual coral cluster 64 and relatively large and dynamic virtual fish 66. Reference Figure 5B , shows a living room scene in which another user 1 wearing a spatial computing system 2 can experience a dynamic virtual aquatic environment characterized by a different central virtual coral cluster 64 and relatively large and dynamic virtual fish 66. Reference Figure 5C , a user can be able to view a living room scene with virtual aquatic elements such as a dynamic turtle 70 and a dynamic seaweed forest 72. Reference Figure 5D , a user can be able to view a living room scene with virtual aquatic elements such as a dynamic school of fish 76, a coral cluster 74, and a central feature 78 including a sunken diver's helmet, a coral cluster, and dynamic aquatic plants. Reference Figure 5E , a user can be able to view a living room scene with virtual aquatic elements such as a dynamic jellyfish 80 and a central feature 82 including a coral cluster and dynamic aquatic plants. For each visual configuration in the visual configuration, the system can be configured to allow the user to not only adjust the coloring theme, lighting, and shadows, but also adjust the lighting placement.

[0183] Reference Figure 6, as in the above reference Figure 2 As pointed out, the system can be configured to assist a user in selecting the size and location of a framing / portal for viewing into a "vista". The user is presented with exemplary framing / portal configurations 84, 86, 88, 90, 92, 94, which the user can select based on the size and geometry of the actual environmental characteristics to which the frame or portal is "coupled" virtually. For example, if the available blank space on a vertical wall in the user's actual office is available but relatively small, the system can be configured to suggest placing a relatively small and geometrically relatively efficient frame or portal.

[0184] Reference Figure 7 , in the case of selecting a frame or portal configuration, as in reference Figure 6 described, various aspects of the "vista" can be selected, such as coloring, lighting, volume, virtual features (such as rocks and fish), etc., resulting in many customizable options, such as seven samples 96, 98, 100, 102, 104, 106, 108.

[0185] Reference Figures 8A - 8C , shows various views of a "vista" room expansion configuration. Reference Figure 8A , a user 1 wearing a head-mounted component 2 of a spatial computing system such as that shown in Figure 1 is shown in his living room 62, where a virtual frame or portal 110 has been created such that he can virtually "see" outside into an ocean "vista" volume 112, which in the depicted illustrative example is characterized by a virtual dynamic shark 114. Reference Figure 8B , a user 1 wearing a head-mounted component 2 of a spatial computing system such as that shown in Figure 1 is shown in her living room 62, where a virtual frame or portal 88 has been created such that he can virtually "see" outside into an ocean "vista" volume 112, which in the depicted illustrative example is characterized by a variety of virtual elements (such as rocks, fish, plants, and coral). Reference Figure 8C , an illustrative example of a living room 62 scene characterized by dynamic virtual elements (such as sea otters 116, schools of fish 118, and a seaweed forest 120) is shown with the "vista", in which case the "vista" extends upward through the ceiling of the actual room, giving the user a relatively deep sense of his living room being in a virtual aquatic environment.

[0186] An augmented reality viewer can include a depth creation module that is located on a storage device and executable by a processor to display a virtual object to a user in three-dimensional space on a side of a surface opposite the user, and the surface is between the user and the virtual object. The depth creation module can be executed by the processor to display a porthole in the surface through which the virtual object is visible to the user. The virtual object can be a three-dimensional virtual object.

[0187] Reference Figures 9A - 9C , as referenced Figure 2 As pointed out, the system can be configured such that a user can interact with various virtual features and such that the system can "learn" to configure the virtual presentation according to the user's needs. For example, in various embodiments, if a user is sitting in a relatively empty room that has a door and a centrally placed desk with a computer monitor on it, the user's spatial computing system can assist the user in locating coral, aquatic plants, and other features within the room, and the system can be configured to "generate" or produce fish to swim dynamically back and forth within the room. When the user is sitting at his or her desk and attempting to use the computer monitor that is located a few feet in front of the user's head, if too many fish start swimming between the user's head and the user's computer display, the user can wave them away with a hand (or use another gesture recognizable by the system), or use other controls (such as a handheld controller 4) to indicate to the system that the user does not want the fish exactly between his or her head and the monitor. In various embodiments, the system can be configured to use artificial intelligence or learning features to learn that the user does not seem to want too many fish swimming within that volume between the user's head and the computer monitor, and the system can be configured to generally keep virtual objects (such as fish, plants, or coral) outside of that volume. Similarly, the system may learn that the user generally does not like too many virtual plants or coral growing on his or her favorite window or door and may respond accordingly by learning not to place them virtually there.

[0188] Return reference Figure 1 , the system can be operatively coupled to additional resources, such as other computing systems, via the cloud or other connections. For example, the spatial computing system can be operatively and securely coupled to a mobile phone or laptop computer that has access to the user's calendar, text messages, or Internet of Things ("IoT") connected resources (such as webcams, door locks, lights, or garage doors). Now referring again to Figures 9A - 9D, such resource connectivity can be utilized in various embodiments such that one of the multiple virtual features presented virtually can be utilized to assist the user; in other words, in the configuration shown, the virtual otter 116 can be configured to swim dynamically generally like a wild otter until the user needs or desires some interaction, in which case the otter can operate similar to a personal assistant connected to relevant resources. For example, in Figure 9A the embodiment, the otter 116 tells the user via a displayed bubble message 122 that the user's lunch meeting is in 12 minutes. The system can be configured such that the user can utilize a simple gesture, such as flicking with an index finger, to clear the message and return the otter 116 to its natural dynamics. Refer to Figure 9B , the system can be configured such that the otter character 116 provides a message bubble notification 124 that the connected calendar resource indicates the user is double-booked; if the user wishes, the otter can automatically send a message using the connected resources to request rescheduling of one of the appointments.

[0189] Thus, the augmented reality viewer can include a transmission agent that is located on a storage device and executable with a processor to sense user instructions with at least one sensor, sense instructions from the user pointing at a swimming life object at the position of the swimming life object relative to a real-world object with at least one sensor, determine an outgoing communication and a resource based on the instructions, and transmit the outgoing communication to the resource.

[0190] At least one sensor for sensing communication can be a microphone suitable for receiving voice instructions from the user.

[0191] Refer to Figure 9C , the system can be configured such that the otter character 116 provides a message bubble notification 126 that the connected webcam and messaging (and / or Amazon Web / web) resources are indicating that the user has received a package; such a system can be further configured to allow the user to open an IOT garage door or other IOT-enabled access point to facilitate delivery by requesting this from the otter character 116, such as through the voice recognition tool of the spatial computing system that utilizes the mutually coupled microphone sensors 30.

[0192] Thus, the augmented reality viewer can include a transmission agent that is located on a storage device and executable with a processor to sense user instructions with at least one sensor, sense instructions from the user pointing at a swimming life object at the position of the swimming life object relative to a real-world object with at least one sensor, determine an outgoing communication and an IOT device based on the instructions, and transmit the outgoing communication to the IOT device to operate the IOT device.

[0193] At least one sensor for sensing communication can be a microphone suitable for receiving voice instructions from the user.

[0194] Reference Figure 9D , the system can be configured to use artificial intelligence and / or learning-type tools to analyze eye gaze information related to a user, which can be captured by an inward-facing camera of a head-mounted component 2 coupled to the user space computing system. Such analysis may result in provisional conclusions based on the system. For example, the user may be trying to look at something on a wall, such as an actual piece of art that is currently actually blocked by a seaweed forest or other virtual features, and it may be beneficial for the otter character 116 with personal assistant capabilities to ask the user if he wishes for the right portion of the seaweed forest to be moved a bit to allow for a better view of the actual wall, as Figure 9D suggested in the message bubble 128 shown in

[0195] The augmented reality viewer can include an artificial intelligence system that is located on a storage device and capable of being executed by a processor to sense a user's action using at least one sensor, execute a routine involving virtual objects, which routine, in response to the user's action, associates the routine with the action as an artificial intelligence cluster, determines the parameters that exist at a first time when the action is sensed, associates the parameters that exist at the first time with the artificial intelligence cluster, senses the parameters at a second time, determines whether the parameters at the second time are the same as the parameters at the first time, and if it is determined that the parameters at the second time are the same as the parameters at the first time, then execute the routine.

[0196] The at least one sensor can include an eye tracking camera, wherein the action is a waving motion initiated by the user, and the parameter is the gaze direction determined by eye tracking the user using the eye tracking camera. The action can be a waving motion initiated in a target area, and the swimming object is moved out of the target area.

[0197] The at least one sensor can include an eye tracking camera positioned to sense the user's gaze direction, and the augmented reality viewer can further include a gaze direction motion routine that is located on a storage device and executable by a processor to move a swimming life object in response to the gaze direction sensed by the eye tracking camera.

[0198] There may be multiple swimming life objects and a swimming subroutine can be executed by the processor to move the multiple swimming life objects relative to real-world objects, and the augmented reality viewer can further include a personal assistant module that is located on a storage device and capable of being executed by a processor to select a personal assistant swimming life object among the multiple swimming life objects, and move at least one swimming life object other than the personal assistant swimming life object among the swimming life objects together with the personal assistant swimming life object.

[0199] Reference Figure 10A and 10B, in various embodiments, the mathematical model can provide a basis for certain cyclic behaviors of various virtual characters (such as various types of fish), and various types of fish can have various types of general motion patterns, which can be used to generate motions that look natural to the user. As described above, in various embodiments, fish can have several states to which they can be configured to return, such as fleeing as if they are afraid, following as if they are interested, hungry, or curious, and these behaviors may be associated with factors such as the speed of the user's hand movement. For example, certain gestures or slow hand movements may be associated with feeding or friendliness / safety, such that one or more fish may be configured to follow or approach the user's hand, while fast movements or certain gestures may be associated with potential danger or fear, in which case one or more fish may be configured to flee the nearby area.

[0200] The swimming living object can be a first type of fish, and the augmented reality viewer can further include a motion module, which is located on a storage device and can be executed by a processor to articulate the body of the fish in a first back-and-forth manner. The motion module can be executed by the processor to articulate the body of a second type of fish in a second back-and-forth manner different from the first back-and-forth manner. The motion module can be executed by the processor to sense the slow hand movement of the user's hand with at least one sensor, and in response to the slow speed of the hand movement, articulate the body of the fish in the first back-and-forth manner at a low speed, sense the fast hand movement of the user's hand with at least one sensor, and in response to the fast speed of the hand movement, articulate the body of the fish in the first back-and-forth manner at a high speed. The motion module can be executed by the processor to move the first fish closer to the hand when the hand moves at a slow speed, and move the first fish away from the hand when the hand moves at a fast speed.

[0201] Reference Figure 11A , from the perspective of spatial computing resources, various configurations can be used to move the fish relative to the user credibly and efficiently. For example, a sine wave output can be utilized to modulate the position of the fish's tail and body relative to each other. Instead of setting keyframing data for each body part, the system can be configured to generally change the speed of the character without precisely mathematically calculating the velocity of each of its components, and in various embodiments, secondary motions can be added for other components. Thus, the system can be configured such that grouped or bulk motions can be generated without having to manually animate each part. Figure 11AA flowchart showing a series of animations, such as "sharp left turn" and then "sharp right turn", which can be fed into a basic motion engine. Transition code can assist in transitions between states, such as between normal straight calm swimming motion and escape state. Interpolation or animation blending and type conversion can be used for smooth transitions.

[0202] Reference Figure 11B , vertex animation can be utilized to assist in converting animated skeletal assets to animated vertex meshes using texture maps, such as those from a trademarked / creative tool obtained under the trade name Houdini (RTM), which can be utilized to process data that may be brought into an interactive development environment such as Unreal Engine (RTM) for integration into the experience to be presented to the user. Such techniques can be used to help make assets such as fish significantly more efficient to display to the user and also more efficient from a computational resources perspective. In other words, a basic 3D model may be associated with a "rig"; then the system can be configured such that the rig is not needed at runtime as the relevant vertices of the 3D model / mesh can be animated directly. For example, instead of manipulating a skeletal type framework under a fish model, the fish itself or its mesh can be moved (i.e., manipulating the mesh vs. the rig). Such efficiency allows for reduced computational overhead and may help, for example, render more fish simultaneously.

[0203] Reference Figures 12A - 12D , a similar approach can be used to not only effectively animate fish such as the clownfish asset shown as 130, but also animate features such as the sea anemone 132. Figures 12A - 12D Shows views of the clownfish 130 and the sea anemone 132 at various levels of natural replication from mesh elements (as Figure 12A shown) to full-color animated elements (as Figure 12D shown). If each sea anemone tentacle had to be animated individually rather than as described above, the computational resources required would increase significantly.

[0204] An augmented reality viewer can include a vertex animation routine that is located on a storage device and is executable by a processor to store a virtual object mesh representing a virtual object, associate a texture with the virtual object mesh, and manipulate the virtual object mesh to cause movement of the texture and the virtual object in the user's field of view. The virtual object mesh can be manipulated to connect virtual objects. The same virtual object mesh can be used multiple times to cause movement of the texture and the virtual object.

[0205] Reference Figure 13, in a mixed reality or spatial computing environment, there are many unpredictabilities because the system is configured to react dynamically with respect to the environment around the user. Before the system scans the room, it is not possible to clearly define in advance what the room around the user will look like geometrically. There are some unique challenges in animating various features, mainly because a very wide range of believable behaviors are typically preferred. For example, in a fully virtual reality system, every rock, every fish, and every coral can be carefully placed in three dimensions to work well together. This is not the case for the unknown room that the user can see and experience in augmented reality. Additionally, as described above, the system is preferably configured such that various elements such as fish can interact with the user, such as via the user's hand or eye movements, and these can also be relatively unpredictable. If every fish makes the same rotation to the left and swims away with the same vector when the user's relatively fast hand approaches, the scene will appear unnatural. Therefore, the system is preferably configured to have many varying dynamic behaviors for various states such as feeding or fleeing.

[0206] Reference Figure 14 , shows a "code blueprint" 134, such as those that can be created using an environment available under the trade name Unreal Engine (RTM). Such code blueprints represent the relatively high-level functional programming of various elements and can be used, for example, to control how the system switches between various perspective frameworks / portal configurations, as referenced above Figure 4 , 6 , 7, and 8A - 8C. For example, the system can be configured to not only locate possible general planar positions for possible perspectives and the frameworks / portals associated therewith, but also maximize the size of such perspectives and frameworks / portals, and / or center them horizontally and / or vertically with respect to the plane. Figure 15 Shows various aspects of an interface for arranging and texturing various elements, such as corals, for presentation to the user. In various embodiments, shading or painting can be performed directly on the vertices of the mesh of an object, and height map controls can be utilized to provide texture and topology for the object such that such configuration does not require polygons.

[0207] Reference Figure 16 , shows a coral cluster 65 similar to those 64, 68, 74, 78, 82 described above, and the coral cluster 65 can include components of smaller coral elements that can be used together like building blocks, with configurable rules to help make such assembly natural (e.g., in various configurations, table-driven settings can be used to control variables such as the number of coral elements that may extend from a given bedrock element, the acceptable vectorization of plants with respect to the light source, etc.). Thus, the system presentation can be iterated using various control panels without modifying the code for each adjustment.

[0208] Reference Figures 17A - 17C Regarding lighting and shadows, in various embodiments, lighting can be faked in a given scene (i.e., without simulating physics) to bypass the shading and shadowing of traditional development environments. For example, shadows can be completed via a shadowless lighting path (i.e., the "emissive channel" in Unreal (RTM) terms, or the "lightless channel" in Unity (RTM); in both cases, it generally means processing textures without calculating any lighting). The traditional overhead of the graphics pipeline can be bypassed to achieve higher visual quality within a tight performance budget, thereby resulting in an important overall computational demand efficiency. The system is configured to utilize the shadowless lighting path technique to bypass the traditional lighting engine, and other effects such as fog, optical, and / or water effects can be added to help make the perceived feeling for the user more natural. Various elements can be "instantiated" to allow for fine-tuning of the overall presentation without having to provide computational resources for each individual element. Figure 17A Shows a high-fidelity model of the diver's helmet 136; Figure 17B and 17C Shows a more efficient reproducible output of the diving helmet 136, still having a relatively high fidelity. Reference Figure 18 shows an embodiment of the core shadowless shader network configuration 138 in the form of a functional "code blueprint" for an environment such as the Unreal Engine (RTM).

[0209] Reference Figure 19 and 20 As described above, such as in Reference Figure 2 , 3 and 4, the system can be configured to utilize the geometry of the room or environment surrounding the user. Content (such as vistas or corals) can be placed using random procedural configurations, and / or a content combination that naturally fits the room can be provided, which is aesthetically pleasing to the user and at least somewhat different each time the system is used for such applications. As described above, the system can be configured to, for example, identify planar surfaces, empty surfaces, and empty volumes, which can be suitable for placing various elements with the help of the user or automatically (such as according to a predetermined general theme that can be selected by the user).

[0210] The augmented reality viewer can include a surface extraction routine that is located on a storage device and is executable by a processor to identify surfaces in real-world objects. The surface can be a two-dimensional surface of a wall or ceiling.

[0211] An augmented reality viewer may include a vista placement routine that is located on a storage device and executable by a processor to capture a space including real-world objects, represent the space as a real-world grid, collect vertical and horizontal planes from the real-world grid, filter the planes by position, orientation, dimensions, and height from the floor relative to the real-world grid, generate a blueprint that includes all content and portal frames in the vista at a selected location, and cut holes in an occlusion material of the real-world grid material so that a user can see the vista through the portal.

[0212] Reference Figure 21 , the system may also be configured to locate a position within the grid of a room, which may be a suitable anchor for an element such as a coral cluster; for example, suitable anchors may be scored and the highest scoring one may be recommended or automatically selected for the user; after scoring and selection, an element such as a coral or a coral cluster may be grown or "spawned" for the user to observe. For example, the score may be affected not only by the geometry but also by the positioning and orientation relative to the user (e.g., if a coral can be placed at a potential anchor point with an orientation relatively perpendicular to the floor of the room surrounding the user rather than, for example, at a 45-degree angle from the floor of the room surrounding the user, the system may be configured to score the potential anchor point higher).

[0213] The virtual object may thus be a coral cluster. The augmented reality viewer may include a coral cluster generator that is located on a storage device and executable by a processor to determine a volume, perform line tracing at random points from the maximum height of the volume to the floor within the volume, determine if a valid position is identified by the line tracing, if a valid position is identified, perform box tracing to test if a random cluster fits without overlapping the world grid when trying different scales and rotations and generating a score for each placement, determine a selected placement with the highest score among the scores, and generate the coral cluster into the placement with the highest score. The augmented reality viewer may thus include a vista placement routine that is located on a storage device and executable by a processor to place a vista, wherein the volume is constrained by the vista.

[0214] Reference Figures 22A - 24 , one of the challenges of the various system configurations described herein is to create interesting and believable interactions between the user and augmented reality elements that may be overlaid on or within the actual world where the user is located. Preferably, various elements such as fish that are animated and configured to interact with the user may be configured to have natural, non-repetitive behaviors that are meaningful to the user relative to the time domain (i.e., they do not swim too fast, they do not switch from a "fleeing" state to a "discovering / curious" state too quickly or too frequently, etc.); in addition, as Figure 1The systems shown typically have limited computing resources. Refer to Figures 23A - 23B , in various embodiments, one technique for achieving computational efficiency is to write certain code elements in C++ rather than in functional code blueprints. Refer to Figure 24 , a three-dimensional navigation and pathfinding plug-in product that can be utilized with an environment such as the Unreal Engine (RTM) can be used to assist in navigating and developing movement paths for various elements such as fish; such plug-ins allow the development team to focus more on higher-order animation issues such as fish-feeding behavior, exploration behavior, and interaction with the user (such as with the user's hand movements).

[0215] Refer to Figure 25 , standard tools can be utilized to visualize the utilization of various computational threads in relation to time, enabling the development team to further optimize the system. Figure 26 Illustrates various aspects of learning that may be applicable to certain development processes and teams.

[0216] One or more users may be able to share their virtual environment with one or more other users such that multiple users experience the same virtual environment characteristics from their own viewing perspectives in a multi-location “navigable world” type of configuration, as described in the incorporated applications above. For example, if a user located in a New York office has virtual aquarium characteristics displayed around him in his office, and if another user from San Francisco is virtually transported into that New York office and virtual world, the user from San Francisco preferably should be able to see the virtual aquarium characteristics from the virtual location / orientation of that San Francisco user within the New York room.

[0217] Refer to Figure 27 , and also refer to Figure 11B , in various embodiments, for an augmented reality aquarium experience, the creative team can re-think animation, technology, and rigging to work with AI pathfinding, which can change states and navigate grid spaces that may be occupied by one or more users. For example, the animation and / or rigging team can invest time in developing standard fish rigging 140 and animation pipelines 142 for standard fish 130, with standardized animation and user interface (“UI”) controls, automatic swimming functionality, automatic level of detail (“LOD”) functionality, fish-eye “observation” tools, customizable tentacle and fin rigging, hybrid spaces that trigger various behaviors, and / or vertex animation texture baking tools (i.e., effectively transferring detail from one model to another). Such features can provide designers with great flexibility, as well as the ability to re-use animations and adapt them to the desired behavior of the creature. In certain configurations, we have found that using tools such as “Automatic LOD” from Unreal Engine 4 (RTM) helps to dynamically reduce the number of bones and polygons when modifying the performance budget.

[0218] Refer toFigure 28 , and return the reference Figure 11A , one challenge developers face involves building and testing animation sets to work with artificial intelligence (“AI”) pathfinding solutions. For example, a fish element can have a set of “blend spaces” that have a set of specific animations that can be triggered by an AI system based on various logic triggers and specific states (such as swimming speed, hovering, searching, fleeing, feeding, etc.). The blend space functionality can be configured to allow developers to specify inputs, animations, and how to blend between animations using the inputs. As Figure 28 shown in the developer panel 144 in, various motion states are shown on the left, while in the middle, “audio animation notifications” and “overall animation notifications” triggers for behaviors such as feeding, turning, etc. are shown.

[0219] Reference Figure 29 , we rely on binding and manual keyframing of animations, but we have also found some valuable solutions for certain environment-based creature configurations that help bake animation data into textures using a pipeline that utilizes the above-mentioned Houdini (RTM) game tools, Maya (RTM), and Unreal Engine4 (RTM). Using such tools, we can develop a final result that provides the required performance leeway, especially in scenarios where it is necessary to balance the loads on the central processing unit (CPU) and the graphics processing unit (GPU) (including texture streaming), such as when moving using Vulkan 3.1 (RTM). Figure 29Displays the shown shipwreck scene 146, where the fish school elements 76, 77 in the mid - ground and background are displayed using the above - mentioned baking technique. Since developers may wish to create a large amount of content in a given experience, having a scene such as real - time flocking of birds may be impractical. For example, for a large number of virtual fish, we have found a solution to reduce the size of the baked texture. For each member fish in a large school of fish, their swimming cycles can be exported from Maya (RTM) to an FBX file; then each of these caches can be brought into Houdini (RTM), where individual vertex animation textures (or "VATs") can be created and then inserted into the Houdini soft - body framework blend vertex shader tool for deformation. The FX artist can generate simulations in Houdini and attach each individual particle to a joint, which can be used to drive each vertex - animated fish through the virtual water. To develop the swimming content of fish and creatures, rigging and animation control can be standardized across a group or all of the fish. An automatic swimming function (such as a procedural sine - wave swimming function) can be developed. A traditional LOD rigging system or an automatic LOD function can be used. A fish - eye viewing tool can be developed to help drive the gaze of the fish. Other specialized environmental creatures (such as eels, crabs, octopuses, seahorses, turtles, rays, sharks) can be developed. Blend spaces and in - situ rotation offsets can be used in combination with AI to trigger behaviors such as speed or various states. For example, an example animation rigging can be shown in Maya, where someone controls it in Maya, and / or the required animations and blend spaces can be set up for the technical team to integrate with AI pathfinding. A vertex - animated fish pipeline function can be developed to help provide flexibility and functionality for a given scene with a specific graphics and computational budget (such as a scene associated with a mobile platform).

[0220] Reference Figures 30A - 30H , and also returns the reference Figure 5A 、 5B 、5D, 5E, and 16, show various coral cluster configurations 83, 152, 154, 156, 158 and their components 148, 150. In various embodiments, priority is given to giving the user experience the opportunity for dynamic placement and procedurally - driven underwater reef experiences. Therefore, our challenge is to build a variable - driven coral / rock generation and placement system using blueprints combined with the real world. In one embodiment, we have chosen a computationally - friendly "hybric" method, where we load and place pre - built rock formations and sand bases, and then "generate" corals and seaweeds on them using our coral generation and placement system configuration. In one embodiment, before doing so, the art team has established a cohesive set of coral rock formations, such as Figure 30C and 30DThose shown in 148, 150 will serve as the basis for different layouts of "kit bashing".

[0221] A system for dynamically generating and placing corals was developed using blueprints. Although blueprints were initially considered more for early prototyping and creating a proof of concept for gameplay, our technical art team derived important functionality from them. Refer to Figure 30H , various vertices 160 are shown as they are verified for simulating potential coral growth. In various embodiments, for example, it may be desirable to have coral clusters with random behavior each time the user returns to the experience.

[0222] Refer to Figure 31A and 31B , images of embodiments of a coral generator object system 162 and a coral growth control table 164 are shown. The system embodiments are capable of adopting a number of notable features, such as: a data table approach to set individual population and species maximum counts, spawning types, height-based percentages, and other categorical variables; vertex crawling and raycasting algorithms for checking placement feasibility, growth, and caching valid points to a file; a runtime coral static mesh loop that first calculates the shadow pass and then creates instantiated static meshes for all corals; and a collision-based exclusion configuration that allows developers to simply place box colliders in areas where they do not want certain species to grow. To assist with the integration and grounding of coral clusters in the user's space, the system may include a set of smaller "support" rock and sand elements with dynamically moving seaweed on them. The generator manager (on the C++ core game side) can then be configured to place these support elements around the main coral clusters based on the available floor planes detected by the system and the space limitations of each user's room. This also helps to ensure that each game space has a unique layout. In various embodiments, although each of the multiple core vista environments can be pre-built, an interchangeable framework and dynamic AI and FX-driven creatures can be utilized to make these pre-built vistas feel completely unique. Each layout can be dynamic and depend on the size of the grid room; for example, Figure 30FShows a procedurally generated coral cluster 154 surrounded by smaller rock pieces and seaweed elements. This system can be configured to use a runtime coral static mesh loop that first calculates shadow passes using actor tags and then immediately creates instantiated static meshes for all corals to reduce draw calls and overhead; growth can be data table-driven and can include vertex-based growth, raycast searches, and caching of valid points, as well as height-based and collision-based species inclusion / exclusion. In various embodiments, we have built and employed reusable and procedurally generable rock and coral reef elements that can be rotated and snapped into random configurations; we ground and integrate the content by using subtle drops on the sand and adding randomly placed support rocks with seaweed around the cluster; and we use Maya (RTM), Zbrush (RTM), Substance Designer (RTM), and Painter (RTM) tools to create unique and tilable procedural and handcrafted assets.

[0223] An augmented reality viewer can include a coral generator system that is located on a storage device and is executable with a processor to store at least a first type of first coral element on the storage device and build a coral cluster from a plurality of coral elements including the first coral element.

[0224] The coral generator system is executable with a processor to build a coral cluster from a plurality of first coral elements.

[0225] The coral generator system is executable with a processor to store at least a second type of second coral element on the storage device, where the plurality of coral elements includes the second coral element.

[0226] The coral generator system can be executed by a processor to determine coral cluster settings, where the processor constructs a coral cluster based on the settings. The settings can be the detected available space, and multiple coral elements are selected based on the available space. The coral generator system can be executed by a processor to simulate ambient light, where the settings are the ambient light. Multiple coral elements can be selected based on the ambient light. The orientation of the coral elements can be selected based on the ambient light. The coral generator system can include a data table with multiple coral cluster settings on a storage device, where the coral generator system constructs a coral cluster according to the multiple coral cluster settings. The coral cluster settings can include at least one of the following: population; species maximum count; generation type; and height-based percentage. The coral generator system may include vertex crawling and ray casting algorithms to check placement feasibility, growth, and cache valid points to a file. The coral generator system can include a runtime coral static mesh loop that first calculates the shadow channel and then creates instantiated static meshes for all coral clusters. The coral generator system can include a collision-based exclusion configuration that places box colliders where certain species should not grow.

[0227] Figure 32 Instructions residing on the storage device as described above are shown, including a dataset 200 of virtual objects, a display module 202, a data stream generator 204, a light generator 206, a positioning adjustment module 208, a swimming subroutine 210, a wave motion routine 212, a retrieval agent 214, a transmission agent 216, an artificial intelligence system 218, a gaze direction motion routine 220, a personal assistant module 222, a movement module 224, a surface extraction routine 226, a depth creation module 228, a perspective placement routine 230, a vertex animation routine 232, and a coral cluster generator 234.

[0228] Various example embodiments of the present invention are described herein. These examples are referenced in a non-limiting manner. They are provided to illustrate more broadly applicable aspects of the present invention. Various changes can be made to the described invention and equivalents can be substituted without departing from the true spirit and scope of the present invention. In addition, many modifications can be made to adapt a particular situation, material, composition of matter, process, process act, or step to the purpose, spirit, or scope of the present invention. Moreover, as will be understood by those skilled in the art, each of the various variations described and illustrated herein has discrete components and features that can be easily separated from or combined with the features of any of the several other embodiments without departing from the scope or spirit of the present invention. All such modifications are intended to fall within the scope of the claims associated with this disclosure.

[0229] The present invention includes methods that can be performed using the present device. The method can include the action of providing such a suitable device. Such provision can be performed by an end user. In other words, the "providing" action only requires the end user to obtain, access, approach, locate, set up, activate, power on, or otherwise act to provide the necessary device in the present method. The methods described herein can be performed in any order of the logically possible recited events and in the recited order of events.

[0230] Exemplary aspects of the present invention and details regarding material selection and manufacturing have been set forth above. As for other details of the present invention, these can be understood in conjunction with the patents and publications cited above and what is known or understood by those skilled in the art. Regarding the method-based aspects of the present invention, this equally applies in terms of additional actions that are commonly or logically employed.

[0231] Furthermore, although the present invention has been described with reference to several examples optionally combining various features, the present invention is not limited to what is described or indicated as being contemplated for each variant of the present invention. Various changes can be made to the described invention and equivalents can be substituted (whether cited herein or not included for some brevity) without departing from the true spirit and scope of the present invention. Additionally, in cases where a range of values is provided, it should be understood that each intermediate value between the upper and lower limits of that range and any other specified or intermediate value within that range is included within the present invention.

[0232] Furthermore, any optional features of the described variants of the invention can be presented and claimed independently, or in combination with any one or more of the features described herein. References to singular items include the possibility of the presence of a plurality of the same items. More specifically, as used herein and in the associated claims, the singular forms "a", "an", "the", and "said" include plural referents unless specifically stated otherwise. In other words, the use of an article permits "at least one" of the subject items described above and the claims associated with the present disclosure. Further, it should be noted that such claims can be drafted to exclude any optional elements. Thus, this statement is intended as a basis for the use of proprietary terms such as "only", "merely", etc. or the use of "negative" limitations in connection with the recitation of claim elements.

[0233] Without the use of such proprietary terms, the term "comprising" in the claims associated with the present disclosure should be allowed to include any additional elements, regardless of whether a given number of elements are recited in such claims or whether a change adding a feature can be considered to change the nature of the elements set forth in those claims. Except as specifically defined herein, all technical and scientific terms used herein will be given the broadest possible meaning as commonly understood while maintaining the validity of the claims.

[0234] The scope of the present invention is not limited to the provided examples and / or the subject specification, but only by the scope of the claim language associated with this disclosure.

[0235] Although certain exemplary embodiments have been described and illustrated in the drawings, it should be understood that these embodiments are merely illustrative and not limiting of the invention, and the invention is not limited to the specific structures and arrangements shown and described, as modifications can be made by those of ordinary skill in the art.

Claims

1. An augmented reality viewer, comprising: At least one sensor positioned to sense the position of at least one real-world object among a plurality of real-world objects and a wave motion initiated by a user in a target area; A storage device; A data set on the storage device, which includes swimming life objects; A processor connected to the storage device; A display module located on the storage device and executable by the processor to determine a desired display of the swimming life objects relative to the position of at least one of the real-world objects; A data stream generator located on the storage device and executable by the processor to generate a data stream based on the data and the desired display; A light generator connected to the processor to receive the data stream and generate light based on the data stream; A display device positioned to receive the generated light and display the light to the user, wherein the light creates a rendering of the swimming life objects, the rendering being visible to the user and rendered according to the desired display; A swimming subroutine located on the storage device and executable by the processor to move the swimming life objects relative to the real-world objects; A wave motion routine located on the storage device and executable by the processor to move the swimming life objects out of and away from the target area in response to and due to the wave motion sensed by the at least one sensor in the target area; A motion module located on the storage device and executable with the processor to: Sense a slow hand movement of the user's hand in the target area with the at least one sensor; In response to the slow speed of the hand movement, the body of the swimming life object is connected in a first back-and-forth manner at a low speed; Sense a fast hand movement of the user's hand with the at least one sensor; In response to the fast speed of the hand movement, the body of the swimming life object is connected in the first back-and-forth manner at a high speed; and When the hand moves at the slow speed and due to the slow speed, move the swimming life object to approach the hand, and when the hand moves at the fast speed and due to the fast speed, move the swimming life object to flee from the hand.

2. The augmented reality viewer according to claim 1, wherein, The data includes a plurality of swimming life objects, wherein the swimming subroutine can be executed by the processor to move the plurality of swimming life objects relative to the real-world objects and relative to each other, and the wave motion routine, which is located on the storage device and executable by the processor to move the plurality of swimming life objects out of and away from the target area in response to and due to the wave motion sensed by the at least one sensor in the target area.

3. The augmented reality viewer according to claim 1, wherein, The at least one sensor that senses the wave motion is a camera that detects an image of the user's hand in the target area.

4. The augmented reality viewer according to claim 1, further comprising: A handheld controller, wherein the sensor is mounted on the handheld controller held by the user's hand to sense the user's hand in the target area.

5. The augmented reality viewer according to claim 1, further comprising: An artificial intelligence system located on the storage device and executable by the processor to: Sense wave motion; In response to sensing the wave motion, associate the movement of the swimming life object with the wave motion as an artificial intelligence cluster; Determine the parameters existing at the first time when an action is sensed; In response to determining the parameters at the first time, associate the parameters existing at the first time with the artificial intelligence cluster; Sense parameters at a second time; In response to determining the parameters at the second time, determine whether the parameters at the second time are the same as the parameters at the first time; And If it is determined that the parameters at the second time are the same as the parameters at the first time, then in response to determining that the parameters at the second time are the same as the parameters at the first time, execute the wave motion routine.

6. The augmented reality viewer according to claim 5, wherein, The at least one sensor includes an eye-tracking camera, wherein the parameters at the first time and the parameters at the second time are each a gaze direction determined by eye-tracking the user using the eye-tracking camera.

7. The augmented reality viewer according to claim 1, wherein, There are a plurality of swimming life objects, and the swimming subroutine can be executed by the processor to move the plurality of swimming life objects relative to the real-world object. The augmented reality viewer further includes: A personal assistant module located on the storage device and executable by the processor to: Select a personal assistant swimming life object among the plurality of swimming life objects; and In response to the user's instruction, move at least one of the swimming life objects other than the personal assistant swimming life object together with the personal assistant swimming life object.

8. The augmented reality viewer according to claim 1, wherein When the user views at least one real-world object among the real-world objects, the display device displays the generated light to the user.

9. The augmented reality viewer according to claim 8, wherein, The display device is a see-through display device that allows light from the at least one real-world object to reach the user's eyes.

10. The augmented reality viewer according to claim 1, further comprising: A head-mounted structure adapted to be worn on the user's head, wherein the display device is mounted on the head-mounted structure, and the at least one sensor is a type of sensor adapted to sense the movement of the display device caused by the movement of the user's head; and A positioning adjustment module that can be executed by the processor to adjust the position of the swimming life object so that the swimming life object remains stationary relative to the at least one real-world object within the user's field of view.

11. An augmented reality visualization method, comprising: Sensing the position of at least one real-world object among a plurality of real-world objects with at least one sensor; Storing data including swimming life objects on a storage device; Use a processor to determine the desired display of the roaming living object relative to at least one of the real-world objects; Generate a data stream by the processor based on the data and the desired display; Generate light by a light generator based on the data stream; Use a display device to display the generated light to the user, wherein the light creates a rendering of the roaming living object, the rendering being visible to the user and being rendered according to the desired display; Move the roaming living object relative to the real-world object by the processor; Sense a wave motion initiated by the user in a target area with the at least one sensor; In response to and due to the wave motion sensed by the at least one sensor in the target area, move the roaming living object out of and away from the target area by the processor; Sense a slow hand movement of the user's hand in the target area with the at least one sensor; In response to the slow speed of the hand movement, connect the body of the roaming living object in a first back-and-forth manner at a low speed by the processor; Sense a rapid hand movement of the user's hand with the at least one sensor; In response to the rapid speed of the hand movement, connect the body of the roaming living object in the first back-and-forth manner at a high speed by the processor; When the hand moves at the slow speed and due to the slow speed, move the roaming living object closer to the hand by the processor; and When the hand moves at the rapid speed and due to the rapid speed, move the roaming living object away from the hand by the processor.

12. The method according to claim 11, wherein, The data includes a plurality of roaming living objects, and the method further includes: Move the plurality of roaming living objects relative to the real-world object and relative to each other by the processor, and move the plurality of roaming living objects out of and away from the target area in response to and due to the wave motion sensed by the at least one sensor in the target area.

13. The method according to claim 11, wherein, The at least one sensor for sensing the wave motion is a camera that detects an image of the user's hand in the target area.

14. The method according to claim 11, wherein, The sensor is mounted on a handheld controller held by the user's hand to sense the user's hand in the target area.

15. The method according to claim 11, further including: Sense the wave motion with the at least one sensor; In response to sensing the wave motion, associate the movement of the roaming living object with the wave motion as an artificial intelligence cluster by the processor; Determine parameters existing at a first time when an action is sensed by the at least one processor; In response to determining the parameters at the first time, associate the parameters existing at the first time with the artificial intelligence cluster by the processor; Sense parameters at a second time with the at least one sensor; In response to determining the parameters at the second time, determine by the processor whether the parameters at the second time are the same as the parameters at the first time; And If it is determined that the parameter at the second time is the same as the parameter at the first time, in response to determining that the parameter at the second time is the same as the parameter at the first time, the movement of the roaming living object is performed.

16. The method according to claim 15, wherein, The action is a wave movement initiated by the user, and the parameter at the first time and the parameter at the second time are each the gaze direction of the user.

17. The method according to claim 11, further comprising: Moving, by the processor, a plurality of roaming living objects relative to the real-world object; Selecting, by the processor, a personal assistant roaming living object from among the plurality of roaming living objects; And In response to an instruction from the user, moving, by the processor, at least one roaming living object other than the personal assistant roaming living object among the roaming living objects together with the personal assistant roaming living object.

18. The method according to claim 11, wherein, When the user views at least one real-world object among the real-world objects, the display device displays the generated light to the user.

19. The method according to claim 18, wherein, The display device is a see-through display device that allows light from the at least one real-world object to reach the user's eyes.

20. The method according to claim 11, further comprising: Sensing, by the at least one sensor, movement of the display device caused by movement of the user's head; And Adjusting the position of the roaming living object so that the roaming living object remains stationary relative to the at least one real-world object within the user's field of view.

Images