An authoring system for interactive virtual reality environments

Abstract

The non-transitory computer-readable medium stores computer-executable code that, when executed by a processor, causes the processor to perform the following steps: projecting a mixed reality environment onto a headset, the mixed reality environment having objects, each of the objects created by object computer code having properties that define how the object interacts within the mixed reality environment; displaying an action menu on a virtual feedback screen having a list of predefined action modules in response to a first input from at least one user interface device, the action modules including modules of computer-executable code having one or more functions, the one or more functions being configured to affect one or more attributes of a first of the objects in the mixed reality environment; and updating the object computer code of the first object to include the selected action module.

Description

Technical Field

[0001] [Incorporation by Reference] This patent application claims priority to U.S. Patent Application No. 16 / 549,609, filed on August 23, 2019, and to a provisional patent application identified by U.S. Provisional Patent Application No. 62 / 884,020, filed on August 7, 2019, the entire contents of both patent applications being hereby incorporated herein by reference.

Background Art

[0002] Virtual reality hardware has advanced and become more capable, the cost of virtual reality hardware has decreased, and the interest in virtual reality and its availability in various industries has increased. However, many people outside the field of computer programming do not have the skills necessary to program virtual reality environments.

[0003] Virtual reality (VR) is a computer-generated simulation of a three-dimensional environment that includes a physics engine that can be interacted with by a person in an apparently real or physical way using a virtual reality system. A virtual reality system includes a multi-projection environment that simulates the physical presence of a user in a virtual environment, a screen that generates images, sounds, and other sensations, and may include an interface device through which the user can provide input to the virtual environment. A virtual reality system enables a user to look around an artificial world, move around it, and interact with virtual objects. This effect is generally created by a VR headset that consists of a head-mounted display with a small screen in front of the eyes, but can also be created through a specially designed room with multiple large screens. Virtual reality generally incorporates auditory and video feedback, but may also include other environmental simulations such as sensory and force feedback through haptic technology.

[0004] Virtual reality environments, also known as virtual environments or VR applications, can include, for example, entertainment such as games and multimedia, or training such as classroom education or virtual training. Programming a virtual reality environment is an even more complex, intricate, and extremely time-consuming task, as it involves rendering a composite 3D environment on a 2D computer monitor. Furthermore, advanced 3D modeling, texturing, and lighting require skilled techniques and are time-consuming processes.

[0005] Currently, building a virtual reality environment generally requires a team of individuals with broad and advanced knowledge and skills, including at least graphic arts design, computer programming, 3D mathematics and physics, and an understanding of acoustic engineering. Adding individuals with any one of these skills can be costly depending on demand and needs. Furthermore, due to the broad and advanced knowledge and skills required to build a virtual reality environment, the adoption of virtual reality is limited to areas where individuals with these skills, or teams with these skills, can be assembled.

[0006] What is needed is a system and method that provides users with the ability to create and modify virtual reality environments that are simple and also exist within editable virtual reality environments. Such a system provides users with the ability to create and modify virtual reality environments that are simple and also exist within editable virtual reality environments, which are the subject of this disclosure. [Overview of the project]

[0007] The problem of creating and modifying a virtual reality environment that is both simple and editable within another virtual reality environment is solved using the system described herein.

[0008] In one embodiment, the disclosure describes an electronic device comprising a processor, a headset, one or more user interface devices, and a non-temporary computer-readable medium. The headset has a virtual feedback screen. The non-temporary computer-readable medium stores computer-executable code, when executed by the processor, causing the processor to project a mixed reality environment onto the headset. The mixed reality environment has one or more objects. Each of the one or more objects is created by object computer code having properties that define how the one or more objects interact within the mixed reality environment. In response to a first input from at least one user interface device, an action menu is displayed on the virtual feedback screen having a predefined list of action modules. An action module includes a module of computer-executable code having one or more functions, the one or more functions configured to affect one or more attributes of a first object of one or more objects in the mixed reality environment. The object computer code of the first object is updated to include the selected action module. [Brief explanation of the drawing]

[0009] The accompanying drawings incorporated into and constituting parts thereof illustrate one or more implementations described herein, and these implementations are described in conjunction with the descriptions. The drawings are not intended to be to exact scale, and certain features and views of the drawings may be exaggerated to scale or, in the case of schematic diagrams of interest, clarity and conciseness. Components may not be labeled in each drawing. Similar reference numbers in the drawings may represent and refer to the same or similar elements or functions.

[0010] [Figure 1] This is an exemplary embodiment of a virtual reality world authoring system.

[0011] [Figure 2] A block diagram of an exemplary embodiment of a vehicle object.

[0012] [Figure 3] A flowchart of an exemplary embodiment of object configuration processing.

[0013] [Figure 4] A block diagram of an exemplary embodiment of a wheel object.

[0014] [Figure 5] A top view of a composite object formed by the shapes of the vehicle object in FIG. 2 and the wheel object in FIG. 4.

[0015] [Figure 6] A block diagram of an exemplary embodiment of an object relationship model forming the composite object in FIG. 5.

[0016] [Figure 7] An exemplary embodiment of a hierarchical object relationship model forming the composite object in FIG. 5.

[0017] [Figure 8] A screenshot of an exemplary embodiment of a virtual environment showing an exemplary room setting menu.

[0018] [Figure 9] A screenshot of an exemplary embodiment of a virtual environment showing an exemplary tool selection menu.

[0019] [Figure 10] An exemplary embodiment of a network diagram for multi-user access to a virtual environment.

[0020] [Figure 11] A screenshot of an exemplary embodiment of a virtual environment showing an exemplary scriptable menu.

[0021] [Figure 12] It is a schematic diagram of an exemplary embodiment of a script builder.

[0022] [Figure 13] It is a schematic diagram of an exemplary embodiment of a virtual environment.

[0023] [Figure 14] It is an exemplary embodiment of a computer system that implements a virtual reality world authoring system according to the present disclosure.

Mode for Carrying Out the Invention

[0024] Before explaining at least one embodiment of the present disclosure in detail, it should be understood that the present disclosure is not limited to the details of the structures, experiments, exemplary data and / or arrangements of components described in the following description or shown in the drawings, unless otherwise specified in its application.

[0025] Other embodiments of the present disclosure can be made and can be trained or executed in various ways. For example, changing the scale of a housing structure may be used as an example, but the method and system may be used to evaluate other characteristics of other artificial objects (examples include, but are not limited to, changes in the structure installation area or structure area), and non-exclusive examples thereof include other types of buildings such as industrial buildings or commercial buildings, and infrastructure such as roads, bridges, utility lines, utility towers, etc. Also, it should be understood that the phrases and terms adopted in this specification are for the purpose of explanation and should not be regarded as limiting.

[0026] The terms “comprises,” “comprising,” “includes,” “including,” “has,” and “having,” as used herein, or any other variations thereof, are intended to cover non-exclusive inclusion. For example, unless otherwise stated, a process, method, article, or apparatus having a list of elements is not necessarily limited to these elements alone, but may include other elements not expressly described or specific to such process, method, article, or apparatus.

[0027] Furthermore, unless explicitly stated otherwise, "or" refers to inclusion, not exclusion. For example, condition A or B is satisfied by one of the following: A is true (or exists) and B is false (or does not exist); A is false (or does not exist) and B is true (or exists); and both A and B are true (or exist).

[0028] Furthermore, the use of "a" or "an" is employed to describe elements and components of embodiments herein. This is done solely for convenience and to give a general meaning to the concepts of the invention. This description should be read as including one or more, and singular also includes plural unless it is evident that it means otherwise. Furthermore, the use of the term "plural" means to convey "more than one" unless the opposite is explicitly stated.

[0029] As used herein, modifiers such as “substantially,” “about,” “approximately,” and combinations and variations thereof are intended to include not only the exact quantity or value they modify, but also any deviations from them that may result from, for example, calculation tolerances, calculation errors, manufacturing tolerances, measurement errors, damage, stresses across various parts, and combinations thereof.

[0030] Any reference to “one embodiment,” “embodiment,” “several embodiments,” “one example,” “for example,” or “example,” as used herein, means that a particular element, function, structure, or characteristic described in relation to an embodiment is included in at least one embodiment and may be used in relation to other embodiments. In various places within this specification, the phrases “in several embodiments” or “one example” do not necessarily refer to all identical embodiments, for example.

[0031] The use of ordinal terminology (i.e., "1st," "2nd," "3rd," "4th," etc.) is solely for the purpose of distinguishing between two or more items and, unless otherwise specified, does not imply that another item is above a given item, or any sequence, order, or importance relative to any additional order.

[0032] The use of the terms "at least one" or "one or more" is understood to include one, as well as any number greater than one. Furthermore, the use of the phrase "at least one of X, V, and Z" is understood to include X alone, V alone, Z alone, as well as any combination of X, V, and Z.

[0033] As used herein, a circuit may be an analog and / or digital component, or one or more suitably programmed processors (e.g., microprocessors) and associated hardware and software, or hardwired logic. A “component” may perform one or more functions. The term “component” may include hardware such as processors (e.g., microprocessors), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), combinations of hardware and software, and / or similar. As used herein, the term “processor” means a single or multiple processors operating independently or together to collectively perform a task.

[0034] A programming language is a set of commands, instructions, and other syntax used to create software. The language used by programmers to write code is called a "high-level language." High-level language code is designed to be read and understood by programmers and is compiled into a "low-level language" that is directly recognized by computer hardware as computer-readable instructions. This allows programmers to write source code in a more natural way using logical words and symbols. Examples of high-level languages ​​include C++, Java®, Perl, and PHP. Low-level languages ​​include assembly language and machine language.

[0035] When software is executed by one or more components, it may include one or more computer-readable instructions that cause the components to perform specific functions. It should be understood that the algorithms described herein may be stored on one or more non-temporary computer-readable media. Exemplary non-temporary computer-readable media may include random-access memory, read-only memory, flash memory, and / or similar. Such non-temporary computer-readable media may be electrical-based, magnetic-based, optical-based, and / or similar. A software module is a reusable portion of computer-executable code having one or more specific functions.

[0036] A virtual reality system includes a device that generates images, sounds, and other sensations that simulate the user's physical presence in a virtual environment, and the user may include an interface device that allows them to provide input to the virtual environment. The virtual reality system may further include processing components and any environment simulator configured to affect one of the user's five senses, for example, but not limited to, devices configured to generate olfactory stimuli, directional stimuli, or force stimuli. The virtual reality system may include, but not limited to, Oculus Quest, Oculus Go, and Oculus Rift (Oculus VR, a division of Facebook® Inc., located in Menlo Park, California), SAMSUNG® HMD Odyssey+ (Samsung Group and Samsung Electronics, San Jose, California), or the Google® Cardboard VR platform (Google, LLC, Mountain View, California). Augmented reality systems may include, but are not limited to, Microsoft® HoloLens (Microsoft® Corporation, Redmond, Washington) or Vuzix Blade (Vuzix Corporation, West Henrietta, New York). The term "virtual reality headset" should be interpreted as including a virtual reality system unless the context makes it clear that "virtual reality headset" is intended to refer solely to a headset unit.

[0037] The virtual environment may include an avatar, i.e., a representation of the user within the virtual environment. The avatar is a virtual reality object and may be a realistic representation of the user, such as a photograph, a general representation of the user, or a representation of a device controlled by the user. In some embodiments, the avatar may move in a manner that mirrors or approximates the user's movements. By mirroring or approximating the user's movements, the avatar further enhances the virtual environment by providing visual feedback to the user's real-world movements and creating a more immersive experience for the user.

[0038] The virtual environment may include a physics engine. The physics engine is a module of computer code that simulates physical and mass interactions with objects in the virtual environment. The physics engine may provide collision detection, i.e., a module of computer code used to determine whether a first object is intersecting with a second object, and raises an event when the intersection status changes. For example, if an attached first object (described in more detail below) having a collision trigger scriptable action intersects with a second object, collision detection raises an input event, and if the first and second objects are no longer intersecting, collision detection raises a termination event. Each event has one or more attributes, which may include the location of the collision and the characteristics of each object at the time of the collision, e.g., the velocity of the object, the mass of the object, and the material of the object. Furthermore, the physics engine may provide dynamic simulations, i.e., simulations of forces acting on one or more objects in the virtual environment, e.g., gravity, or the structure of materials.

[0039] As used herein, the term virtual reality further includes supplemental realities such as augmented reality and mixed reality. Augmented reality, an interactive experience of a real-world environment in which objects present in the real world are enhanced by computer-generated perceptual information, which may include multiple sensory modalities, including visual, auditory, tactile, somatosensory, and olfactory; mixed reality, a fusion of the real and virtual worlds that creates a new environment; and visualizations in which physical and digital objects coexist and interact in real time may use the same or similar equipment as a virtual reality system. Furthermore, virtual environments may be interchangeable between supplemental reality and virtual reality systems, respectively.

[0040] A repository is a database or other file storage system stored locally in the virtual environment generation component, on a computer within the local network, or on a computer or server connected to the Internet, otherwise accessible and available to user 14 with permission from the repository owner. The repository owner is an entity that creates or possesses all rights to the repository, including the right to delete the repository, copy the repository, edit the repository, or share the repository with user 14, while varying the degree of permission granted to user 14. Types of repositories include online repositories 304, personal repositories, and core repositories. An online repository 304 (described in detail below) is a repository that can store rooms, objects, and / or items available for reuse by access provided by the owner to any user. An online repository 304 may or may not be connected to the Internet, but it is available via network 420 (described in detail below). An online repository 304 may reside on a server either within the local network or on the Internet. A personal repository is owned by user 14 and is not accessible to other users unless user 14 grants access. Furthermore, objects, action scripts, scriptable actions, and other items may not be copied or edited by other users without permission from the owner, user 14. The virtual reality world authoring system 10 also includes a core repository. The core repository is a repository of rooms, objects, and / or items provided by the virtual reality world authoring system 10, owned, and included by default for user 14. User 14 has permission to copy, then modify, and use objects in the core repository while the user is creating a virtual environment.

[0041] Referring here to Figure 1, shown and specified therein by reference no. 10 is an exemplary embodiment of a virtual reality world authoring system having a user 14 using a VR headset 18 that displays a virtual environment 22 together with an object 26. Also shown is a virtual environment generation component 28 that houses one or more computer applications having VR design characteristics, including an object builder 30, an environment construction tool 34, a script builder 38, a trigger object 42, a physics engine 46, and a bidirectional state API 50.

[0042] In one embodiment, user 14 is not a programmer and does not know a programming language. User 14 uses a VR world authoring system to build a virtual environment 22 without writing computer code. User 14 may build the virtual environment 22 alone or may work with additional users 14. When working with others, user 14 may work on a local copy of the virtual environment 22 or in a virtual environment 22 shared with other users 14.

[0043] The VR headset 18 is a head-mounted device that covers the user's eyes for an immersive three-dimensional experience. The VR headset 18 may be self-contained; that is, it may have a virtual environment generation component having non-temporary memory for storing computer code and a processor that generates a virtual environment 22 when the computer code is executed. In other embodiments, the VR headset 18 may be isolated from the virtual environment generation component, or it may communicate with the virtual environment generation component through a communication component. The VR headset 18 may include other components such as a human interface device or an environment simulator.

[0044] A communication component is a component of a VR headset having circuitry or other hardware that implements a communication protocol that provides communication between the VR headset 18 and other components. The communication protocol may include, but is not limited to, one or more of the following: wired communication, Wi-Fi®, Bluetooth®, wireless PAN, IR, or other wireless-based or optical-based communication agreements.

[0045] In one environment, user 14 displays the virtual environment 22 using a VR headset 18, while in another environment, user 14 displays the virtual environment 22 on one or more other screens so that the virtual environment 22 displayed on one or more screens immerses user 14 in the virtual environment 22.

[0046] The virtual environment 22 is established and executed by the virtual environment generation component. The virtual environment 22 may be available only to user 14, or it may be available simultaneously to more than one user 14. In one embodiment, the virtual environment 22 is hosted on a server accessible by user 14. The server may be connected to the headset via either a local (intranet) or external (Internet) network connection.

[0047] The virtual environment 22 may be run by the user in at least run mode or design mode. In design mode, user 14 experiences the virtual environment 22 in a more static simulation; that is, user 14 can add or remove objects from the virtual environment 22, or enable or disable physical and other object properties. In run mode, user 14 experiences the virtual environment 22 in a simulation that is enabled as if user 14 were part of the virtual environment 22. For user 14 to be able to run mode, the virtual environment 22 must first be created in design mode. When the simulation is enabled, the physics engine is fully enabled.

[0048] In one embodiment, the virtual environment 22 comprises one or more simulation parameters and a plurality of virtual reality objects 26 (objects 26). An object is an empty container for items. Each item is a criterion having a pointer to associated files and content data associated with an object, and embedded files, i.e., an object, contain file data for a particular item or another object. Each object and each item contained within an object prohibits a hierarchical relationship such that the parent of each item that is sequentially sibling to one another can be considered. In another embodiment, an item may also be a container for an item, or a sub-item such that the item is a parent to a sub-item and a child to an object. In one embodiment, an object 26 includes metadata items, i.e., files that store properties of the object 26, such as the mass of the object 26, the elasticity of the object 26, and the material of the object 26. Each of the properties in the metadata items may be required by a physics engine to appropriately simulate the interaction between the object 26 and any other objects in the virtual environment.

[0049] Referring here to Figure 2, what is shown is an exemplary embodiment of a vehicle object 70 having a parent container 74 and several items including a 3D mesh 78, textures 82, audio 86, and an ActionScript 90. As seen in Figure 2, each of the items attached to the parent container 74 of the vehicle object 70 extends the vehicle object 70. The items identified are for illustrative purposes only and are not intended to be limiting. As defined above, an item is any file that may extend or provide additional information about the object 26. For example, an item may also include, but are not limited to, document files such as doc, docx, ppt, pptx, pdf, and txt, hyperlinks, but are not limited to, URLs, video files such as mp4, avi, mpeg, or mov or dedicated items.

[0050] 3D Mesh 78 is a three-dimensional wireframe representation of the vehicle object 70. It is the base model of the object before any textures or smoothing are applied. Exemplary 3D mesh file types are not limited to, but include stl, obj, fbx, collada, 3ds, vrml, and x3d.

[0051] Texture 82 is a 2D graphics file containing the visual texture of the vehicle object 70. When the vehicle object 70 is rendered in 3D by the graphics processor, Texture 82 is applied to the 3D mesh 78. Exemplary 2D graphics file types include, but are not limited to, jpg, gif, png, bmp, and tiff.

[0052] Audio 86 is an audio clip associated with the vehicle object 70. Audio 86 may contain any sounds emitted by the vehicle object 70, or any sound modifications to another sound file that may be played when another object interacts with the vehicle object 70. Exemplary audio file types include, but are not limited to, midi, mp3, wav, and ogg.

[0053] A dedicated item is an item that exhibits complex functionality that would otherwise be difficult to obtain without programming, and can be attached to an object similar to other items. Dedicated items enable user 14 to build a feature-rich virtual environment without programming. In one embodiment, dedicated items may include, but are not limited to, items such as a video player item, an audio player item, a document viewer item, or a quiz item.

[0054] A video player item provides the functionality to play video file items, such as those attached to an object, or video streams, such as those at the endpoint of a hyperlink object. By providing this item, users can extend it onto a 3D mesh to form a virtual display, such as a virtual TV screen, movie screen, or computer monitor, but not limited to these.

[0055] Similarly, an audio player item provides the functionality to play audio file items, such as those attached to an object, or audio streams, such as those at the endpoint of a hyperlink object. User 14 may implement this item in a simple feature, such as a radio, within a vehicle object 70.

[0056] The document viewer item provides code that allows user 14 to implement a presentation by implementing a document display item on a mesh. Similarly, the quiz item may be implemented by user 14 by attaching a quiz item having multiple elements to a mesh, each element relating to an answer, and the output of the quiz item may be an index of the selected answer.

[0057] When an item's file data is saved along with the file data for an object, the item is embedded in the object. For example, a 3D mesh 78 is embedded in a vehicle object 70 if, for example, the 3D mesh 78 data is saved in the same file as a parent container 74, regardless of whether or not encoding is used. Therefore, moving the file location of the parent container 74 will also move the data for the 3D mesh 78. Similarly, deleting the parent container 74 will also delete the embedded data, including the data for the 3D mesh 78.

[0058] If a link or reference is stored within the parent container 74 of the vehicle object 70, the object references the item. The link or reference may be a pointer to a memory location, a file system, a web resource such as a hyperlink, or a database, for illustrative purposes only. When an item is referenced, the data to which the item is encoded is not stored with the parent container 74, nor is it serialized and compressed with the parent container 74. Therefore, if the parent container 74 is deleted, as well as a smaller parent container 74 in computer memory, the pointer to the item is deleted, but the item data is not.

[0059] In order to implement object 26 within the virtual environment 22, object 26 must first be configured. Referring to Figure 3, an exemplary embodiment of the object configuration process 100 is shown. Generally, the object configuration process comprises the steps of loading a 3D mesh from a selected repository (step 104), automatically creating a parent object (step 108), and adding one or more items to a parent container (step 112). In one embodiment, the step of adding one or more items to a parent container (step 112) may further include a step of linking the objects to each other using coordinate links in order to form a composite object.

[0060] Referring here to Figure 4, an exemplary embodiment of a wheel object 120 having a wheel parent container 124, a 3D mesh 128, a texture 132, and an action script 136 is shown therein. The wheel parent container 124 contains references to or copies thereof of the 3D mesh 128, the texture 132, and the action script 136. The 3D mesh 128 is a three-dimensional wireframe structure representing a wheel. The wireframe structure consists of points or vertices joined by lines or edges. Each vertex is connected to another vertex by at least one edge. The texture 132 is a texture file that wraps the 3D mesh 128 in an image when processed using the 3D mesh 128. For example, the texture 132 may wrap the 3D mesh 128 so that the 3D mesh 128 looks like an actual wheel. In other embodiments, the texture 132 is calculated based on other properties of the object, such as the material of the object and the properties of that material.

[0061] Referring here to Figure 5, an exemplary embodiment of a composite vehicle object 150 having one vehicle object 70 and four wheel objects 120a-d is shown therein. The 3D mesh 78 of the vehicle object 70 exists in three-dimensional space and has coordinates that define the boundaries of the 3D mesh wireframe, where the coordinates are locations in three dimensions defined, for example, by Cartesian coordinates in (X, Y, Z) form. The wheel objects 120a-d are attached to the vehicle object 70 by coordinate links 154a-d such that a first coordinate link 154a links the coordinates of wheel object 120a to the first coordinates of the vehicle object 70, a second coordinate link 154b links the coordinates of wheel object 120b to the second coordinates of the vehicle object 70, a third coordinate link 154c links the coordinates of wheel object 120c to the third coordinates of the vehicle object 70, and a fourth coordinate link 154d links the coordinates of wheel object 120d to the fourth coordinates of the vehicle object 70. In one embodiment, the coordinate link includes a three-dimensional range, also referred to as the connection strength, which provides the range over which objects are connected to each other.

[0062] The coordinate links 154a-d attach each of the wheel objects 120a-d to the vehicle object 70 by maintaining the relative distance between each of the wheel objects 120a-d and the vehicle object 70. Each of the items of the composite vehicle object 150, the vehicle object 70, and the wheel objects 120a-d can operate independently of each other while maintaining a physical connection with the connecting links 154a-d.

[0063] As shown in the illustrative object relationship model 158 of Figure 6, the parent container 74 and wheel containers 124a and 124b (and similarly 124c and 124d, which are not shown) are connected not in a hierarchical relationship, but rather in a lateral relationship through connecting links 154a-d to form a composite vehicle object 150.

[0064] As shown in another exemplary hierarchical object relationship model 158a, the composite vehicle object 150 may be formed through a hierarchical parent-child relationship between the vehicle object 70 (parent) and the wheel objects 120a-d, respectively (children), as shown in Figure 7. Again, the wheel objects 120c and 120d are not shown for clarity. The hierarchical object relationship model 158a shows the wheel containers 124a and 124b, where the items within the wheel containers 124a and 124b are at a lower hierarchical level than the items within the parent container 74, while the items within the wheel containers 124a and 124b are at a lower hierarchical level than the items within the parent container 74.

[0065] In one embodiment, objects in a parent-child relationship forming a composite object retain their respective capabilities and interact independently of the composite object through the use of ActionScript items, while not maintaining relative distances such that the child objects are attached to the parent object, rather than being determined by the strength of the connection between the objects, such that movement of the parent object in any direction also moves all the child objects.

[0066] User 14 places one or more objects in the virtual environment using the environment construction system. As shown in Figure 8, in room 160, the environment construction system provides User 14 with a room settings menu 170 that lists controls including a room name field 174, gravity control 178, scale control 182, private control 186, editable control 190, anonymous public control 194, execution control 198, reset control 202, destruction control 206, public control 210, or settings menu 214. In addition, virtual hands 220a and 220b and cursor 224 are displayed in the virtual environment 22.

[0067] The cursor 224 is a virtual environment representation of a point in the virtual environment 22 that is influenced by input from the user 14 and is movable by the user 14. For example, the cursor 224 may be a line extending from a first position in the virtual environment 22 that the user 14 interprets as corresponding to the user's hand, to a second position in the virtual environment 22 that the user 14 interprets as a point in the virtual environment 22 selected by the user's hand. In another embodiment, the cursor 224 is a geometric figure placed at the second position in the virtual environment 22 that the user 14 interprets as a point in the virtual environment 22 corresponding to the user's hand.

[0068] Room 160 is a specific virtual environment 22 created or downloaded by user 14. Room 160 provides the initial structure of the virtual environment 22 and can store environment settings applicable to objects within Room 160. For example, user 14 can adjust the gravity within the room using gravity control 178 in the room settings menu 170. By using gravity control 178, user 14 adjusts how gravity reacts to all objects within this specific virtual environment 22. Furthermore, user 14 can scale the room using scale control 182. In one embodiment, scale control 182 adjusts the relative size of all objects within Room 160 so that, from the user's perspective, it appears as if the user is zooming in or out. By selecting private control 186, user 14 can prevent Room 160 from being searchable within the repository. User 14 can also adjust whether their Room 160 is editable by other users by toggling editable control 190. User 14 also has the option to publish their room 160 anonymously to one or more repositories by using the anonymous publishing control 194. By using the reset control 202, User 14 can revert changes to the most recent save. The room settings menu 170 also allows User 14 to initialize a new room by initializing the room to an empty virtual environment similar to the environment, by selecting the destroy control 206, which deletes room 160 and all instances of the objects within room 160. If selected, the publishing control 210 allows User 14 to publish room 160 to one or more repositories that match other settings in the room settings menu 170, such as anonymous publishing, whether the room is private or not, and whether the room is editable or not. Furthermore, publishing the room also publishes all objects within the room, which allows third-party users to download and set up a fully functional room with all custom objects and mirror the room most recently published by User 14.In another embodiment, when the public control 210 is selected, the user 14 is presented with a repository menu, which allows the user to select a repository from which room 160 is public. The room settings menu 170 also provides settings menus 214, for example, but not limited to, these menus which are described in more detail below, allowing the user 14 to select different menus within the environment building system.

[0069] In one embodiment, Room 160 may be saved as a room save file containing all settings for the room, such as those adjusted within the room settings menu 170, as well as references to all objects within the room. For each object within the room, the room save file may store default settings, including, but not limited to, the starting position for the object, the starting rotation for the object, the starting scaling for the object, whether the object is affected by gravity, and whether the object is grabbable. Default settings for an object, such as the starting position, starting rotation, starting scaling, and other similar properties, may have values ​​corresponding to three dimensions, such as X, Y, and Z values, for positioning or scaling, and angular rotation values ​​for the starting rotation for the object. By saving Room 160 with a value in the room name field 174, the user 14 can search for the room through a simply recognizable label, the room name.

[0070] In one embodiment, the room settings menu 170 further includes a save control that allows user 14 to save room 160 in its current state to a save file having a save name. If the save control is selected, a save file location screen may or may not be loaded, allowing user 14 to save room 160 to a different file location than the most recent save file. Alternatively, if user 14 selects a save control to save room 160, room 160 is saved to the same save file as the most recent save file. The save file may have the same name as the room name field 174 value, or the save file may further include a date stamp in the save name, allowing user 14 who has selected a virtual environment to load a previously saved file to select as needed. In one embodiment, if the publish control 210 is selected, room 160 is automatically saved, or a save confirmation screen appears, providing user 14 with at least the same options as the save control provides.

[0071] Referring here to Figure 9, an exemplary embodiment of the tool menu 230 for room 160 in the virtual environment 22 is shown there. The tool menu 230 lists a set of three-dimensional object manipulation tools, including the move tool 234, the rotate tool 238, the scale tool 242, the info tool 246, the wire tool 250, the group tool 254, the ungroup tool 258, the duplicate tool 262, and the erase tool 266. The tool menu 230 also provides the undo tool 270 and the settings menu 214. In addition, virtual hands 220a and 220b and a cursor 224 are shown in room 160. Only the above tools are listed in the tool menu 230; the tool menu 230 is not a complete list of tools available to user 14. Other tools may include those that allow the user to copy dimensions from one object to another, copy properties such as texture or color from one object to another, copy one or more object items from one object to another, and reset an object for its basic settings and / or other object operations. Furthermore, the organization of tools in the tool menu 230 is for convenience only, and tools may be organized in other configurations. In one embodiment, user 14 can move one or more tools in the tool menu 230 to rearrange the tools in the order that user 14 finds most convenient.

[0072] The move tool 234 allows the user 14 to select an object in room 160 and move that object to another location in room 160. Similarly, the rotate tool 238 allows the user 14 to rotate a selected object in three-dimensional space, and the scale tool 242 allows the user to scale an object in room 160, i.e., enlarge or reduce its size. Other tools include the info tool 246, which displays the properties and metadata of a selected object; the duplicate tool 262, which duplicates a selected object and its children; and the erase tool 266, which erases a selected object and its child items. Tools available for working with or creating composite objects further include the wire tool 250, which provides the user with a way to wire objects together using coordinate links 154; the group tool 254, which allows the user to group a selection of objects that together form a composite object with selected objects; and the ungroup tool 258, which separates a composite object into its constituent objects. In one embodiment, the ungrouping tool 258 can also separate an object and its items, i.e., remove one or more items from the parent container, so that the items become independent of the object and are considered as the object itself. The tools menu 230 also provides an undo tool 270 that allows the user 14 to undo or disable recent changes to one or more objects in the room 160. Similarly, the tools menu 230 includes a settings menu 214, which allows the user 14 to select different menus within the environment building system, for example, but not limited to, the room settings menu 170 (described in detail below) or a scriptable menu 340.

[0073] Referring here to Figure 10, an exemplary embodiment of network diagram 300 is shown, which has an online repository 304 and a virtual environment component 308 hosted on one or more servers 312 that enable second user 316 and third user 320g to connect to a virtual environment 22 deployed and owned by user 14. The virtual environment component 308 comprises a hosted environment 324 and a user state synchronization component 328. The hosted environment 324 is a virtual environment owned by user 14 and may be online, uploaded and hosted from within the virtual reality world authoring system 10. User 14 may enable others, in this case second user 316 and third user 320, to combine the hosted environment 324. User 14 may also prohibit second user 316 and third user 320 from copying the hosted environment 324, the virtual environment 22, one or more objects within the virtual environment, or one or more items within the virtual environment to their personal repository or to any other environment accessed by the user. Permission to control access to the hosted environment 324 or the virtual environment 22, etc., can be implemented using standard computer techniques, such as access control lists.

[0074] Any room created is said to be persistent; that is, if a room is saved, it stores the state and location of all objects in the room at a first point in time, and when the user opens the room at a second point in time, the state and location of all objects in the room are the same as at the first point in time. At a second point in time after the first point in time, user 14 can load room 160 in design mode and edit room 160. Furthermore, user 14 can copy room 160 to create a second room. Objects and items in the second room will have the same state and location as all objects and items in room 160 at the time the second room was created.

[0075] In one embodiment, there is an online marketplace where a person can give or sell one or more action modules to a user 14, or submit actions to one or more repositories. A user 14 who needs a new action module or set of action modules can connect to one or more additional repositories where additional action modules or sets of action modules are available. By connecting to a repository, user 14 accesses action modules or sets of action modules available in the repository for use within the user's virtual environment 22.

[0076] User 14 or another user 14 can search one or more searchable rooms 160 within one or more repositories. In one embodiment, if user 14 has not selected anonymous sharing control 194, a search may be performed based on the room name field 174 and / or the user's username or handle. In other embodiments, a search may be performed against environmental settings of room 160, such as gravity level, theme, and / or any other searchable characteristics.

[0077] When a second user 316 or a third user 320 connects to the hosted environment 324, the connecting user 316 or 320 automatically connects to the online repository 304 to download the rooms, objects, and items that need to be virtualized in the hosted environment 324 before joining room 160. If the connecting users 316 and / or 320 have permission to join room 160, they also each have at least read permission for each of the objects and items in room 160.

[0078] Furthermore, in order for a second user 316 or a third user 320 to connect to the hosted environment 324, the coordinates of each connecting user in the virtual environment 22, each of the necessary state parameters of the connecting user, and all objects and items are synchronized through the user state synchronization component 328 and sent to each of the one or more users in the hosted environment 324. Thus, when user 14 moves within the hosted environment 324, both the second user 316 and the third user 320 observe user 14's movements as if each user could meet the other directly. The user state synchronization component 328 further provides any connected user with the ability to meet each other, move around the hosted environment 324, chat with each other using a voice or audio communication component, process objects, transmit objects from one user to another, and create notes to display at specific three-dimensional coordinate points within the hosted environment 324.

[0079] In one embodiment, server 312 is a component of the user's virtual environment generation component, where second user 316 and third user 320 connect to the user's virtual reality environment component via a network. As will be discussed further below, the network can be the Internet, a local area network, or any other data connection protocol that provides access to the user's virtual environment generation component by second user 316 or third user 320.

[0080] The virtual reality world authoring system 10 provides user 14 with a script builder 38 that enables user 14 to create interactive composite objects within the virtual environment 22. The virtual reality world authoring system 10 does not require user 14 to write programming code to create composite objects through the use of ActionScript.

[0081] Referring here to Figure 11, a virtual environment 22 of Room 160 is shown having an exemplary embodiment of a scriptable menu 340, the scriptable menu 340 having a number of scriptable functions 344a-n. A scriptable function is a computer function having at least one input and at least one output. Scriptable functions may include functions used to manipulate dynamic three-dimensional objects, such as a move scriptable that moves an object from one place to another with animation effects, a rotater scriptable that describes the rotation of an object in three dimensions, an object position scriptable that instantly sets the three-dimensional position of an object, and a disable scriptable that disables an action script attached to an object. Scriptable functions may also include mathematical functions, the input may include a value to be evaluated, and the output may be the result of the mathematical function. Mathematical functions may include addition, multiplication, logical AND, logical OR, exclusive OR, and logical negation. One or more mathematical functions may be connected to the input or output of another scriptable to modify their respective input or output values. Other scriptable functions may include functions related to time or the passage of time, such as delay or always on. The available scriptable functions may be extended. For example, if user 14 routinely uses NAND logic operations, user 14 may form a new scriptable function by linking the output of a function that is not logical scriptable to the input of a logical AND scriptable function and saving the combined scriptable function as a new NAND scriptable function. The scriptable menu 340 further includes a settings menu 214, which, as described above, allows user 14 to select different menus within the environment configuration system, for example, but not limited to, the room settings menu 170 or the tools menu 230.

[0082] ActionScript is a Turing-complete computer-programmable code module that has one or more scriptable functions. By using ActionScript, users 14 can construct composite objects by wiring them together with action modules without understanding a computer programming language.

[0083] Referring here to Figure 12, an exemplary embodiment of the script builder diagram 360 is shown therein. Here, the trigger object 42 (discussed in further detail below) becomes available within room 160. The trigger scriptable action 362 is associated with the trigger object 42. The trigger scriptable action output 364a is connected to the rotator scriptable action 344a by the scriptable connector 368a. The scriptable connector 368a is a visual representation of the path from the trigger scriptable action output 364a to the rotator scriptable action active input 372a. By connecting the first output to the first input using the scriptable connector 368a, user 14 instructs the script builder 38 to use the value of the first output as the input value to the first input. Furthermore, a static rotator scriptable action X372b of 45 degrees is shown, i.e., 45 degrees is a preset input to a rotator scriptable action 344a, representing the number of degrees of rotation in the "x" direction in three-dimensional coordinate space. Rotator scriptable action 344a also accepts inputs in the "y" and "z" directions. The rotator scriptable action object output 364b is connected to the bathtub object input 372c of the bathtub object 26a by a scriptable connector 368a. By connecting the rotator scriptable action object output 364b to the bathtub object input 372c of the bathtub object 26a, user 14 indicates that the rotator scriptable action 344a is applied to the bathtub object 26a, so that when the trigger object 42 invokes the rotator scriptable action 344a, the bathtub object 26a is rotated 45 degrees in the "x" direction. By connecting the trigger scriptable action 362 to the rotator scriptable action active input 372a, user 14 can configure the system so that when the rotator scriptable action 344a is active, and when the rotator scriptable action 344a is inactive based on the output of the trigger scriptable action 362.The connected trigger scriptable action 362, rotate task scriptable action 344a, and object property 376 create a first action script which is stored as an item in the parent container of the bathtub object 26a. If the script builder diagram 360 is saved, the virtual reality world authoring system 10 processes the script builder diagram 360 and compiles the diagram into a computer language.

[0084] The behavior of the same composite object shown in the script builder diagram 360 can be created using the text script builder. If user 14 is very inclined, user 14 can program the same bathtub object 26a that is created using the script builder 38 using the text script builder. The first action script is written to a text editor and the text file can be imported later into room 160 of the virtual environment 22. When the virtual environment 22 receives the text file, the virtual environment 22 converts the first action script into the script builder diagram 360. An example text file is: / / TriggeredRotator.act-commented out filename [fActivated]>Trigger>[fTriggerCheck] [fTriggerCheck 45 $ $]>Rotate>[aRotateAction] That's fine.

[0085] In the code above, "fActivated" is a function that receives a condition for the trigger scriptable action, outputs the result of the fActivated function, and calls the fTriggerCheck function. The fTriggerCheck function receives three parameters 45 $ $, where $ represents null or an empty value. The input value to the Rotate scriptable action is processed as input to the action input of the bathtub object 26a, and the output to aRotateAction is processed.

[0086] In some embodiments, the schematic diagram 360 of the script builder may show a scriptable action that is not connected to any other scriptable action. In this case, a scriptable action that is not connected to its output has no effect on object 26. For example, in the schematic diagram 360 of the script builder, if the scriptable connector 368a is not there, the output of the trigger scriptable action 362 has no effect on the bathtub object 26a, and therefore, if the rotater scriptable action active input 372a is set to true, it will rotate the bathtub object 26a continuously.

[0087] Furthermore, user 14 may want to create a composite action script involving many separate scriptable actions. One solution is for user 14 to select a second action script similar to the desired action script, download the second action script, make minor modifications, and achieve the desired result. Additionally, if user 14 finds an action script through a search, user 14 may drag the action script from the search onto an object in the virtual environment 22 to which user 14 wishes to add actions. In other embodiments, the action script may include composite actions such as collision avoidance and / or A* algorithm path following among other complex 3D actions.

[0088] The trigger object 42 is an object having a trigger scriptable action 362 that, when active, activates one or more scriptable actions attached to one or more objects 26. The trigger scriptable action 362 can be activated when an object 26, including a user's avatar, interacts with the trigger object 42 in a predetermined manner, such as the user's avatar landing on or passing through the trigger object 42. In one embodiment, the interaction may be determined by the physics engine such that the trigger scriptable action 362 is activated if one or more objects 26 are stacked on the trigger object 42 and one or more objects 26 have a total mass or weight exceeding a trigger threshold. In this scenario, the total mass of the objects is directed to the input of the trigger scriptable action 362. In other embodiments, any property of one or more objects 26 interacting with the trigger object 42 may be used as an input to the trigger scriptable action 362.

[0089] Action scripts are created by the script builder 38 and used to add interactivity to the virtual environment 22. By creating action scripts that include trigger scriptable actions, user 14 can quickly and easily create composite objects within the virtual environment without using programming knowledge. While the wide variety of uses of triggers is constrained solely by the user's imagination, some other examples of using triggers to create interactive composite objects include adding a trigger to a button trigger object such that when pressed, an elevator door opens; adding a trigger to a gun trigger such that when the trigger is pulled, the gun fires; or adding a cross trigger to an entrance / exit such that when the user's avatar crosses an entrance / exit threshold, a quiz object becomes viewable and the quiz object prompts user 14 to answer a quiz present in it. In another embodiment, the input to the trigger scriptable action 362 may be a time-delay scriptable action such that when the time delay ends, an active signal is sent from the output of the time-delay scriptable action to the input of the trigger scriptable action.

[0090] In one embodiment, collision detection is not enabled for object 26 by default. Instead of enabling collision detection, trigger objects are added to, scaled, and connected to object 26 in a hierarchical or parent-child relationship. By adding trigger object 42 to surround object 26, any trigger object interacting with object 26 must first pass through trigger object 42 and then activate trigger scriptable action 362. When a trigger object enters trigger object 42, an entered() event is raised for that trigger and an entered() event message is created. Similarly, when a trigger object no longer intersects, the trigger object raises an exited() event. An event handler and / or message receiver processes the entered() or exited() event or message and activates the trigger scriptable action.

[0091] In one embodiment, the dynamic simulation of the physics engine limits the calculated laws of physics to collision detection, Newton's second law and gravity, and rigid body dynamics. If an object 26 in the virtual environment 22 has its physical properties enabled and possesses valid mass properties, the physics engine subjects the object 26 to a physics simulation. The rigid body dynamics simulation of the physics engine must account for the angular rotation behavior of the object 26. The combined effect of the calculated laws of physics provides a realistic simulation for most usage conditions. In another embodiment, additional physics simulations, such as, but not limited to, soft body dynamics and / or electromagnetic forces, are utilized.

[0092] In some embodiments, the virtual reality world authoring system 10 further provides a bidirectional state API 50. The bidirectional state API 50 provides bidirectional state information from the real world to the virtual environment 22 and from the virtual environment 22 to the real world. When implementing the bidirectional state API 50, real-world objects may be modeled into model objects that act as proxies within the virtual environment 22. In one embodiment, a heart rate monitor may be attached to the user 14. The heart rate monitor may then be connected to the bidirectional state API and transmit heart rate data from the heart rate monitor to the virtual environment 22. A heart rate model object within the virtual environment 22 may receive the heart rate data as input and connect the output from the heart rate model object to the input of a display object, thereby providing the user 14 with a display of the user's real-world heart rate within the virtual environment 22. Similarly, information from within the virtual environment 22 can be transmitted to the outside of the virtual environment 22 and to the inside of the real world. For example, arm movement data of the user's avatar may be recorded and transmitted to the real world through the bidirectional state API 50. In the real world, arm movement data may be collected and transmitted to the robotic arm so that the movements of the user's avatar in the virtual environment are mirrored by the robotic arm in the real world.

[0093] One such need for the bidirectional state API 50 is remote robot operation in hazardous areas. A robot with one or more cameras may be deployed in a hazardous area that is inaccessible to or harmful to humans. One or more cameras on the robot can collect images from the real world and transmit the images to a virtual environment. The virtual environment 22 can then reflect objects of the real-world images by using the real-world images as texture files for the generated objects. A user 14 with a robot avatar can then interact with the objects, and such interactions can then be mirrored to the real world by collecting motion data of the robot avatar and sending that data back to the robot via the bidirectional state API 50, thereby causing the robot to mirror the movements of the robot avatar to the real world.

[0094] The bidirectional state API 50 is an event processing system that queues events. API events are related to messages that should be sent and received by the system. For example, an API event handler may implement the following pseudocode, where received messages are called the message_received function and sent messages are called the message_sent function.

[0095] switch(event.type) { case: "api_message_received:" api_message_recv(state); case: "api_message_send" : api_message_send(object); }

[0096] The pseudocode example of the above functionality is: api_message_recv(state) { objectname = state.objectname object = getobjectwithname(objectname) object.property1 = state.property1 object.property2 = state.property2 object.property3 = state.property3 ... } api_message_send(object) { state = new state state.objectname = object.name; state.property1 = object.property1 state.property2 = object.property2 state.property3 = object.property3 ... send_state(state) } It can be expressed as follows.

[0097] The send state function sends state information when the api_message_send event occurs. It can do this by wrapping the message, serializing it, and sending it over the network, through any one or more of a variety of protocols, both connected and stateless. One or more protocols may include http, https, telnet, web sockets, signalr, sockiet.io, or tcp / ip sockets.

[0098] Referring now to Figure 13, an exemplary embodiment of the virtual reality world authoring system 10 is shown therein, in which user 14 is in a virtual environment 22, which is a basketball court having a composite basketball object 380, a basketball object 382, ​​a basketball trigger object 384, a basketball goal object 388 and a composite basketball ring object 392.

[0099] To create a composite basketball object 380, user 14 creates a basketball object 382 having a basketball metadata item that stores the basketball's texture, the sound of the basketball bouncing, and physical characteristics of the basketball such as mass and elasticity, and an item containing a bounce action script. User 14 also creates a basketball trigger object 384, which is an ellipsoid with a radius larger than the radius of the basketball object. User 14 creates the composite basketball object 380 by connecting the basketball object 382 and the basketball trigger object 384 to their center points, causing the basketball trigger object 384 to completely contain the basketball object 382 within the virtual environment 22. The bounce action script may take physical properties of the basketball, including mass and elasticity, as well as velocity values, as input values ​​to calculate the bounce action.

[0100] For the composite basketball object 380 to bounce, there must be a force acting on it, and therefore collision detection is implemented along with a trigger scriptable action attached to the basketball trigger object 384. For collision detection, a trigger event is raised when intersecting objects cross the basketball trigger object 384. The trigger handler responds to the event by activating the trigger scriptable action by calling its input. If user 14 selects a trigger scriptable action from the scriptable menu, collision detection and event handling are automatically implemented for the trigger scriptable action. User 14 can then input a script builder 38, where user 14 can use a scriptable connector to connect the output of the trigger scriptable action on the basketball trigger object 384 to the input on the bounce scriptable action on the basketball object 382. Other inputs to the bounce scriptable action may include the velocity and / or mass of the interfering object, thereby preventing the bounce scriptable action from bouncing the composite basketball object 380 when the composite basketball object 380 intersects with a trigger object or another object that should not affect the composite basketball object. The output of the bounce scriptable action can then be connected to an action input to the basketball object via a second scriptable connector, and the connection from the trigger scriptable action to the bounce scriptable action to the basketball object causes the basketball to bounce when the trigger scriptable action is activated by collision detection.

[0101] Similarly, the composite basketball ring object 392 is created by forming a hierarchical relationship between the basketball ring object and the basketball ring trigger object, which has a rim trigger scriptable action, where the basketball ring trigger object is a circular plane surrounded by the rim, and the center points of both objects are connected. Here, collision detection needs to test again whether one or more interfering objects intersect with the basketball ring trigger object. The trigger scriptable action may have the condition that the intersecting object is a composite basketball object, as the trigger scriptable action activates the next scriptable action. User 14 can then input the script builder 38, where User 14 can use a scriptable connector to connect the output of the rim trigger scriptable action of the basketball ring trigger object to the input on the goal scriptable action of the scoreboard object. Other inputs to the goal scriptable action are characteristics of the composite basketball object 380, such as which team last had control of the ball and the location on the court where the composite basketball object 380 was released. The output from the goal scriptable action may include the number of points to score and the team scoring. The output of the goal scriptable action can then be connected to an action input on the scoreboard object via a second scriptable connector, and the connection from the rim trigger scriptable action to the goal scriptable action on the scoreboard object causes the score to change when the rim trigger scriptable action is activated by collision detection.

[0102] Referring here to Figure 14, an exemplary embodiment of a virtual reality world authoring system 10 running on one or more computer systems 400 is shown. The computer system 400 may comprise one or more processing components 404, each having a computer processor 408, one or more non-temporary memories 412, and one or more communication components 416 (or communicatively connected to them). The non-temporary memories 412 may store computer code that implements the virtual reality world authoring system 10. The computer system 400 may include a network 420 enabling bidirectional communication between the processing components 404, one or more VR headsets 18, and / or a server system 424. The computer processors 408 or more computer processors 408 may or may not be located in a single physical location. In one embodiment, the processing components 404 are integrated with the VR headsets 18.

[0103] In one embodiment, network 420 is the Internet, and the VR headset 18 interfaces with a computer processor 408 via a communication component 416. However, it should be noted that network 420 may be virtually any type of network, including the World Wide Web (or the Internet), local area networks (LANs), wide area networks (WANs), metropolitan networks, wireless networks, cellular networks, Bluetooth networks, Global Systems for Mobile Communications (GSM®) networks, code division multiple access (CDMA) networks, 3G networks, 4G networks, 5G networks, satellite networks, wireless networks, optical networks, cable networks, public switched telephone networks, Ethernet® networks, combinations thereof, and / or similar. It is conceivable that embodiments of the present disclosure may utilize more advanced networking topologies in the near future.

[0104] In one embodiment, a processing component 404 having a computer processor 408 and non-temporary memory 412 may be implemented using a server system 424 having multiple servers in a configuration suitable for providing commercial computer-based business systems such as commercial websites and / or data centers.

[0105] The following is a numbered list of non-limiting and exemplary embodiments of the concepts of the present invention disclosed herein.

[0106] 1. Processor and, A headset having a virtual feedback screen, One or more user interface devices, Non-temporary computer-readable media for storing computer executable code and Equipped with, Computer executable code, when executed by a processor, is given to the processor, A procedure for projecting a mixed reality environment onto a headset, wherein the mixed reality environment has one or more objects, and each of the one or more objects is created by object computer code having properties that define how the one or more objects interact within the mixed reality environment. A procedure for displaying an action menu on a virtual feedback screen having a predefined list of action modules in response to a first input from at least one user interface device, wherein the action modules include modules of computer executable code having one or more functions, and the one or more functions are configured to affect one or more attributes of a first object of one or more objects in a mixed reality environment; The procedure for updating the object computer code of the first object to include the selected action module and An electronic device that performs an action.

[0107] 2. The first object is an electronic device of exemplary embodiment 1, identified before the first input.

[0108] 3. The first object is an electronic device of an exemplary embodiment 1, identified after the first input.

[0109] 4. A non-temporary computer-readable medium is an electronic device, one of the exemplary embodiments 1-3, which, when executed by a processor, stores computer-executable code that causes the processor to perform a procedure to update the object computer code of a first object to include a selected action module in response to a trigger event.

[0110] 5. An exemplary electronic device of Embodiment 4 in which a trigger event is raised when a predetermined trigger condition is met by a user interacting with a trigger object.

[0111] 6. A non-temporary computer-readable medium, when executed by a processor, stores computer-executable code that causes the processor to perform a procedure to display a trigger condition menu having a list of one or more trigger conditions for a trigger object in response to a second input from at least one user interface device for selecting a specific trigger object from a trigger menu, and a procedure that allows a user to place a trigger object in a mixed reality environment in response to a selection of a first trigger condition for the trigger object, wherein if the trigger condition is met, the processor causes the processor to update the object computer code of the first object to include the selected action module. This is an example of an electronic device in any one of embodiments 1-5.

[0112] 7. Processor and, A headset having a virtual feedback screen, One or more user interface devices, Non-temporary computer-readable media for storing computer executable code and Equipped with, Computer executable code, when executed by a processor, is given to the processor, A procedure for projecting a mixed reality environment onto a headset, wherein the mixed reality environment has one or more objects, and each of the one or more objects is created by object computer code having properties that define how the one or more objects interact within the mixed reality environment. A procedure to display a trigger condition menu having a list of one or more trigger conditions for a trigger object in response to a first input from at least one user interface device in which a specific trigger object is selected from a trigger menu, and a procedure to enable the user to place the trigger object in a mixed reality environment in response to the selection of a first trigger condition for the trigger object. An electronic device that performs an action.

[0113] 8. The trigger object is an electronic device of an exemplary embodiment 7, identified before the first input.

[0114] 9. The trigger object is an electronic device of an exemplary embodiment 7, identified after the first input.

[0115] 10. A non-temporary computer-readable medium is an electronic device from any one of the exemplary embodiments 7-9 that stores computer-executable code, when executed by a processor, causing the processor to perform a procedure to update the object computer code of a trigger object to include a selected action module in response to a trigger event.

[0116] 11. An exemplary embodiment of an electronic device 10 in which a trigger event is raised when a predetermined trigger condition is met by a user interacting with a trigger object.

[0117] 12. When executed by the processor, the processor will be given the following: A procedure for projecting a mixed reality environment onto a headset, wherein the mixed reality environment has one or more objects, and each of the one or more objects is created by object computer code having properties that define how the one or more objects interact within the mixed reality environment. A procedure for displaying an action menu on a virtual feedback screen having a predefined list of action modules in response to a first input from at least one user interface device, wherein the action modules include modules of computer executable code having one or more functions, and the one or more functions are configured to affect one or more attributes of a first object of one or more objects in a mixed reality environment; The procedure for updating the object computer code of the first object to include the selected action module and A non-temporary, computer-readable medium that stores computer-executable code that causes a program to run.

[0118] 13. The first object is a non-temporary computer-readable medium of exemplary embodiment 12, identified prior to the first input.

[0119] 14. The first object is a non-temporary computer-readable medium of an exemplary embodiment 12, which is identified after the first input.

[0120] 15. A non-temporary computer-readable medium, when executed by a processor, stores computer-executable code that causes the processor to perform a procedure to update the object computer code of a first object to include a selected action module in response to a trigger event, one of the non-temporary computer-readable mediums of exemplary embodiments 12-14.

[0121] 16. A non-temporary computer-readable medium of an exemplary embodiment 15 in which a trigger event is raised when a user interacts with a trigger object and satisfies predetermined trigger conditions.

[0122] 17. A non-temporary computer-readable medium, when executed by a processor, stores computer-executable code that causes the processor to perform a procedure to display a trigger condition menu having a list of one or more trigger conditions for a trigger object in response to a second input from at least one user interface device for selecting a specific trigger object from a trigger menu, and a procedure that allows a user to place a trigger object in a mixed reality environment in response to a selection of a first trigger condition for the trigger object, wherein if the trigger condition is met, the non-temporary computer-readable medium in any one of the exemplary embodiments 12-16 causes the processor to update the object computer code of the first object to include the selected action module.

[0123] 18. Processor and, A headset having a virtual feedback screen, One or more user interface devices, Non-temporary computer-readable media for storing computer executable code and Equipped with, Computer executable code, when executed by a processor, is given to the processor, A procedure for projecting a virtual reality environment onto a headset, wherein the virtual reality environment has one or more objects, each of which is created by a container having properties that define how the one or more objects interact within the virtual reality environment, and each of the one or more objects has a three-dimensional mesh having coordinates that define the boundaries of the one or more objects. The procedure for selecting the first object of one or more objects, A procedure for displaying an action menu in a virtual reality environment on a virtual feedback screen in response to a first input from at least one user interface device, wherein the action menu has a list of predefined action scripts, each action script includes a module of computer executable code having one or more functions, and each or more functions are configured to affect one or more attributes of a first object among one or more objects in the virtual reality environment; The procedure to update the container of the first object to include the selected action script and An electronic device that performs an action.

[0124] 19. The first object is an electronic device of an exemplary embodiment 18, which is identified before the first input.

[0125] 20. The first object is an electronic device of an exemplary embodiment 18, which is identified after the first input.

[0126] 21. A non-temporary computer-readable medium is an electronic device, one of the exemplary embodiments 18-20, which, when executed by a processor, stores computer-executable code that causes the processor to perform a procedure to update a container of a first object to include a selected action script in response to a trigger event.

[0127] 22. An exemplary embodiment of an electronic device 21 in which a trigger event is raised when a predetermined trigger condition is met by a user interacting with a trigger object.

[0128] 23. A non-temporary computer-readable medium, when executed by a processor, stores computer-executable code that causes the processor to perform a procedure to display a trigger condition menu having a list of one or more trigger conditions for a trigger object in response to a second input from at least one user interface device for selecting a specific trigger object from a trigger menu, and a procedure that allows a user to place a trigger object in a virtual reality environment in response to a selection of a first trigger condition for the trigger object, wherein if the trigger condition is met, the processor causes the processor to update the container of the first object to include the selected action script. This is an example of an electronic device from any one of the 18-22 electronic devices.

[0129] 24. Processor and, A headset having a virtual feedback screen, One or more user interface devices, Non-temporary computer-readable media for storing computer executable code and Equipped with, Computer executable code, when executed by a processor, is given to the processor, A procedure for projecting a virtual reality environment onto a headset, wherein the virtual reality environment has one or more objects, and each of the one or more objects is created by a container having properties that define how the one or more objects interact within the virtual reality environment. A procedure for forming a composite object by connecting a trigger object selected from a trigger menu to a specific object among one or more objects in response to input from at least one user interface device, wherein the composite object includes one or more action scripts and displays a scriptable menu having one or more trigger scriptable actions for the trigger object in a virtual reality environment on a virtual feedback screen. Depending on the selection of a trigger scriptable action for a trigger object, this allows the user to place the trigger object in a virtual reality environment and to associate the trigger scriptable action with one or more action scripts of a composite object. An electronic device that performs an action.

[0130] 25. The trigger object is an electronic device of an exemplary embodiment 24, which is identified before the input.

[0131] 26. The trigger object is an electronic device of an exemplary embodiment 24, which is identified after the input.

[0132] 27. A non-temporary computer-readable medium is an electronic device, one of the exemplary embodiments 24-26, which, when executed by a processor, stores computer-executable code that causes the processor to perform a procedure to update a composite object to activate an action script in response to a trigger event.

[0133] 28. An exemplary embodiment of an electronic device in 27 in which a trigger event is raised when a user interacts with a trigger object and a predetermined trigger condition is met.

[0134] 29. When executed by the processor, the processor, A procedure for projecting a virtual reality environment onto a headset, wherein the virtual reality environment has one or more objects, each of which is created by a container having properties that define how the one or more objects interact within the virtual reality environment, and each of the one or more objects has a three-dimensional mesh having coordinates that define the boundaries of the one or more objects. A procedure for displaying an action menu on a virtual feedback screen having a predefined list of action scripts in response to a first input from at least one user interface device for selecting a first object among one or more objects in a virtual reality environment, wherein the action script includes a module of computer executable code having one or more functions, and the one or more functions are configured to affect one or more attributes of the first object among one or more objects in the virtual reality environment; The procedure to update the container of the first object to include the selected action script and A non-temporary, computer-readable medium that stores computer-executable code that causes a program to run.

[0135] 30. The first object is a non-temporary computer-readable medium of an exemplary embodiment 29, which is identified prior to the first input.

[0136] 31. The first object is a non-temporary computer-readable medium of an exemplary embodiment 29, which is identified after the first input.

[0137] 32. A non-temporary computer-readable medium, when executed by a processor, is a non-temporary computer-readable medium in any one of the exemplary embodiments 29-31 that stores computer-executable code causing the processor to perform a procedure to update a container of a first object to include a selected action script in response to a trigger event.

[0138] 33. A non-temporary computer-readable medium of an exemplary embodiment 32, in which a trigger event is raised when a user interacts with a trigger object and satisfies predetermined trigger conditions.

[0139] 34. A non-temporary computer-readable medium, when executed by a processor, stores computer-executable code that causes the processor to perform a procedure to display a trigger condition menu having a list of one or more trigger conditions for a trigger object in response to a second input from at least one user interface device that selects a specific trigger object from a trigger menu, and a procedure that allows a user to place a trigger object in a virtual reality environment in response to a selection of a first trigger condition for the trigger object, wherein if the trigger condition is met, the processor causes the processor to update a container of the first object to include the selected action script. This is a non-temporary computer-readable medium in any one of the exemplary embodiments 29-33.

[0140] From the above description and examples, the concepts of the invention disclosed and claimed herein are well adapted to achieve the advantages referred to herein. While exemplary embodiments of the concepts of the invention have been described for the purposes of this disclosure, it will be understood that numerous modifications can be made which are readily apparent to those skilled in the art and which are carried out within the spirit of the concepts of the invention disclosed and claimed herein. For illustrative purposes, the examples of a VR environment of a basketball court and a composite object of a vehicle are used, however, it should be understood that the examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

Claims

1. A processor; a headset having a visual feedback screen; one or more user interface devices; A non-transitory computer-readable medium storing computer executable code; Equipped with The computer executable code, when executed by the processor, causes the processor to: projecting a mixed reality environment into the headset, the mixed reality environment having one or more objects, each of the one or more objects created by object computer code having properties that define how the one or more objects interact within the mixed reality environment; displaying an action menu on the visual feedback screen having a list of predefined action modules in response to a first input from at least one user interface device, the action modules including modules of computer executable code having one or more functions, the one or more functions configured to affect one or more attributes of a first one of the one or more objects in the mixed reality environment; updating the object computer code of the first object to include a selected action module; An electronic device that runs

2. The electronic device of claim 1 , wherein the first object is identified prior to the first input.

3. The electronic device of claim 1 , wherein the first object is identified after the first input.

4. 4. The electronic device of claim 1, wherein the non-transitory computer-readable medium stores computer executable code that, when executed by the processor, causes the processor to perform a procedure for updating the object computer code of the first object to include a selected action module in response to a trigger event.

5. The electronic device of claim 4 , wherein the trigger event is raised when a user interacts with a trigger object, thereby satisfying a predetermined trigger condition.

6. 6. The electronic device of claim 1, wherein the non-transitory computer-readable medium stores computer-executable code that, when executed by the processor, causes the processor to perform the steps of: displaying a trigger condition menu having a list of one or more trigger conditions for the trigger object in response to a second input from at least one user interface device selecting a particular trigger object from a trigger menu; and enabling a user to place the trigger object in the mixed reality environment in response to a selection of a first trigger condition for the trigger object, wherein the first trigger condition, if satisfied, causes the processor to update the object computer code of the first object to include a selected action module.

7. A processor; a headset having a visual feedback screen; one or more user interface devices; A non-transitory computer-readable medium storing computer executable code; Equipped with The computer executable code, when executed by the processor, causes the processor to: projecting a mixed reality environment into the headset, the mixed reality environment having one or more objects, each of the one or more objects created by object computer code having properties that define how the one or more objects interact within the mixed reality environment; displaying a trigger condition menu having a list of one or more trigger conditions for a particular trigger object in response to a first input from at least one user interface device selecting the particular trigger object from a trigger menu; and enabling a user to place the trigger object in the mixed reality environment in response to a selection of a first trigger condition for the trigger object. An electronic device that runs

8. The electronic device of claim 7 , wherein the trigger object is identified before the first input.

9. The electronic device of claim 7 , wherein the trigger object is identified after the first input.

10. 10. The electronic device of claim 7, wherein the non-transitory computer readable medium stores computer executable code that, when executed by the processor, causes the processor to perform a procedure for updating the object computer code of the trigger object to include a selected action module in response to a trigger event.

11. The electronic device of claim 10 , wherein the trigger event is raised when a user's interaction with the trigger object satisfies a predetermined trigger condition.

12. The processor: A method of projecting a mixed reality environment into a headset, the mixed reality environment having one or more objects, each of the one or more objects created by object computer code having properties that define how the one or more objects interact within the mixed reality environment; displaying an action menu on a visual feedback screen having a list of predefined action modules in response to a first input from at least one user interface device, the action modules including modules of computer executable code having one or more functions, the one or more functions configured to affect one or more attributes of a first object of the one or more objects in the mixed reality environment; updating the object computer code of the first object to include a selected action module; A computer program for executing the above.

13. The computer program product of claim 12 , wherein the first object is identified prior to the first input.

14. The computer program product of claim 12 , wherein the first object is identified after the first input.

15. 15. A computer program product as claimed in any one of claims 12 to 14, causing the processor to perform the steps of: updating the object computer code of the first object to include a selected action module in response to a trigger event.

16. 16. The computer program product of claim 15, wherein the trigger event is raised when a user's interaction with a trigger object satisfies a predetermined trigger condition.

17. 17. A computer program product as claimed in any one of claims 12 to 16, further comprising: in response to a second input from at least one user interface device selecting a particular trigger object from a trigger menu, displaying a trigger condition menu having a list of one or more trigger conditions for the trigger object; and in response to a selection of a first trigger condition for the trigger object, allowing a user to place the trigger object in the mixed reality environment, wherein if the first trigger condition is satisfied, the computer program product causes the processor to update the object computer code of the first object to include a selected action module.

18. A processor; a headset having a visual feedback screen; one or more user interface devices; A non-transitory computer-readable medium storing computer executable code; Equipped with The computer executable code, when executed by the processor, causes the processor to: projecting a virtual reality environment onto the headset, the virtual reality environment having one or more objects, each of the one or more objects being created with a container having properties that define how the one or more objects interact within the virtual reality environment, the one or more objects having a three dimensional mesh having coordinates that define a boundary of the one or more objects; selecting a first object of the one or more objects; displaying, in response to a first input from at least one user interface device, on the visual feedback screen an action menu in the virtual reality environment, the action menu having a list of predefined action scripts, the action scripts including modules of computer executable code having one or more functions, the one or more functions configured to affect one or more attributes of the first object of the one or more objects in the virtual reality environment; updating the container of the first object to include a selected action script; An electronic device that runs

19. The electronic device of claim 18 , wherein the first object is identified prior to the first input.

20. The electronic device of claim 18 , wherein the first object is identified after the first input.

21. 21. The electronic device of claim 18, wherein the non-transitory computer-readable medium stores computer-executable code that, when executed by the processor, causes the processor to perform a procedure for updating the container of the first object to include a selected action script in response to a trigger event.

22. 22. The electronic device of claim 21, wherein the trigger event is raised when a user interacts with a trigger object, thereby satisfying a predetermined trigger condition.

23. 23. The electronic device of claim 18, wherein the non-transitory computer-readable medium stores computer-executable code that, when executed by the processor, causes the processor to perform the steps of: displaying a trigger condition menu having a list of one or more trigger conditions for a particular trigger object in response to a second input from at least one user interface device selecting a particular trigger object from a trigger menu; and, in response to a selection of a first trigger condition for the trigger object, allowing a user to place the trigger object in the virtual reality environment, wherein the first trigger condition, if satisfied, causes the processor to update the container of the first object to include a selected action script.

24. A processor; a headset having a visual feedback screen; one or more user interface devices; A non-transitory computer-readable medium storing computer executable code; Equipped with The computer executable code, when executed by the processor, causes the processor to: projecting a virtual reality environment into the headset, the virtual reality environment having one or more objects, each of the one or more objects being created with a container having properties that define how the one or more objects interact within the virtual reality environment; connecting a trigger object selected from a trigger menu to a particular object of the one or more objects in response to an input from at least one user interface device to form a composite object, the composite object including one or more action scripts, and displaying a scriptable menu having one or more trigger scriptable actions for the trigger object in the virtual reality environment on the visual feedback screen; in response to a selection of a trigger scriptable action for the trigger object, enabling a user to place the trigger object in the virtual reality environment and associating the trigger scriptable action with the one or more action scripts of the composite object. An electronic device that runs

25. The electronic device of claim 24 , wherein the trigger object is identified prior to the input.

26. The electronic device of claim 24 , wherein the trigger object is identified after the input.

27. 27. The electronic device of claim 24, wherein the non-transitory computer-readable medium stores computer-executable code that, when executed by the processor, causes the processor to perform a procedure for updating the composite object to activate an action script in response to a trigger event.

28. 28. The electronic device of claim 27, wherein the trigger event is raised when the user's interaction with the trigger object satisfies a predetermined trigger condition.

29. The processor: A method of projecting a virtual reality environment into a headset, the virtual reality environment having one or more objects, each of the one or more objects being created with a container having properties that define how the one or more objects interact within the virtual reality environment, the one or more objects having a three dimensional mesh having coordinates that define a boundary of the one or more objects; displaying an action menu on a visual feedback screen having a list of predefined action scripts in response to a first input from at least one user interface device selecting a first one of the one or more objects in the virtual reality environment, the action script comprising a module of computer executable code having one or more functions, the one or more functions configured to affect one or more attributes of the first one of the one or more objects in the virtual reality environment; updating the container of the first object to include a selected action script; A computer program for executing the above.

30. 30. The computer program product of claim 29, wherein the first object is identified prior to the first input.

31. 30. The computer program product of claim 29, wherein the first object is identified after the first input.

32. 32. A computer program product as claimed in any one of claims 29 to 31, causing the processor to perform the steps of updating the container of the first object to include a selected action script in response to a trigger event.

33. 33. The computer program product of claim 32, wherein the trigger event is raised when a user interacts with a trigger object, thereby satisfying a predetermined trigger condition.

34. 34. The computer program product of claim 29, further comprising: in response to a second input from at least one user interface device selecting a particular trigger object from a trigger menu, displaying a trigger condition menu having a list of one or more trigger conditions for the trigger object; and in response to a selection of a first trigger condition for the trigger object, allowing a user to place the trigger object in the virtual reality environment, wherein if the first trigger condition is satisfied, the computer program product causes the processor to update the container of the first object to include a selected action script.

35. A processor, a headset having a visual feedback screen; one or more user interface devices; A non-transitory computer-readable medium storing computer executable code; Equipped with The computer executable code, when executed by the processor, causes the processor to: projecting a virtual reality environment onto the headset, the virtual reality environment having an object, the object being created by a container in the virtual reality environment having properties that define how the object interacts within the virtual reality environment, and the object having a three dimensional mesh having coordinates that define a boundary of the object; displaying, in response to a first input from the one or more user interface devices, on the visual feedback screen an action menu within the virtual reality environment, the action menu having a list of predefined action scripts, the action scripts including modules of computer executable code having one or more functions, the one or more functions configured to affect the one or more properties of the container within the virtual reality environment; updating the container for the object to include a selected action script in response to a second input from the one or more user interface devices, the second input including dragging the selected action script from the action menu onto the object within the virtual reality environment on the visual feedback screen; An electronic device that runs 36. A processor, a headset having a visual feedback screen; one or more user interface devices; A non-transitory computer-readable medium storing computer executable code; Equipped with The computer executable code, when executed by the processor, causes the processor to: projecting a virtual reality environment onto the headset, the virtual reality environment having one or more objects, each of the one or more objects being in the virtual reality environment and created by a container having properties defining how the one or more objects interact within the virtual reality environment, and having a first three dimensional mesh having first coordinates defining a first boundary of the one or more objects; in response to input from the one or more user interface devices, connecting a trigger object selected from a trigger menu to the container of a particular object of the one or more objects in the virtual reality environment, the trigger object having a second three-dimensional mesh having second coordinates defining a second boundary of the trigger object, thereby forming a composite object within the container by linking the particular object and the trigger object with a coordinate link between the first coordinates and the second coordinates, the composite object including one or more action scripts; displaying a scriptable menu in the virtual reality environment on the visual feedback screen having trigger scriptable actions for the trigger object; in response to a selection of a trigger scriptable action for the trigger object, allowing a user to associate the trigger scriptable action with the one or more action scripts of the composite object; An electronic device that runs 37. A processor comprising: A method of projecting a virtual reality environment onto a headset, the virtual reality environment having one or more objects, each of the one or more objects being created from a container within the virtual reality environment and having properties that define how the one or more objects interact within the virtual reality environment, the one or more objects having a three dimensional mesh having coordinates that define a boundary of the one or more objects; in response to a first input from at least one user interface device, selecting a first one of the one or more objects in the virtual reality environment and displaying an action menu on a visual feedback screen having a list of predefined action scripts, the action scripts including modules of computer executable code having one or more functions, the one or more functions configured to affect one or more properties of the first one of the one or more objects in the virtual reality environment; updating the container for the first object to include a selected action script in response to a second input from the at least one user interface device, the second input including dragging the selected action script from the action menu onto the first object within the virtual reality environment on the visual feedback screen; A computer program for executing the above.