Sound effect simulation by creating virtual reality obstacles

Abstract

A method, computer system, and computer program product are provided for modulating external sounds to reflect sound effects of virtual objects in a mixed reality environment. The invention may include creating a knowledge corpus, recording sound effects occurring outside the mixed reality environment experienced by a user operating a mixed reality device; identifying one or more objects in the mixed reality environment, including at least one virtual object; modulating the sound effects based on the knowledge corpus to simulate one or more sound effects of the one or more objects in the MR environment; and playing the modulated sound effects to the user.

Description

[Technical Field]

[0001] The present invention relates generally to the field of computing, and more particularly to mixed reality. [Background technology]

[0002] Mixed reality is a field related to blending the real and virtual worlds so that physical and digital objects coexist and interact in real time. Mixed reality does not occur exclusively in either the physical or virtual world, but is a hybrid of real and virtual reality; as such, mixed reality represents everything on the real-virtual continuum except for the two extremes, i.e., purely physical and purely virtual environments. Mixed reality therefore includes augmented virtuality (AV), augmented reality (AR), and virtual reality (VR). Mixed reality has found practical applications in remote work, military and commercial training, gaming, and hybrid amusement park-style rides. Summary of the Invention

[0003] According to one embodiment, a method, computer system, and computer program product are provided for modulating external sounds to reflect sound effects of virtual objects in a mixed reality environment. The method may include creating a knowledge corpus, recording sound effects occurring outside the mixed reality environment experienced by a user operating a mixed reality device, identifying one or more objects in the mixed reality environment, including at least one virtual object, modulating the sound effects based on the knowledge corpus to simulate one or more sound effects of one or more objects in the MR environment, and playing the modulated sound effects to the user. [Brief explanation of the drawings]

[0004] These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings, in which the illustrations are for clarity in facilitating understanding of the invention by those skilled in the art together with the detailed description, and in which various features of the drawings are not drawn to scale. In the drawings, the following is shown:

[0005] [Figure 1] 1 illustrates an exemplary networked computer environment in accordance with at least one embodiment.

[0006] [Figure 2] 1 is an operational flowchart illustrating a mixed reality audio modulation process according to at least one embodiment.

[0007] [Figure 3] FIG. 1 illustrates a use case for a mixed reality audio modulation system, according to at least one embodiment.

[0008] [Figure 4A] FIG. 1 illustrates a use case for a mixed reality audio modulation system, according to at least one embodiment.

[0009] [Figure 4B] FIG. 1 illustrates a use case for a mixed reality audio modulation system, according to at least one embodiment.

[0010] [Figure 5] FIG. 2 is a block diagram of the internal and external components of the computer and server shown in FIG. 1 according to at least one embodiment.

[0011] [Figure 6] 1 illustrates a cloud computing environment according to an embodiment of the present invention.

[0012] [Figure 7]1 illustrates abstraction model layers according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] Detailed embodiments of the claimed structures and methods are disclosed herein; however, it should be understood that the disclosed embodiments are merely exemplary of the claimed structures and methods, which may be embodied in various forms. The present invention may, however, be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. In the description, details of well-known features and techniques may be omitted to avoid unnecessarily obscuring the presented embodiments.

[0014] FIELD OF THE INVENTION

[0002] Embodiments of the present invention relate to the field of computing, and more particularly, to mixed reality. The exemplary embodiments described below provide, among other things, systems, methods, and program products for modulating sound effects originating outside a mixed reality environment to reflect the combined virtual and physical conditions of the mixed reality environment. Accordingly, the present embodiments have the ability to improve the technical field of mixed reality by improving the fidelity of sounds perceived by a user within the mixed reality environment, particularly with respect to sounds originating outside the mixed reality environment, thereby improving the sense of immersion within the mixed reality environment and the accuracy of simulation within the mixed reality environment.

[0015] As mentioned above, mixed reality is a field related to blending the real and virtual worlds so that physical and digital objects coexist and interact in real time. Mixed reality does not occur exclusively in either the physical or virtual world, but is a hybrid of real and virtual reality; as such, mixed reality represents everything on the real-virtual continuum except for the two extremes, i.e., purely physical and purely virtual environments. Mixed reality therefore includes augmented virtuality (AV), augmented reality (AR), and virtual reality (VR). Mixed reality has found practical applications in, for example, remote work, military and commercial training, games, and simulations.

[0016] Mixed reality presents its own unique challenges; sounds produced in any environment are altered and distorted by, for example, the presence, shape, and material composition of obstacles in the environment, the properties of the medium through which the sound propagates, and the relative distance between the sound's source, obstacles and surfaces, and the user or device perceiving the sound. Mixed reality environments comprise virtual elements as well as physical elements; the influence of these virtual elements must be considered in addition to the influence of physical elements to ensure that the sounds reproduced to a user accurately reflect the environment experienced by the user and to maintain the immersion of the user's mixed reality experience. However, currently, mixed reality experiences only modulate sounds created by the mixed reality software running the mixed reality experience. As a result, sounds such as music playing on a background application, a background chat application, or voices emanating from people physically near the user can sound harsh and unnatural within the user's mixed reality environment, distracting the user from the experience and reducing their enjoyment. Additionally, without a way to simulate the effects of virtual and physical elements on sound, a user who wants to hear how different environmental obstacles affect sound must either construct those obstacles in the physical world or build a detailed simulation that cannot be easily modified to reflect dynamic changes to the environment, such as adding or removing virtual elements.

[0017] Thus, it may be advantageous to implement a system that, among other things, records sound effects originating outside a user's mixed reality environment and modulates the sound to accurately reflect both physical and virtual elements within the mixed reality environment; allows a user to create virtual objects in the mixed reality environment and hear how sounds change based on the added virtual objects; and allows multiple participating users in any multi-user mixed reality collaborative environment to place and position different types of virtual objects in the mixed reality collaborative environment and accordingly modify the impact of sounds generated in the environment. It may be advantageous to implement such a system within a mixed reality environment to improve the ease with which a user can place and position virtual objects to hear changes in sound. Such a system may enable a user to perceive sounds as if they were occurring in the mixed reality environment in which the user is present, which provides many potential advantages: for example, the system may allow a user to accurately preview the acoustics of rooms in a property and even simulate the acoustics of the user's belongings virtually placed within the property before the user makes a bid on the property. The system may allow users to preview the acoustic properties of a destination and learn how different objects and materials affect sound by creating virtual objects of different sizes, positions, shapes, and compositions. The system may allow a group of users to collaboratively place objects in a mixed reality space and hear how the sound is affected.

[0018] According to one embodiment, the present invention is a system for recording sound effects that originate outside of a mixed reality environment experienced by a user wearing a mixed reality headset, modulating the sound effects to reflect the presence of virtual and physical objects within the mixed reality environment, and playing the modulated sound effects to the user.

[0019] According to at least one embodiment, the present invention is a system for enabling a user to create or position virtual objects within a mixed reality environment to be experienced by the user, dynamically modulating sound effects originating outside the mixed reality environment to take into account acoustic effects on the sounds of the virtual objects, and playing the modulated sounds to the user.

[0020] According to at least one embodiment, the present invention is a system for enabling a user to modify a mixed reality environment experienced by the user, dynamically modulate sound effects occurring outside the mixed reality environment to take into account the acoustic effects of the modified mixed reality environment, and play the modulated sounds to the user.

[0021] In some embodiments of the present invention, a sound effect may be any sound that originates outside of the mixed reality environment. For example, a sound effect may originate in the user's physical environment and be close enough or loud enough for the user to hear it with the naked ear. A sound effect may also originate in the user's virtual environment but is not part of the mixed reality environment; for example, the sound may be music or voice chat emanating from an application separate from the application generating the mixed reality environment in which the user is currently immersed. In some embodiments, for example, when there are multiple coordinated users, the sound effect may originate in the vicinity of at least one of the users.

[0022] As referred to herein, a user's physical environment can be the user's real-world surroundings, consisting of all physical elements of the user's location, such as soil, air, sky, trees, furniture, animals, people, etc. A virtual environment can be the user's virtual surroundings, i.e., all virtual elements that the user can see and / or interact with, including virtual objects rendered by a processor on dedicated display hardware. Virtual objects can be objects of any shape, size, location, material composition, etc., and often mimic or simulate physical objects, simulated through software within the virtual environment, such as an application's visual interface, computer-generated text or symbols, or images or objects overlaid on the user's field of view, a gaming environment the user sees through a headset, etc.

[0023] A mixed reality environment can be an environment experienced by one or more users through devices such as VR headsets, smart glasses, tablets, mobile phones, etc., that comprises a composite of the user's physical and virtual environments; a mixed reality environment can be any combination of both virtual and physical elements. Virtual elements within a mixed reality experience can include digital components of the virtual experience modeled in the virtual environment that may be mapped to or otherwise correspond to places and objects in the physical world, including virtual scenes and augmented reality elements such as safety instructions, advertisements, navigation instructions, virtual signage, etc.; virtual elements can be interactive, responding to participants' movements, speech, orientation, etc. The ratio of virtual elements to physical elements can vary significantly; for example, at one end of the spectrum, a mixed reality experience can be primarily virtual with minimal physical elements, such as an interactive virtual environment mapped to physical locations, such that the movements of one or more participants in the virtual environment are mapped to the location and movements of those participants in the physical world. At the other end of the spectrum, a mixed reality experience can be primarily physical with minimal virtual elements, such as a virtual navigation aid overlaid on the user's physical environment.

[0024] In some embodiments of the present invention, virtual objects may be created by users, and existing virtual objects may be edited; editing may include changing the shape, size, position, material composition, and other properties of the virtual object. In some embodiments, for example, when there are multiple coordinated users, any number of users may create or edit virtual objects and may perceive virtual objects created by any or all other users participating in the coordinated multi-user environment, and each individual participating user may hear sound effects occurring within the user's own external environment and / or emanating from the external virtual and / or physical environments of all or a subset of the other coordinated users.

[0025] In some embodiments of the present invention, the sound effects may be properties of an object and / or material that dictate how the object and / or material responds to sound waves. The system may simulate the sound effects of a virtual object by modifying the sound effects based on the material, shape, and position attributed to the virtual object, the location of the sound effect source, the user's position relative to the object and the virtual object, etc., so that the sound effects sound to the user in the same way as they would sound if the virtual object were physical.

[0026] In some embodiments of the present invention, a user may modify a mixed reality environment by changing settings, where the setting comprises virtual elements such as time, geography, terrain features, and / or structures with the user's visible surroundings. For example, a user may be physically located in the living room of the user's home, but the user may modify the setting of the mixed reality environment to be a hilly outdoor area, whereby sounds may be modulated to reflect the hilly outdoor area. In some embodiments, a user may modify the mixed reality environment by changing conditions, where conditions may be circumstances that modify settings such as the presence and severity of weather phenomena, atmospheric conditions such as humidity, whether the setting is underwater, etc., and change acoustic conditions.

[0027] In some embodiments of the present invention, the mixed reality environment may be a multi-user collaborative environment, where multiple participating users may collaboratively place and position different types of virtual objects, and where all or a subset of the participating users may hear sounds outside the multi-user collaborative environment modulated to reflect the virtual objects within the multi-user collaborative environment. In some embodiments, the multiple participating users may modify the mixed reality environment by changing settings and / or conditions.

[0028] In some embodiments of the present invention, a user can select a sample sound effect, and the system can graphically show the user in the mixed reality environment what changes need to be applied to the current surroundings to achieve the matching sound effect. For example, a user can desire a selected sound effect, such as a recording of the user saying "hello" in the user's living room, and the system can display what changes need to be applied to the living room, such as repositioning a sofa, reorienting a wall, and adding thicker carpeting, so that a similar or matching sound effect can be created. The system can display changes in the mixed reality environment by creating virtual objects to represent new or moved objects in the surroundings, and can remove virtual objects and / or mask physical objects that will be removed or relocated.

[0029] In some embodiments of the invention, the system may create a knowledge corpus, which is a body of knowledge accessible to the system that comprises general knowledge about the properties of sound, sound reflections, properties of reflections, and related physical properties, including the acoustic properties of different materials, sound transmission media, shapes and sizes of surfaces and obstacles, changes in sound propagation over distance, etc. The system may reference the knowledge corpus to modulate sounds. In some embodiments, the knowledge corpus may include specific data about the effects of particular objects, settings, conditions, etc. on particular sounds, for example, based on historical data and real-world examples provided by the system as feedback.

[0030] The present invention may be a system, method and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0031] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.

[0032] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may comprise copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium within the respective computing / processing device for storage.

[0033] The computer-readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or may be source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk® or C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer-readable program instructions may run entirely on the user's computer, as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions to personalize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the present invention.

[0034] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0035] These computer-readable program instructions may be provided to a processor of a computer, or other programmable data processing apparatus, to produce a machine whereby the instructions, executed via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, whereby the computer-readable storage medium having instructions stored therein has an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0036] Furthermore, computer-readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, whereby the instructions executing on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0037] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may actually be realized as a single step and executed concurrently, substantially concurrently, partially, or fully in a time-overlapping manner, or the blocks may possibly be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0038] The exemplary embodiments described below provide systems, methods, and program products that modulate sound effects that originate outside of a mixed reality environment to reflect the combined virtual and physical conditions of the mixed reality environment.

[0039] 1, an exemplary networked computing environment 100 is shown in accordance with at least one embodiment. Networked computing environment 100 may include a client computing device 102, a mixed reality device 118, and a server 112 interconnected via a communications network 114. According to at least one implementation, networked computing environment 100 may include multiple client computing devices 102, mixed reality devices 118, and servers 112, although only one of each is shown for simplicity of illustration.

[0040] The communications network 114 may include various types of communications networks, such as a wide area network (WAN), a local area network (LAN), a telecommunications network, a wireless network, a public switched telephone network, and / or a satellite network. The communications network 114 may include connections, such as wired, wireless communication links, or fiber optic cables. It should be understood that Figure 1 is merely provided as an illustration of one implementation and does not suggest any limitation with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made based on design and implementation requirements.

[0041] According to one embodiment of the present invention, client computing device 102 may include processor 104 and data storage device 106 capable of hosting and executing knowledge corpus 108 and mixed reality sound modulation program 110A and communicating with server 112 via communication network 114. Client computing device 102 may be, for example, a mobile device, a phone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device capable of executing programs and accessing a network. In some embodiments, client computing device 102 may be and / or integrated into mixed reality device 118. As discussed with respect to FIG. 5, client computing device 102 may each include internal components 502a and external components 504a.

[0042] The mixed reality (MR) device 118 may be any device that allows a user to perceive a mixed reality environment; the mixed reality device 118 may be any device equipped with a display capable of rendering a virtual environment and hardware or software that allows the device to track its position and movement relative to the physical world and, therefore, relative to virtual objects mapped to positions in the physical world. The mixed reality device 118 may be a general-purpose device owned by a user, or may be customized or specialized for an individual mixed reality experience or class of mixed reality experiences. The mixed reality device 118 may include devices such as a VR headset, an AR headset, smart glasses, a tablet, a mobile phone, etc. The user may wear or utilize the mixed reality device 118 while experiencing the mixed reality environment. According to one embodiment of the present invention, the mixed reality device 118 may be capable of communicating with the user experience orchestrator program 110A, 110B residing in the client computing device 102 and / or the server 112 via a communication network 114.

[0043] In some embodiments of the invention, the mixed reality device 118 or the client computing device 102 may be equipped with or in communication with one or more microphones for recording sounds occurring in the user's physical environment. The microphones may be positioned within the user's general vicinity and, for example, integrated into a headset or mobile device worn by the user, such that sounds recorded by the microphones are audible to the user's naked ears, regardless of whether the user actually hears the sounds.

[0044] In some embodiments of the present invention, the mixed reality device 118 or the client computing device 102 may be equipped with or in communication with one or more speakers for playing modulated sounds to the user. The speakers may be devices attached to or in close proximity to the user's ears so that sounds may be played directly to the user. In some embodiments of the present invention, the speakers may be equipped with active or passive noise attenuation features, such as sound isolation or noise cancellation techniques, to reduce the likelihood that the user will hear sounds occurring in the user's physical environment, thereby allowing the user to hear modulated versions of such sounds played through the speakers.

[0045] According to an embodiment of the present invention, the server computer 112 may be a laptop computer, a netbook computer, a personal computer (PC), a desktop computer, or any programmable electronic device or any network of programmable electronic devices capable of hosting and executing the mixed reality sound modulation program 110B and the database 116 and communicating with the client computing device 102 via the communications network 114. As discussed with respect to FIG. 5 , the server computer 112 may include internal components 502b and external components 504b, respectively. The server 112 may also operate in a cloud computing service model, such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). The server 112 may also be located within a cloud computing deployment model, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud.

[0046] According to this embodiment, the mixed reality sound modulation programs 110A, 110B may be programs capable of modulating sound effects originating outside the mixed reality environment to reflect the combined virtual and physical conditions of the mixed reality environment. The mixed reality sound modulation programs 110A, 110B may be located on the client computing device 102, the server 112, or any other device located within the network 114. Furthermore, the mixed reality sound modulation programs 110A, 110B may be distributed in operation across multiple devices, such as the client computing device 102, the mixed reality device 118, and / or the server 112. The mixed reality sound modulation method is described in further detail below with respect to FIG. 2.

[0047] 2, an operational flowchart illustrating a mixed reality sound modulation process 200 according to at least one embodiment is shown. At 202, the mixed reality sound modulation programs 110A, 110B create a knowledge corpus 108 comprising data on how environmental factors affect sound characteristics. The knowledge corpus 108 may be a body of knowledge accessible to the mixed reality sound modulation programs 110A, 110B comprising general knowledge about sound characteristics, sound reflections, reflection characteristics, and related physical properties, including acoustic properties of different materials, sound transmission media, shapes and sizes of surfaces and obstacles, changes in sound propagation over distance, etc. In some embodiments, the knowledge corpus 108 may include specific data on the effects of particular objects, settings, conditions, etc. on particular sounds, based on historical data and real-world examples, for example, provided as feedback by the mixed reality sound modulation programs 110A, 110B. In some embodiments of the present invention, the mixed reality sound modulation programs 110A, 110B can create a knowledge corpus 108 by storing such general and specific data, or the knowledge corpus 108 can be provided in advance to the mixed reality sound modulation programs 110A, 110B. In some embodiments of the present invention, the knowledge corpus 108 can include machine learning algorithms that use a knowledge base to detect patterns and infer how sound effects interact with virtual objects.

[0048] At 204, the mixed reality sound modulation program 110A, 110B records sound effects originating outside the mixed reality (MR) environment of the user operating the MR device. The sound effects can be any sound originating outside the mixed reality environment. For example, the sound effects can originate in the user's physical environment and be close enough or loud enough for the user to hear it with the naked ear, such as speech from the user or others nearby. The sound effects can also originate in the user's virtual environment but are not part of the mixed reality environment; for example, the sound can be music or voice chat emanating from an application separate from the application generating the mixed reality environment in which the user is currently immersed. In some embodiments, for example, when there are multiple collaborative users, the sound effects can originate in the vicinity of at least one user. The mixed reality sound modulation program 110A, 110B can record the sound effects with, for example, a microphone integrated into the virtual reality device 118 or the client computing device 102.

[0049] At 206, the mixed reality sound modulation program 110A, 110B identifies all virtual and physical objects in the user's MR environment. The mixed reality sound modulation program 110A, 110B may identify virtual objects by receiving information about the virtual objects from a program creating the mixed reality environment. The mixed reality sound modulation program 110A, 110B may identify physical objects by using object recognition to identify physical objects in images recorded by a camera attached to a user's wearable device, such as a mixed reality headset or phone. The mixed reality sound modulation program 110A, 110B may also identify physical objects in the user's environment using any other method, such as sonar, lidar, etc. In some embodiments of the present invention, the mixed reality sound modulation program 110A, 110B may identify all virtual and / or physical objects within a threshold distance of the user, which may represent a distance at which sounds audible to the user may be affected by the acoustic characteristics of objects within the threshold distance. In some embodiments of the present invention, the mixed reality sound modulation program 110A, 110B may only identify physical objects in the user's mixed reality environment when the sound effects originate from the user's virtual environment, for example, when the sound effects are music or voices from a background application that are not naturally affected by physical objects in the user's physical environment, as are sounds that originate from the user's physical environment.

[0050] In some embodiments of the present invention, the mixed reality sound modulation program 110A, 110B may continuously identify virtual and / or physical objects in the user's MR environment to maintain an updated map of the user's mixed reality surroundings, for example, by identifying objects at regular time intervals, such as every second. In some embodiments of the present invention, the mixed reality sound modulation program 110A, 110B may identify virtual and / or physical objects in the MR environment in response to changes in the objects in the user's MR environment, for example, each time the user creates, deletes, or edits a virtual object, modifies a setting or condition, moves around beyond a threshold margin such that physical and virtual objects are in a new position relative to the user, or modifies a physical object by moving the object or changing its shape, orientation, color, etc. In at least such embodiments, the mixed reality sound modulation program 110A, 110B may continuously monitor the mixed reality environment for physical or virtual changes and may track the location and movement of the mixed reality device 118 and / or the user using location tracking methods such as accelerometers, RFID tags, GPS, camera-based object and face detection, etc.

[0051] At 208, the mixed reality sound modulation programs 110A, 110B modulate the recorded sound effects to reflect the presence of virtual and physical objects in the user's mixed reality environment according to the knowledge corpus 108. The mixed reality sound modulation programs 110A, 110B may modulate the recorded sound effects to reflect the presence of virtual and / or physical objects in the user's mixed reality environment by altering the amplitude or frequency of the sound effects. In some embodiments of the present invention, the mixed reality sound modulation programs 110A, 110B may alternatively or additionally modulate the sound effects to represent settings or conditions present in the mixed reality environment. The mixed reality sound modulation programs 110A, 110B may reference the knowledge corpus 108 to determine how the sounds are affected by the presence, setting, and / or conditions of virtual and / or physical objects. In some embodiments of the present invention, the mixed reality sound modulation programs 110A, 110B may modulate sound effects in response to changes in objects within the user's MR environment; for example, whenever the user creates or edits a virtual object, modifies a setting or condition, moves physical and virtual objects around beyond a threshold margin such that they are in a new position relative to the user, modifies a physical object, such as by moving the object or changing its shape, orientation, color, etc.

[0052] At 210, the mixed reality sound modulation program 110A, 110B plays the modulated sound effects to the user. Here, the mixed reality sound modulation program 110A, 110B plays the modulated sound effects to the user through speakers. In some embodiments of the present invention, such as when multiple users are present in the same virtual environment, e.g., in a multi-user collaborative environment, the mixed reality sound modulation program 110A, 110B may play the modulated sound effects to a subset of users based on the preferences of the individual users. For example, users in the same physical environment can choose to hear modulated sound effects emanating from their environment and can choose not to hear sound effects emanating from other users' physical and / or virtual environments.

[0053] 3, a diagram illustrating a use case 300 of a mixed reality sound modulation system according to at least one embodiment is shown. Here, a first user 302 and a second user 304, each equipped with a mixed reality device 118, are seated in a physical environment 306, i.e., a living room, but are experiencing a mixed reality environment 314 that is an open outdoor landscape. In this mixed reality landscape 314, the first user 302 is represented by a virtual avatar 310, and the second user 304 is represented by a virtual avatar 312. A sound effect 308 occurs in the physical environment 306 of users 302 and 304. The sound effect 308 may be of the first user 302 speaking to the second user 304; the sound effect 308 does not reverberate in the physical environment 306 because it is an indoor space. However, because the mixed reality environment 314 is outdoors, the mixed reality sound modulation programs 110A, 110B modulate the sound so that the second user 304 hears the first user 302's voice with reverberation.

[0054] 4A, a diagram illustrating a use case 400 of a mixed reality sound modulation system according to at least one embodiment is shown, in which a user 302 equipped with a mixed reality device 118 is experiencing a mixed reality environment 402 comprising a collection of physical objects 404, including a door, a table, and two chairs. Sound effects 406 are played in the user's 302 physical environment, and mixed reality sound modulation programs 110A, 110B modulate the sound effects 406 based on the physical objects 404 in the mixed reality environment 402 and play the modulated sound effects 406 to the user.

[0055] 4B, user 302 has added virtual object 406 to mixed reality environment 402. Sound effects 406 are played in user 302's physical environment, and mixed reality sound modulation programs 110A, 110B modulate sound effects 406 based on physical objects 404 and virtual objects 408 in mixed reality environment 402, and play modulated sound effects 406 to user 302 that reflect the addition of virtual object 408 to mixed reality environment 402.

[0056] In an alternative example, the user 302 may select the sample sound effect 406 from a list of sound effects already modulated based on a particular environment; the mixed reality sound modulation program 110A, 110B may identify the mixed reality environment associated with the sample sound effect 406, for example, by accessing the environment that existed when the mixed reality sound modulation program 110A, 110B originally modulated the sound effect 406, or by analyzing the modulation of the sound effect 406 to determine what virtual and physical objects, conditions, and / or settings needed to be present in the mixed reality environment of the sound effect 406 to generate the modulation. The mixed reality sound modulation program 110A, 110B may determine that a virtual object 408 of a particular size and shape needs to be added to the user's mixed reality environment 402 to match the mixed reality environment that the sound effect 406 was originally modulated to match, and may create the virtual object 408 accordingly.

[0057] It should be appreciated that Figures 2-4 are merely illustrative of one implementation and are not intended to suggest any limitations on how different embodiments may be implemented. Many modifications to the depicted environments may be made based on design and implementation requirements.

[0058] 5 is a block diagram 500 of internal and external components of the client computing device 102 and the server 112 shown in FIG. 1 in accordance with an embodiment of the present invention. It should be appreciated that FIG. 5 is merely provided as an illustration of one implementation and is not intended to suggest any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made based on design and implementation requirements.

[0059] Data processing systems 502, 504 represent any electronic device capable of executing machine-readable program instructions. Data processing systems 502, 504 may represent smartphones, computer systems, PDAs, or other electronic devices. Examples of computing systems, environments, and / or configurations that may be represented by data processing systems 502, 504 may include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0060] The client computing device 102 and the server 112 may include respective sets of internal components 502a, b and external components 504a, b shown in Figure 5. Each of the set of internal components 502 includes one or more processors 520, one or more computer-readable RAMs 522, and one or more computer-readable ROMs 524 on one or more buses 526, one or more operating systems 528, and one or more computer-readable tangible storage devices 530. One or more operating systems 528, software programs 108, and mixed reality audio modulation program 110A in the client computing device 102 and mixed reality audio modulation program 110B in the server 112 are stored in one or more of the respective computer-readable tangible storage devices 530 for execution by one or more of the respective processors 520 via one or more of the respective RAMs 522 (which typically include cache memory). In the embodiment shown in Figure 5, each of the computer-readable tangible storage devices 530 is an internal hard drive magnetic disk storage device. Alternatively, each of the computer-readable tangible storage devices 530 is a semiconductor storage device such as a ROM 524, an EPROM, a flash memory, or any other computer-readable tangible storage device capable of storing computer programs and digital information.

[0061] Each set of internal components 502a,b also includes a R / W drive or interface 532 for reading from and writing to one or more portable computer-readable tangible storage devices 538, such as CD-ROMs, DVDs, memory sticks, magnetic tapes, magnetic disks, optical disks, or semiconductor storage devices. Software programs, such as mixed reality audio modulation programs 110A, 110B, may be stored on one or more of the respective portable computer-readable tangible storage devices 538, read via the respective R / W drive or interface 532, and loaded onto the respective hard drives 530.

[0062] Each set of internal components 502a, b also includes a network adapter or interface 536, such as a TCP / IP adapter card, a wireless Wi-Fi interface card, a 3G or 4G wireless interface card, or other wired or wireless communication link. The software program 108 and mixed reality audio modulation program 110A in the client computing device 102 and the mixed reality audio modulation program 110B in the server 112 may be downloaded to the client computing device 102 and the server 112 from an external computer via a network (e.g., the Internet, a local area network, or other wide area network) and the respective network adapters or interfaces 536. From the network adapters or interfaces 536, the software program 108 and mixed reality audio modulation program 110A in the client computing device 102 and the mixed reality audio modulation program 110B in the server 112 are loaded onto their respective hard drives 530. The network may include copper wire, optical fiber, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers.

[0063] Each of the set of external components 504a,b may include a computer display monitor 544, a keyboard 542, and a computer mouse 534. The external components 504a,b may also include touch screens, virtual keyboards, touchpads, pointing devices, and other human interface devices. Each of the set of internal components 502a,b also includes a device driver 540 that interfaces with the computer display monitor 544, the keyboard 542, and the computer mouse 534. The device driver 540, the R / W drive or interface 532, and the network adapter or interface 536 comprise hardware and software (stored in the storage device 530 and / or ROM 524).

[0064] While this disclosure includes embodiments for implementing the invention with respect to cloud computing, it is understood in advance that implementation of the teachings recited herein is not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0065] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with the service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0066] The characteristics are as follows: On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed, without requiring human interaction with the service provider. Wide network access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, and PDAs). Resource Pooling: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, with various physical and virtual resources dynamically allocated and reallocated according to demand. Consumers generally have no control or knowledge over the exact location of the resources provided, although there is some location independence in that it may be possible to specify location at a higher level of abstraction (e.g., country, state, or data center). Rapid scalability: Capacity is provisioned quickly and elastically, sometimes automatically, and can be instantly scaled out or quickly released and instantly scaled in. In many cases, the capacity available for provisioning appears unlimited to the consumer, and can be purchased in any quantity at any point in time. Metering Services: Cloud systems automatically control and optimize resource utilization by leveraging metering capabilities appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts) at a certain level of abstraction. Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services used.

[0067] The service model is as follows: Software as a Service (SaaS): The consumer is offered the ability to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through a thin-client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings. Platform as a Service (PaaS): The ability offered to consumers is to deploy applications they create or acquire, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but controls the deployed applications and, in some cases, the configuration of the application hosting environment. Infrastructure as a Service (IaaS): The ability offered to consumers is to provision processing, storage, network, and other basic computing resources, upon which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but controls the operating system, storage, deployed applications, and, in some cases, limited control over selected networking components (e.g., host firewalls).

[0068] The deployment models are as follows: Private Cloud: The cloud infrastructure operates solely for an organization. It may be managed by the organization or a third party and may reside on-premises or off-premises. Community Cloud: The cloud infrastructure is shared by several organizations and supports a specific community with common concerns (e.g., mission, security requirements, policies, and compliance considerations). It may be managed by the organization or a third party and may reside on-premises or off-premises. Public Cloud: The cloud infrastructure is made available to the general public or large industry associations and is owned by organizations that sell cloud services. Hybrid Cloud: The cloud infrastructure remains a unique entity but is a combination of two or more clouds (private, community, or public) joined by standardized or proprietary technologies that allow for data and application portability (e.g., cloud bursting for load balancing between clouds).

[0069] A cloud computing environment is a service oriented environment with an emphasis on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that comprises a network of interconnected nodes.

[0070] Referring now to FIG. 6, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 comprises one or more cloud computing nodes 100 with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, and / or an automobile computer system 54N, may communicate. The nodes 100 may communicate with each other. They may be physically or virtually grouped (not shown) into one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or combinations thereof, as described hereinabove. This enables the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service for which cloud consumers are not required to maintain resources on their local computing devices. It is understood that the types of computing devices 54A-N shown in FIG. 6 are intended for illustrative purposes only, and that the computing nodes 100 and the cloud computing environment 50 may communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0071] 7, there is shown a set of functional abstraction layers 700 provided by the cloud computing environment 50. It should be understood in advance that the components, layers, and functions shown in FIG. 7 are intended to be illustrative only, and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0072] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframe 61; RISC (reduced instruction set computer) architecture-based servers 62; servers 63; blade servers 64; storage devices 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0073] The virtualization layer 70 provides an abstraction layer over which the following examples of virtual entities can be provided: virtual servers 71; virtual storage 72; virtual networks, including virtual private networks 73; virtual applications and operating systems 74; and virtual clients 75.

[0074] In one example, management layer 80 may provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and accounting or billing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection of data and other resources. User portal 83 provides access to the cloud computing environment to consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management so that required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides advance arrangements and procurement of cloud computing resources where future requirements are anticipated according to SLAs.

[0075] The workload tier 90 provides examples of functions for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this tier include mapping and navigation 91; software development and lifecycle management 92; virtual classroom instructional delivery 93; data analytics processing 94; transaction processing 95; and mixed reality sound modulation 96. Mixed reality sound modulation 96 may be capable of modulating sound effects that originate outside of the mixed reality environment to reflect the combined virtual and physical conditions of the mixed reality environment.

[0076] The description of various embodiments of the present invention has been presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been selected to best explain the principles of the embodiments, practical applications, or technical improvements over technologies found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A processor-implemented method for operating a mixed reality device, comprising: recording sound effects originating outside of the mixed reality environment experienced by a user operating the mixed reality device; identifying one or more objects in the mixed reality environment, including at least one virtual object; modulating the sound effects to simulate one or more sound effects of the one or more objects in the mixed reality environment; graphically indicating to the user within the mixed reality environment one or more modifications of the at least one virtual object required to configure the mixed reality environment to play the sample sound selected by the user; and playing the modulated sound effect to the user. A method comprising:

2. The method of claim 1 , wherein the modulating step is performed dynamically in response to the virtual object being created or edited by the user.

3. The method of claim 1 or 2, further comprising modulating the sound effects to simulate one or more sound effects of a setting or condition of the mixed reality environment as modified by the user.

4. The method of claim 3 , wherein the modulating step is performed dynamically in response to the user modifying the settings or conditions of the mixed reality environment.

5. The method of claim 1 or 2, wherein the mixed reality environment comprises a multi-user collaborative environment, in which multiple participating users can position or edit the at least one virtual object.

6. The method of claim 1 or 2, wherein the modulating step is based on a created knowledge corpus.

7. 1. A computer system for operating a mixed reality device, comprising: one or more mixed reality devices, one or more microphones, one or more speakers, one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more tangible storage media for execution by at least one of the one or more processors via at least one of the one or more memories. wherein the computer system comprises: recording sound effects originating outside of the mixed reality environment experienced by a user operating said mixed reality device; identifying one or more objects in the mixed reality environment, including at least one virtual object; modulating the sound effects to simulate one or more sound effects of the one or more objects in the mixed reality environment; graphically indicating to the user within the mixed reality environment one or more modifications of the at least one virtual object required to configure the mixed reality environment to play the sample sound selected by the user; and playing the modulated sound effect to the user. A computer system capable of performing a method comprising:

8. The computer system of claim 7 , wherein the modulating step is performed dynamically in response to the virtual object being created or edited by the user.

9. 9. The computer system of claim 7 or 8, further comprising modulating the sound effects to simulate one or more sound effects of a setting or condition of the mixed reality environment as modified by the user.

10. The computer system of claim 9 , wherein the modulating step is performed dynamically in response to the user modifying the settings or conditions of the mixed reality environment.

11. 9. The computer system of claim 7 or 8, wherein the mixed reality environment comprises a multi-user collaborative environment, in which multiple participating users can position or edit the at least one virtual object.

12. 9. The computer system of claim 7 or 8, wherein the modulating procedure is based on a created knowledge corpus.

13. 1. A computer program for operating a mixed reality device, comprising: recording sound effects originating outside of the mixed reality environment experienced by a user operating said mixed reality device; identifying one or more objects in the mixed reality environment, including at least one virtual object; modulating the sound effects to simulate one or more sound effects of the one or more objects in the mixed reality environment; graphically indicating to the user within the mixed reality environment one or more modifications of the at least one virtual object required to configure the mixed reality environment to play the sample sound selected by the user; and playing the modulated sound effect to the user. A computer program that executes

14. The computer program product of claim 13 , wherein the modulating step is performed dynamically in response to the virtual object being created or edited by the user.

15. the processor, 15. The computer program product of claim 13 or 14, further comprising modulating the sound effects to simulate one or more sound effects of a setting or condition of the mixed reality environment as modified by the user.

16. The computer program product of claim 15 , wherein the modulating step is performed dynamically in response to the user modifying the settings or conditions of the mixed reality environment.

17. 15. The computer program product of claim 13 or 14, wherein the mixed reality environment comprises a multi-user collaborative environment, in which multiple participating users can position or edit the at least one virtual object.

Images