System and method for training a model to determine the type of environment surrounding a user - Patents.com

The system uses acoustic profiling and machine learning to enable users to identify objects in their environment through sound, addressing the challenge of visual impairment in HMDs, enhancing safety and gameplay experience.

JP2024544017AActive Publication Date: 2024-11-26SONY INTERACTIVE ENTERTAINMENT LLC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024533065
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-12-03
Filing Date
2022-11-18
Publication Date
2024-11-26
Estimated Expiration
2042-11-18

AI Technical Summary

Technical Problem

In multiplayer games, players wearing head-mounted displays (HMDs) cannot see the environment around them, leading to challenges in identifying objects and navigating safely.

Method used

A system using microphones to detect acoustic characteristics of the environment, employing machine learning to identify objects and generate acoustic profiles, allowing users to perceive their surroundings through sound-based 'acoustic vision', which can be combined with camera data for enhanced object identification.

Benefits of technology

Enables users to recognize objects in their environment without visual input, providing safety and enhancing gameplay experience by accurately identifying objects and environments using acoustic profiling and machine learning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024544017000001_ABST
    Figure 2024544017000001_ABST
Patent Text Reader

Abstract

A method is described for determining an environment in which a user is located. The method includes receiving a plurality of audio data sets based on sounds emitted in the plurality of environments, each of the plurality of environments having a different combination of objects. The method further includes receiving input data related to the plurality of environments and training an artificial intelligence (AI) model based on the plurality of audio data sets and the input data. The method includes applying the AI ​​model to audio data captured from an environment surrounding the first user to determine a type of environment.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a system and method for training a model that determines the type of environment surrounding a user. [Background technology]

[0002] In a multiplayer game, there are multiple game players. Each player wears a head mounted display (HMD) and plays the game or sees the environment of the application. While playing the game or running the application, each player may not be able to see the environment in front of him / her.

[0003] It is in this context that embodiments of the present invention arise. Summary of the Invention

[0004] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure provide systems and methods for training a model to determine the type of environment surrounding a user.

[0005] In an embodiment, one or more microphones are used to detect acoustic characteristics of the current environment. An example of the current environment is a location, such as a room in a house or building where the user is located, or an outdoor environment. Another example of the current environment is a location, such as a room in a building, where the user is playing a video game and making selections to generate input to the video game.

[0006] In one embodiment, the acoustic profile is generated in response to a user request to generate an acoustic profile of an environment. The acoustic profile is generated by capturing sounds in an environment, such as a space or room, including sound reflections and reverberations from various objects located in the environment. The configuration of the environment around the user, or around a microphone or microphone array, is used to define the current environment.

[0007] In an embodiment, the environment includes sounds generated as background noise. Background noise may be generated by other people talking, music playing, extraneous sounds from outside, noises made by mechanical objects in or around the environment, etc. By profiling the soundscape of the environment, a machine learning system can be used to identify certain identifying characteristics. For example, training can be performed by running an environment profiler application in many different environments. Over time, the learning process using the machine learning system identifies objects in or around the different environments that generate noise or sound. In profiling, the machine learning system also identifies the acoustics of the different environments in which the sound is being monitored. Once training has been processed to a sufficient level using the machine learning system, the machine learning model trained on the sound profiles of the different environments is used to automatically identify things, objects, or sounds present in the current environment. These objects are present in the current environment, generate and reflect sounds, and these sounds are detected to generate the acoustics of the current environment. For example, if a user is playing a game in front of one or more monitors, there will be reflections of sounds coming from the monitors during gameplay. These types of reflections are processed using a machine learning system to identify unique characteristics and determine that the user is playing the game in front of another monitor. This type of characteristic identification can be used to detect other objects within the current environment in which the user is located.

[0008] In an embodiment, the microphone array may be mounted on a user peripheral device, such as eyeglasses, augmented reality (AR) glasses, a head mounted display (HMD) device, or a handheld controller.

[0009] In one embodiment, during training, if the microphone array is located on the glasses or HMD, the user is asked to turn his / her head in different environments to capture different sounds. In another embodiment, the configuration of the different environments is done passively, and the audio signals and acoustics of objects in the different environments are tracked over time as the user moves from one of the different environments to another one of the different environments.

[0010] In an embodiment, profiling of various environments for acoustic properties provides a type of acoustic vision of the current environment. For example, if a microphone array is located on an HMD or AR glasses or spectacles, as the user moves and looks around the various environments, the machine learning system can almost instantly identify what is in front of the user based on the acoustic reflections and bounceback of signals from the current environment.

[0011] In an embodiment, as a person moves within the current environment, the acoustic signal in front of the user changes, and based on the profile of the acoustic signal, acoustic vision can be used to identify or verify what may be present in front of the user. In an embodiment, acoustic vision of various environments can be blended with data received from the camera to identify or verify that an object is present based on the acoustic profile of the object in front of the user in the current environment.

[0012] In another embodiment, it is possible to create a virtual profile of a space. For example, if a user wants to appear to be located in a particular location, e.g., a concert, a park, a gaming event, a studio, etc., a machine learning system can generate sounds using a known acoustic profile. The sounds generated based on the acoustic profile can be blended with the sounds generated by the application program so that the user appears to be at a particular location, rather than the user's actual location. For example, if a user is publishing a YouTube®™ video, the generated soundscape can be customized for the user based on the type of current environment they want to project or virtually project to third parties watching the YouTube®™ video. For example, if a user wants to provide commentary of a sporting event, a soundscape can be virtually generated behind the commentary to mimic the sporting event, taking into account the acoustic profile that exists or may exist at the sporting event.

[0013] In an embodiment, a method is described for mapping the acoustic properties of materials, surfaces, and geometry of a space. The mapping is performed by extracting reverberation information, such as reflections and diffusion of sound detected by one or more microphones. Audio data, once sound-based and captured by one or more microphones, is separated into direct and reverberant components. The direct component is then resynthesized with different reverberant characteristics, effectively replacing or modifying the listener's acoustic environment with some other acoustic profile. The other acoustic profiles are used in combination with visual or geometric mapping of the space, such as by cameras, SLAM (Simultaneous Localization And Mapping), or LiDAR (Light Detection and Ranging), to build a more complete audiovisual mapping. The reverberant component is also used to convey features about the geometry of the space and the properties of materials and surfaces.

[0014] In one embodiment, a method for determining an environment in which a user is located is described. The method includes receiving a plurality of audio data sets based on sounds emitted in the plurality of environments. Each of the plurality of environments has a different combination of objects. The method further includes receiving input data related to the plurality of environments and training an artificial intelligence (AI) model based on the plurality of audio data sets and the input data. The method includes applying the AI ​​model to audio data captured from an environment surrounding the first user to determine a type of environment.

[0015] Some advantages of the systems and methods described herein include helping a user or robot learn objects in an environment without having to acquire images of the objects. For example, a robot learns the identity of objects in an environment, the placement of objects, and the state of objects without acquiring images of the objects. Once the robot has learned about the objects, it can be programmed to navigate around the objects and then deployed in the environment and used within the environment.

[0016] Further advantages of the systems and methods described herein include providing a visually impaired person with a layout of an environment, which is determined based on sounds emitted by objects in the environment before the visually impaired person visits the environment, thereby making the layout perceptible to the visually impaired person.

[0017] Further advantages of the systems and methods described herein include providing the identity of objects and the location of objects in an environment in front of a user when the user is wearing an HMD. When a user wears an HMD, such as in a virtual reality (VR) mode, the user may not be able to see the environment in front of him / her. The systems and methods described herein facilitate providing the identity of objects and the location of objects to a user to prevent accidents between the user and the objects.

[0018] Other aspects of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the embodiments described in the present disclosure.

[0019] The various embodiments of the disclosure can be best understood by referring to the following description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0020] [Figure 1A-1] FIG. 1 is a diagram of an embodiment of an environment in which a user is playing a game. [Figure 1A-2] FIG. 1 illustrates an embodiment of a system including an input controller, glasses, and a server system. [Figure 1A-3] FIG. 13 is an illustration of an embodiment showing a training session in which a user is asked to turn his or her head to capture a view of the environment. [Figure 1A-4] FIG. 2 is a diagram of an embodiment of a list of objects displayed by a graphical processing unit (GPU) to generate an acoustic profile of multiple objects. [Figure 1B] FIG. 1 illustrates an embodiment of an environment in which multiple users are playing a game. [Diagram 2] FIG. 1 is a diagram of an embodiment of a system for illustrating processing an input dataset and an audio dataset to train an artificial intelligence (AI) model. [Diagram 3] FIG. 1 is a diagram of an embodiment of a system for illustrating multiple correlations between an audio data set and an environmental system. [Figure 4A] FIG. 2 is a diagram of an embodiment of an object state. [Figure 4B] FIG. 2 is a diagram of an embodiment of object types. [Figure 5A] FIG. 1 is a diagram of an embodiment of an environment to illustrate the use of an AI model to identify objects within an environment, determine the state of the objects within the environment, and the placement of the objects within the environment. [Figure 5B] FIG. 1 is a diagram of an embodiment of a model output from an AI model. [Figure 6A] FIG. 1 illustrates an embodiment of an acoustic profile database stored on one or more memory devices. [Figure 6B] FIG. 1 is a diagram of an embodiment of a method to illustrate applying audio data to output a corresponding sound to an environmental system. [Figure 7] FIG. 7 is a diagram of an embodiment of a system for illustrating the method of FIG. 6. [Figure 8] FIG. 1 is a diagram of an embodiment of a system for illustrating a microphone. [Figure 9] FIG. 1 illustrates an embodiment of a system for illustrating how direct audio data is used to create environmental system effects and reverb audio data is used to determine the placement of objects within the environmental system, the material type of the objects, and the surface type of the objects. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] A system and method for training a model to determine the type of environment surrounding a user is described. It should be noted that various embodiments of the present disclosure may be practiced without some or all of these specific details. In other instances, well known process operations have not been described in detail in order to not unnecessarily obscure various embodiments of the present disclosure.

[0022] FIG. 1A-1 is a diagram of an embodiment of an environment 102 in which a user 1 is playing a game G1. An example of the environment 102 is a room in a house 110. The environment 102 includes a number of objects 108A, 108B, 108C, 108D, 108E, 108F, 108G, 108H, 108I, 108J, 108K, 108L, 108M, 108N, and 108O. Object 108A is a chair, object 108B is a side table, object 108C is a display device such as a desktop monitor or a television monitor, and object 108D is a window in a wall of the room in the house 110. Object 108D is a window with closed blinds. Object 108E is a desktop table on which the display device is placed, object 108F is a keyboard coupled to the display device, and object 108G is a mouse coupled to the display device. For example, the keyboard is coupled to a central processing unit (CPU) of the display device, and the mouse is also coupled to the CPU.By way of example, object 108E is made of plastic and has a smooth upper surface.

[0023] Object 108H is a carpet covering the floor of the room, and object 108I is the floor of the room. Object 108J is a wall where a window is located. Object 108K is a speaker coupled to the display device, and object 108L is another speaker coupled to the display device. For example, the speaker is coupled to the CPU of the display device. Note that object 108K is in front of object 108C. Also, object 108M is a soda can or container, and object 108N is a handheld controller coupled to a game console 112. A soda can is open in the environment 102. The game console 112 is coupled to the display device. For example, the game console 112 is coupled to the CPU or graphical processing unit (GPU) of the display device. Object 108O is glasses worn by user 1. Examples of glasses described herein include a head mounted display (HMD), prescription glasses, and augmented reality (AR) glasses.

[0024] Note that object 108P, a vehicle, is located outside of house 110 and passes by house 110 while user 1 is playing game G1. Also note that while user 1 is playing game G1, child 1 and child 2 are playing and talking to each other in different rooms inside house 110. Each of child 1 and 2 is an example of an object.

[0025] The glasses include a camera C1 and a microphone M1 that converts sound into electrical energy. Examples of the camera C1 include a depth camera, a video camera, and a digital camera.

[0026] User 1 accesses game G1 from a game cloud, such as a server system, via a computer network and plays game G1, which has a virtual scene 114 rendered on a display device. For example, user 1 selects one or more buttons on a handheld controller to provide authentication information, such as a username and password. The handheld controller transmits the authentication information to game console 112, which forwards the authentication information to the game cloud via the computer network. An authentication server of the game cloud determines whether the authentication information is authentic and, if so, provides access to user account 1 and a game program executing on the game cloud. When the game program is executed by one or more processors of the game cloud, image frames of game G1 are generated and encoded, and the encoded image frames are output. The encoded image frames are transmitted to game console 112. Game console 112 decodes the encoded image frames and provides them to a display device to display the virtual scene 114 of game G1 and further enable user 1 to play game G1.

[0027] During the play of the game G1, sounds of the game G1 are output from a speaker coupled to the display device. For example, when a virtual object 116A in the virtual scene 114 is shooting another virtual object 116B, a shooting sound is output. As an example, the virtual object 116A is controlled by the user 1 via a handheld controller. During the play of the game G1, a vehicle honks its horn or makes sounds such as engine noise or tire screeching. During the play of the game G1, the children 1 and 2 make sounds by talking to each other, arguing with each other, playing with each other, etc. During the play of the game G1, the user 1 opens a can and makes a can opening sound. Moreover, during the play of the game G1, the user 1 utters words, which are examples of sounds.

[0028] The microphone M1 captures audio data, such as audio frames, generated from sounds associated with, such as sounds emanating from, or reflected from, one or more of the objects 108A-108O located in the environment 102. For example, the microphone M1 captures audio data generated based on sounds emanating from the object 108K and received from the object 108K via a path 106A. In this example, the sound path 106A is a direct path from the speaker to the microphone M1 and does not enter any other surface between the speaker and the microphone M1. As another example, the microphone M1 captures audio data generated based on sounds emanating from the object 108K and received from the object 108K via a path 106B. In this example, the sound path 106B is an indirect path from the object 108K to the microphone M1. For purposes of illustration, sounds emanating from the object 108K enter one or more other objects, such as a display device, in the environment 102 and are reflected from the one or more other objects toward the microphone M1. As yet another example, microphone M1 captures audio data generated based on sound emanating from object 108K and received from object 108K via path 106D. In this example, sound path 106D is an indirect path from the speaker to microphone M1. To illustrate, sound emanating from object 108K impinges on one or more other objects, such as a carpet, in environment 102 and is reflected from the one or more other objects toward microphone M1.

[0029] As another example, microphone M1 captures audio data generated based on sound emanating from object 108L and received from object 108L via path 106D. In this example, sound path 106D is an indirect path from the speaker to microphone M1. For illustration purposes, sound emanating from object 108L is incident on one or more other objects, such as a can, in environment 102 and is reflected from the one or more other objects toward microphone M1. As yet another example, microphone M1 captures audio data generated based on sound emanating from object 108L and received from object 108L via path 106E. In this example, sound path 106E is a direct path from speaker to microphone M1 and does not impinge on any other surface between speaker and microphone M1. As yet another example, microphone M1 captures audio data generated based on sound emanating from object 108L and received from object 108L via path 106F. In this example, sound path 106F is an indirect path from the speaker to microphone M1. For purposes of illustration, sound originating from object 108L impinges on one or more other objects in environment 102, such as a desktop table, and is reflected from the one or more other objects towards microphone M1.

[0030] The microphone M1 of the glasses also captures audio data generated based on sounds, such as background noise, emanating from one or more of the object 108P, child 1, and child 2 located outside but near the environment 102. The environment outside the environment 102 may be referred to herein as the external environment 116. As an example, the microphone M1 captures audio data generated based on sounds emanating from the object 108P and received from the object 108P via the path 106G. In this example, the path 106G extends through a wall or a door or a doorway of the environment 102. In this example, if the door of the environment 102 is open, the sound spreads through the doorway, and if the door is closed, the sound spreads through the door. As another example, the microphone M1 captures audio data generated based on sounds emanating from the child 1 or child 2 or both child 1 and child 2 and received from the child 1 or child 2 or both child 1 and child 2 via the path 106H. In this example, the path 106H extends through a wall or door or doorway of the environment 102.

[0031] Note that an external environment 116 is in proximity to the environment 102 if sound emanating from an object in the external environment 116, such as, for example, a room next to the environment 102 or a street outside the house 110, can reach and be detected by the microphone M1. For example, sound emanating from the external environment 116 passes through a wall of the environment 102 and is detected by the microphone M1.

[0032] Microphone M1 captures audio data, such as audio frames generated from sounds associated with objects 108A-108O in environment 102, such as sounds emanating from or reflected from the objects, and background noises emanating from one or more of object 108P, child 1, and child 2 in external environment 116. Encoded audio frames based on the audio frames captured by microphone M1 are generated and provided to one or more processors of the game cloud via a computer network for processing and training artificial intelligence (AI) models.

[0033] In an embodiment, sound as used herein includes sound waves.

[0034] In one embodiment, the terms object and item are used interchangeably herein.

[0035] In one embodiment, the display device includes a memory device. The CPU of the display device is coupled to the memory device.

[0036] In the embodiment, the sound emitted by the speaker is reflected from the user 1 and captured by the microphone M1. In this embodiment, the user 1 is an example of an object.

[0037] In an embodiment, the virtual scene 114 is not displayed on a display device, but rather on a display screen of glasses worn by the user 1 .

[0038] In an embodiment, the user 1 accesses the game G1 from the game cloud without the need to use the game console 112. For example, the encoded image frames of the game G1 are sent from the game cloud to the glasses or a display device placed on a desktop table through a computer network without being sent to the game console 112 for video decoding. The encoded image frames are decoded by the glasses or the display device. In this embodiment, the encoded audio frames generated based on the audio frames output from the microphone M1 are sent from the glasses to the game cloud through a computer network without using the game console 112.

[0039] In one embodiment, the virtual scene 114 includes other virtual objects, and based on the movement of these other virtual objects, sounds are output from speakers placed on a desktop table.

[0040] In an embodiment, instead of or in addition to microphone M1, one or more additional microphones, such as standalone microphones, are present to capture sounds emanating from objects in environment 102 and from objects located in external environment 116. For example, a display device located on a desktop table includes an additional microphone. As another example, environment 102 includes one or more standalone microphones. As yet another example, a handheld controller includes an additional microphone.

[0041] In one embodiment, User 1 is not playing game G1. In this embodiment, instead of a game program, one or more processors of the game cloud execute another application program, such as a video conferencing application program or a multimedia program. For example, when a video conferencing application is executed, video image frames captured from an additional environment are transferred over a computer network to a display device or game console or to glasses worn by User 1.

[0042] In one embodiment, environment 102 is replaced by an outdoor environment such as a concert, a lake, or a park.

[0043] In one embodiment, instead of or in addition to the sound output from objects 108K or 108L, sound is output from a speaker integrated within object 108C and detected by microphone M1 to capture audio data.

[0044] FIG. 1A-2 is a diagram of an embodiment of system 118, including input controller 122, glasses 120, and server system 136. Server system 136 is an example of a game cloud. Glasses 120 are an example of object 108O (FIG. 1A-1). Handheld controller is an example of input controller 112. Glasses 120 include camera C1, video encoder 124, audio encoder 125, network transport device 126, video decoder 128, GPU 130, display screen 132, and CPU 135. An example of video encoder 124 is a circuit that applies a video conversion protocol, such as a video encoding protocol, to image frames to output encoded data, such as encoded image frames, and provides the encoded data to network transport device 126. For illustrative purposes, video encoder 124 generates I-frames, P-frames, or B-frames, which are examples of encoded image frames. Examples of video encoding protocols, such as video compression protocols, include H.262, H.263, and H.264. An example of the audio encoder 125 is a circuit that compresses audio frames into encoded audio frames. For illustrative purposes, the audio encoder applies an audio encoding protocol, such as lossless or lossy compression, to encode the audio frames into encoded audio frames. An example of lossy compression includes a modified discrete cosine transform (MDCT) for transforming a sampled waveform in the time domain into the frequency domain. Another example of lossy compression is linear predictive coding (LPC), which analyzes generated audio data based on speech sounds.

[0045] An example of a network forwarding device as used herein is a network interface controller, such as a network interface card (NIC). Another example of a network forwarding device is a wireless access card (WAC). An example of a video decoder as used herein is a circuit that performs H.262, H.263, or H.264 decoding, or another type of video decompression, and outputs decoded data, such as image frames. An example of a display screen 132 is a liquid crystal display (LCD) screen or a light-emitting diode (LED) display screen. An example of a communication device of the device is a circuit that applies a communication protocol, such as a wired communication protocol or a wireless communication protocol, to communicate with another device or system. Examples of CPU 135 include a processor, a microprocessor, a microcontroller, an application specific integrated circuit (ASIC), and a programmable logic device (PLD). The camera C1 includes a lens L1 that faces an environment, such as the environment 102 (FIG. 1A-1). For example, the camera C1 is an outward-facing camera.

[0046] The CPU 135 is coupled to other components of the glasses 120. For example, the CPU 135 is coupled to the communication device 134, the camera C1, the network transfer device 126, the video encoder 124, the GPU 130, the video decoder 128, the audio encoder 125, and the microphone M1, and controls other components of the glasses 120. The camera C1 is also coupled to the video encoder 124, which is coupled to the network transfer device 126. The network transfer device 126 is coupled to the computer network 142, the audio encoder 125, and the video decoder 128. The audio encoder 125 is coupled to the microphone M1. The video decoder 128 is coupled to the CPU 130, which is coupled to the display screen 132. Examples of the computer network 142 include a local area network (LAN), a wide area network (WAN), and combinations thereof. For purposes of illustration, the computer network 142 is the Internet or an intranet, or a combination thereof.

[0047] The server system 136 includes a network transfer device 138, a video decoder 140, an audio decoder 144, and one or more servers 1-N (N is an integer greater than 0). An example of an audio decoder is a circuit that decompresses encoded audio frames into audio frames. For illustrative purposes, the audio decoder applies an audio decoding protocol (such as an audio decompression protocol) to decode the encoded audio frames into audio frames. Each of the servers 1-N includes a processor and a memory device. For example, server 1 includes processor 1 and memory device 1, server 2 includes processor 2 and memory device 2, and server N includes processor N and memory device N. The network transfer device 126 is coupled to a computer network 142 and is also coupled to a video decoder 140, which is coupled to one or more of the servers 1-N. The network transfer device 126 is coupled to an audio decoder 144, which is coupled to one or more of the servers 1-N. The operation of the system 118 is described with reference to FIGS. 1A-3.

[0048] In an embodiment, when the glasses 120 are AR glasses, the input controller 122 is separate from the handheld controller used to play the game G1.

[0049] In one embodiment, glasses 120 include multiple display screens instead of display screen 132. Each display screen has a similar structure and function to display screen 132.

[0050] In an embodiment, glasses 120 include one or more additional lenses in addition to lens L1 for capturing an image of an environment, such as environment 102.

[0051] In one embodiment, glasses 120 include one or more memory devices, such as random access memory (RAM) or read only memory (ROM). The one or more memory devices are coupled to CPU 135 or GPU 130 or both CPU 135 and GPU 130. For example, CPU 135 includes a memory controller for accessing data from and writing data to the one or more memory devices.

[0052] In an embodiment, the memory controller is a separate device from the CPU 135 .

[0053] FIG. 1A-3 is an illustration of an embodiment illustrating a training session in which user 1 is requested to turn his / her head to capture a view of an environment, such as environment 102 (FIG. 1A-1). Once authentication information received from user 1 is authenticated, user 1 is provided with access to user account 1 by the authentication server. User 1 may then access the training session or the game program or other application program. During the training session, before or during execution of the game program or other application program, the training program is executed by one or more processors 1-N of the game cloud to generate a message 121. For example, before or during execution of the game program, server system 136 (FIG. 1A-2) executes the training program to generate an image frame having message 121, encodes the image frame to output the encoded image frame having message 121, applies a network transport protocol to the encoded image frame to generate a data packet having message 121, and transmits the data packet to glasses 120 (FIG. 1A-2) via computer network 142. In this example, the network transport device 126 (FIG. 1A-2) of the glasses 120 applies a network transport protocol to obtain encoded image frames from the data packets and provides the encoded image frames to a video decoder 128 (FIG. 1A-2) of the glasses 120. In this example, the video decoder 128 applies a video conversion protocol, such as a video decoding protocol, to decode the encoded image frames, output image frames having the message 121, and provide the image frames to the GPU 130 (FIG. 1A-2). In this example, the GPU 130 applies a rendering program to display the message 121 on a display screen 132 (FIG. 1A-2). In this example, the training program is computer software stored in one or more memory devices 1-N of the server system 136. An example of a network transport protocol includes Transmission Control Protocol (TCP) over Internet Protocol (IP).

[0054] Upon viewing the message 121, the user 1 turns his head to capture a view (e.g., a 360 degree view) of the environment 102. For example, as the user 1 turns his head in the environment 102 to view the environment 102, the camera C1 of the glasses 120 captures images of the objects 108A-108O in the environment 102. Referring to FIG. 1A-1, the camera C1 transmits the images of the objects 108A-108O-P to the video encoder 124 of the glasses 120. The video encoder 124 applies a video encoding protocol to encode the images received from the camera C1 to output encoded image frames, and transmits the encoded image frames to the network forwarding device 126. The network forwarding device 126 applies a network forwarding protocol to generate data packets from the encoded image frames, and transmits the data packets to the network forwarding device 138 of the server system 136 via the computer network 142.

[0055] A network forwarding device 138 of the server system 136 obtains the data packets from the glasses 120 via a computer network 142 and applies a network forwarding protocol to the data packets to extract encoded image frames of the objects 108A-108O. The network forwarding device 138 sends the encoded image frames to a video decoder 140 of the server system 136. The video decoder 140 applies a video decoding protocol to the encoded image frames to output the image frames and provides the image frames to one or more processors 1-N of the game cloud.

[0056] Microphone M1 also generates audio frames based on sounds emanating from user 1 and objects such as speakers in environment 102 (FIG. 1A-1) and background noise emanating from object 108P and children 1 and 2 in external environment 116 (FIG. 1A-1). The audio frames are transmitted from microphone M1 to an audio encoder 125 of glasses 120. Audio encoder 125 applies an audio encoding protocol to output encoded audio frames. The encoded audio frames are provided from audio encoder 125 to network forwarding device 126. Network forwarding device 126 applies a network forwarding protocol to the encoded audio frames to generate data packets and transmits the data packets to server system 136 via computer network 142.

[0057] A network transport device 138 of the server system 136 receives the data packets from the glasses 120 and applies a network transport protocol to the data packets to output encoded audio frames. The network transport device 128 provides the encoded audio frames to an audio decoder 144. The audio decoder 144 applies an audio decoding protocol to the encoded audio frames to output audio frames and provides the audio frames to one or more processors 1-N for storage in one or more memory devices 1-N.

[0058] It should be noted that although FIG. 1A-3 is described with reference to a gaming program, FIG. 1A-3 is equally applicable to other application programs, such as a video conferencing application program.

[0059] In embodiments, FIGS. 1A-3 are described with reference to a user 1, an environment 120, and glasses worn by the user 1, but FIGS. 1A-3 are equally applicable to other users, other environments, and glasses worn by other users.

[0060] FIG. 1A-4 is a diagram of an embodiment of a list 150 of objects displayed by GPU 130 (FIG. 1A-2) to generate an acoustic profile of objects 108A-108P (FIG. 1A-1). List 150 includes the state of objects in an environment system. For example, list 150 includes a first entry where there are no blinds covering windows of environment 102, and a second entry where there are blinds covering windows. As another example, list 150 includes a third entry where a can is open, and a fourth entry where the can is closed. As yet another example, list 150 includes the placement of objects in environment 102. For illustrative purposes, list 150 includes a fourth entry indicating that a can is closer to microphone M1 than objects 108K or 108L, and a fifth entry indicating that object 108L is closer to microphone M1 than object 108K.

[0061] Instead of or in addition to providing message 121, during a training session, before or during execution of the game program, a training program is executed by one or more processors 1-N (FIG. 1A-2) to generate list 150. For example, before or during execution of the game program, server system 136 executes a training program to generate image frames having list 150, encodes the image frames to output encoded image frames having list 150, applies a network transport protocol to the encoded image frames to generate data packets having list 150, and transmits the data packets over a computer network to glasses 120 (FIG. 1A-2). In this example, network transport device 126 (FIG. 1A-2) of glasses 120 applies a network transport protocol to obtain the encoded image frames from the data packets and provides the encoded image frames to video decoder 128 (FIG. 1A-2) of glasses 120. In this example, the video decoder 128 applies a video conversion protocol, such as a video decoding protocol, to decode the encoded image frames, output image frames having a list 150, and provide the image frames to the GPU 130 (FIG. 1A-2). In this example, the GPU 130 applies a rendering program to display the list 150 on the display screen 132 (FIG. 1A-2). In this example, the training program is computer software stored in one or more of the memory devices 1-N of the server system 136.

[0062] Upon viewing the list 150, the user 1 uses the input controller 122 to select one or more checkboxes next to one or more items in the list 150 to identify the objects O108A-O108P and the children 1 and 2. The communication device of the input controller 122 applies a communication protocol to the selection of one or more checkboxes in the list 150 to generate one or more forwarding packets and transmits the forwarding packets to the communication device 134 of the glasses 120. The communication device 134 applies a communication protocol to the forwarding packets to obtain the list 150 from the forwarding packets and transmits the selection of one or more checkboxes in the list 150 to the CPU 135 of the glasses 120. The CPU 135 transmits the selection of one or more checkboxes in the list 150 to the network forwarding device 126 of the glasses 120. The network forwarding device 126 applies a network forwarding protocol to the selection of one or more checkboxes in the list 150 to generate data packets. The network forwarding device 126 transmits the data packets to one or more processors 1-N of the server system 136 via the computer network 142. The network forwarding device 138 receives data packets from the computer network 142, applies a network forwarding protocol to the data packets to obtain a selection of one or more checkboxes in the list 150, and provides the selection to one or more processors 1-N of the server system 136.

[0063] In one embodiment, instead of list 150, a list of blank lines is generated by one or more processors 1-N and transmitted over computer network 142 (FIGS. 1A-3) to glasses 120 (FIGS. 1A-3). User 1 fills in the blank lines using input controller 122 to provide a list of objects 108A-108P and children 1 and 2.

[0064] 1B is a diagram of an embodiment of another environment 152 in which user 1 and user 2 are playing game G1. Environment 152 is a room in a high rise building. For example, the room is on the top floor of a building. Environment 152 includes spectators 1 and 2. Each spectator 1 and 2 is an example of an object of environment 152. Environment 152 further includes objects 154A, 154B, 154C, 154D, 154E, 154F, 154G, 154H, 154I, 154J, 154K, 154L, 154M, 154N, 108O, 154P, and 154Q. Object 154A is a banner that is suspended from the ceiling of environment 152. Object 154B is a display device. An example of a display device is a desktop monitor or a television. Object 154C is a display device and object 154D is another display device. Each of objects 154C and 154D includes a CPU and a memory device located within the object. User 1 plays game G1 displaying a virtual scene on desktop monitor 154D. User 2 plays game G1 displaying a virtual scene on desktop monitor 154C.

[0065] Object 154E is a speaker coupled to object 154C, and object 154F is a speaker coupled to object 154D. Object 154E is behind object 154C, and object 154F is behind object 154D. Object 154G is a can or container, and object 154H is a table on which objects 154C and 154D are placed. As an example, object 154H has a top made of marble and has an uneven surface. In environment 152, the can is closed. Also, object 154I is the floor of environment 152. The floor is not carpeted and is bare. For example, the floor has a tiled surface. Object 154J is a chair on which user 1 sits, object 154K is a mouse coupled to object 154D, and object 154L is a keyboard coupled to object 154D. Object 154M is a mouse that is coupled to object 154C, and object 154N is a keyboard that is coupled to object 154C.

[0066] Object 154P is a pair of glasses worn by user 2. Object 154P includes microphone M2 and camera C2. Object 154Q is a cabinet stand on which object 154B is placed. Object 154R, an airplane, is located outside environment 152 and flies above the building while users 1 and 2 are playing game G1.

[0067] The environment 152 further includes an object 154S and another object 154T. The object 154S is a window without blinds, and the object 154T is a can or container.

[0068] Note that user 1 moves from one location, such as environment 102 (FIG. 1A-1), to another location, such as environment 152. User 1 accesses game G1 from the game cloud via computer network 142 (FIG. 1A-2) and user account 1 to play game G1, which has a virtual scene depicted on the display screen of object 154D. For example, user 1 selects one or more buttons on one or more of objects 154L and 154K to provide authentication information, such as a username and password. Objects 154L and 154K transmit the authentication information to object 154D, which forwards the authentication information to the game cloud via computer network 142. An authentication server of the game cloud determines whether the authentication information is authentic, and if so, provides access to user account 1 and a game program executing on the game cloud. When the game program is executed by one or more of processors 1-N of the game cloud, image frames of game G1 are generated and encoded, and the encoded image frames are output. The encoded image frames are transmitted to object 154D. Object 154D decodes the encoded image frames and provides the image frames to the display screen of object 154D in the virtual scene displayed on object 154D, enabling user 1 to play game G1.

[0069] During the play of the game G1, the sound of the game G1 is output from the object 154F. For example, when a virtual object in the virtual scene displayed on the object 154D jumps in the virtual scene and lands on the virtual ground, a landing sound is output via the object 154F. As an example, the virtual object in the virtual scene displayed on the object 154D is controlled by the user 1 via the objects 154L and 154K. During the play of the game G1, an airplane flying on the environment 152 makes a sound such as a sonic boom. During the play of the game G1, the spectators 1 and 2 make sounds by talking to each other, arguing with each other, playing with each other, etc. During the play of the game G1, the user 1 opens a container placed on the object 154H, and when the container is opened, a sound is made. Furthermore, during the play of the game G1, the user 1 utters words, which are examples of sounds.

[0070] Both user 1 and user 2 are in the same environment 152 and therefore in the same location. Similarly, user 2 accesses game G1 from the game cloud via computer network 142 and user account 2 to play game G1, which has a virtual scene depicted on the display screen of object 154C. For example, user 2 selects one or more buttons on objects 154N and 154M to provide authentication information, such as a username and password. Objects 154N and 154M transmit the authentication information to object 154C, which forwards the authentication information to the game cloud via computer network 142. An authentication server of the game cloud determines whether the authentication information is authentic and, if so, provides access to user account 2 and a game program executing on the game cloud. When the game program is executed by one or more of processors 1-N of the game cloud, image frames of game G1 are generated and encoded, and the encoded image frames are output. The encoded image frames are transmitted to object 154C. Object 154C decodes the encoded image frames and provides the image frames to the display screen of object 154C in a virtual scene displayed on object 154C, enabling user 2 to play game G1.

[0071] During the play of the game G1, the sound of the game G1 is output from the object 154E. For example, when a virtual object in the virtual scene displayed on the object 154C flies in the virtual scene, the flying sound is output via the object 154E. As an example, the virtual object in the virtual scene displayed on the object 154C is controlled by the user 2 via the objects 154N and 154M. During the play of the game G1, the user 2 opens the object 154T placed on the object 154H, and when the object 154T is opened, a sound is made. During the play of the game G1, the user 2 utters a word, which is an example of a sound.

[0072] Each of the microphones M1 and M2 captures audio data, such as audio frames, generated from sounds associated with them, such as sounds emanating from or reflected from one or more of the objects located in the environment 152. For example, the microphones M1 and M2 capture audio data generated based on sounds emanating from the object 154F and received from the object 154F via a path 156A. In this example, the sound path 156A is a direct path from the object 154F to the microphones M1 and M2 and does not enter any other surface between the object 154F and the microphones M1 and M2. As another example, the microphones M1 and M2 capture sounds emanating from the object 154F and received from the object 154F via a path 156B. In this example, the sound path 154F is an indirect path from the object 154F to the microphones M1 and M2. To illustrate, sound emanating from object 154F impinges on one or more other objects in environment 152 (eg, object 154D) and is reflected from the one or more other objects towards microphones M1 and M2.

[0073] As yet another example, each microphone M1 and M2 captures audio data generated based on sound emanating from and received from object 154E via path 156C. In this example, sound path 156C is a direct path from object 154E to microphones M1 and M2 and does not strike any other surfaces between object 154E and microphones M1 and M2. As yet another example, microphones M1 and M2 capture sound emanating from and received from object 154E via path 156D. In this example, sound path 156D is an indirect path from object 154E to microphones M1 and M2. To illustrate, sound emanating from object 154E strikes one or more other objects (e.g., object 154C) in environment 152 and is reflected from the one or more other objects toward microphones M1 and M2.

[0074] Each microphone M1 and M2 also captures audio data generated based on sounds, such as background noise, emanating from an object 154R located outside the environment 152. The environment outside the environment 152 may be referred to herein as the external environment 158. As an example, microphones M1 and M2 capture sounds emanating from object 154R and received through the ceiling of the environment 152.

[0075] Microphones M1 and M2 capture sounds associated with objects 154A-154N, 108O, 154P, 154Q, 154S, 154T, and one or more of spectators 1 and 2 in environment 152, such as sounds emanating from or reflected by them, and background noise emanating from object 154R in external environment 158 ​​to generate audio frames, such as audio data. Encoded audio frames based on the audio frames are generated and provided to one or more of processors 1-N of the game cloud via computer network 142 for processing and training of AI models.

[0076] In one embodiment, user 1 plays a different game than game G1 and user 2 plays a different game than game G1.

[0077] In an embodiment, the external environment 158 ​​includes any other number of the objects, such as two or three.

[0078] In one embodiment, instead of or in addition to the sound output from object 154E, sound is output from object 154C and detected by microphones M1 or M2 or a combination thereof, and audio data is captured.

[0079] In an embodiment, instead of or in addition to the sound output from object 154F, sound is output from object 154D and detected by microphones M1 or M2 or a combination thereof, and audio data is captured.

[0080] In an embodiment, instead of or in addition to microphone M2, one or more additional microphones, such as stand-alone microphones, are present to capture sounds emanating from objects in environment 152 and from objects located in external environment 158. For example, a display device located on a table in environment 152 includes an additional microphone.

[0081] In an embodiment, the virtual scene displayed on object 154D is instead displayed on the display screen of glasses worn by user 1. Similarly, the virtual scene displayed on object 154C is instead displayed on the display screen of glasses worn by user 2.

[0082] 2 is a diagram of an embodiment of a system 200 illustrating the processing of input data sets 1, 2, and 3 and audio data sets 1, 2, and 3. The system 200 includes a client device 1, a client device 2, and a client device 3. The system 200 further includes a processor system 202. Examples of the client device 1 operated by the user 1 include glasses worn by the user 1, an object 108C (FIG. 1A-1), an object 108N, a game console 112 (FIG. 1A-1), an object 108L (FIG. 1A-1), an object 108K (FIG. 1A-1), an object 154D (FIG. 1B), an object 154L, an object 154K, an object 154F, and a combination of two or more thereof. Examples of the client device 2 operated by the user 2 include glasses worn by the user 2, an object 154C (FIG. 1B), an object 154M, an object 154N, an object 154E, a game console, a handheld controller, and a combination of two or more thereof. Note that glasses 120 (FIGS. 1A-2) are an example of glasses worn by user 2, except that camera C1 is replaced with camera C2 and microphone M1 is replaced with microphone M2. An example of client device 3 is provided below with reference to FIG. 5A. Client device 3 is operated by user 3.

[0083] The processor system 202 is coupled to the client devices 1, 2, and 3. For example, the processor system 202 is coupled to the client devices 1-3 via a computer network 142 (FIG. 1B). For purposes of illustration, the processor system 202 includes one or more processors 1-N of a server system 136 (FIGS. 1A-2) that is coupled to the client devices 1-3 via the computer network 142.

[0084] An example of audio dataset 1 includes audio data captured by microphone M1 based on sounds associated with environment 102 (FIG. 1A-1) and background noise received from external environment 116 (FIG. 1A-1). An example of audio dataset 2 includes audio data captured by microphone M1 based on sounds associated with environment 152 (FIG. 1B) and background noise received from external environment 158 ​​(FIG. 1B). An example of audio dataset 3 includes audio data captured by microphone M2 based on sounds associated with environment 152 and background noise received from external environment 158.

[0085] An example of input dataset 1 includes one or more images of objects of environment 102 (FIG. 1A-1) captured by camera C1, an identification of the objects of environment 102, and an identification of objects in external environment 116 (FIG. 1A-1). For illustrative purposes, examples of the identification of objects of environment 102 and objects in external environment 116 are received in list 150 (FIG. 1A-4).

[0086] An example of input dataset 2 includes one or more images of objects of environment 152 (FIG. 1B) captured by camera C1, identifications of objects of environment 152 received via user account 1, and identifications of objects in external environment 158 ​​(FIG. 1B) received via user account 1. For purposes of illustration, the identifications of objects of environment 152 and objects in external environment 158 ​​are received in the form of a list via user account 1 from user 1 in the same manner as the identifications of objects of environment 102 and objects in external environment 116 are received via list 150. In this illustration, user 1 selects one or more objects in the list to provide the identifications of objects of environment 152 and objects in external environment 158. The list is generated in the same manner as list 150 is generated and is displayed on objects 154D (FIG. 1B) manipulated by user 1.

[0087] An example of input data set 3 includes one or more images of objects of environment 152 (FIG. 1B) captured by camera C2, identifications of objects of environment 152 received via user account 2, and identifications of objects in external environment 158 ​​(FIG. 1B) received via user account 2. For purposes of illustration, the identifications of objects of environment 152 and objects in external environment 158 ​​are received in the form of a list via user account 2 from user 2 in the same manner as the identifications of objects of environment 102 and objects in external environment 116 (FIGS. 1A-1) are received via list 150. In this illustration, user 2 selects one or more objects in the list to provide identifications of objects of environment 152 and objects in external environment 158. The list is generated in the same manner as list 150 is generated and is displayed on objects 154C (FIG. 1B) manipulated by user 2.

[0088] The processor system 202 includes a game engine and an inference training engine. An example of an engine includes hardware, such as one or more controllers. In this example, each controller includes one or more processors, such as processors 1-N, or one or more processors of game console 112 (FIGS. 1A-1), or a combination thereof. As another example, an engine is software, such as a computer software program. For purposes of illustration, a game engine is a game program. Another example of an engine is a combination of hardware and software.

[0089] The inference training engine includes an AI processor and a memory device 204 that is an example of one of the memory devices 1-N. The AI ​​processor is coupled to the memory device 204 and is an example of one of the processors 1-N. Input data sets 1-3 and audio data sets 1-3 are stored in the memory device 204. For example, the AI ​​processor receives the input data sets 1-3 and the audio data sets 1-3 from the client devices 1 and 2 via the computer network 142 and stores the input data sets 1-3 and the audio data sets 1-3 in the memory device 204. A game engine is coupled to the inference training engine.

[0090] The AI ​​processor includes a feature extractor, a classifier, and an AI model. For example, the AI ​​processor includes a first integrated circuit that applies a function of the feature extractor, a second integrated circuit that applies a function of the classifier, and a third integrated circuit that applies a function of the AI ​​model. As another example, the AI ​​processor executes a first computer program that applies a function of the feature extractor, a second computer program that applies a function of the classifier, and a third computer program that applies a function of the AI ​​model. The feature extractor is coupled to the classifier, and the classifier is coupled to the AI ​​model.

[0091] The feature extractor extracts, for example determines, parameters such as one or more amplitudes and one or more frequencies and one or more sensing directions from the audio datasets 1-3. For example, the feature extractor determines the magnitude or peak-to-peak amplitude or zero-to-peak amplitude and frequency of the audio datasets 1-3. For illustration, the feature extractor determines the absolute maximum power of the audio dataset 1 or the absolute minimum power of the audio dataset 1 to determine the magnitude of the audio dataset 1. In this description, the absolute power is the magnitude within the entire period during which the audio dataset 1 is generated. In another description, the feature extractor determines a local maximum of the magnitude of the audio dataset 1 and a local minimum of the magnitude of the audio dataset 1. In this description, the local magnitude is the magnitude within a predetermined period, which is shorter than the entire period during which the audio dataset 1 is generated. In this description, multiple local maxima of the magnitude and multiple local minima of the magnitude are determined from the audio dataset 1, and the feature extractor applies a best fit or an average or a median to the local maxima of the magnitude and the local minima to determine the maximum of the magnitude and the minimum of the magnitude.

[0092] Alternatively, the feature extractor determines a first time when the audio dataset 1 reaches a predetermined magnitude and a second time when the audio dataset 1 reaches the same predetermined magnitude, and calculates the difference between the first and second times to determine a time interval. The feature extractor inverts the time interval to determine an absolute frequency of the audio dataset 1. In this description, the absolute frequency is a frequency within an entire period during which the audio dataset 1 is generated. As yet another description, the feature extractor determines a local frequency of the audio dataset 1. In this description, the local frequency is a frequency within a predetermined period, which is shorter than the entire period during which the audio dataset 1 is generated. In this description, a plurality of local frequencies are determined from the audio dataset 1, and the feature extractor applies a best fit or average or median to the local frequencies to determine the frequency. In this description, each local frequency is determined in the same manner as the absolute frequency is determined, except that the local frequency is determined for each predetermined period.

[0093] As another explanation, the feature extractor determines the direction in which the audio data set 1 is sensed. In this explanation, the microphone M1 includes an array (such as a linear array) of transducers arranged in a direction. In this explanation, the array includes a proximal transducer and a distal transducer. In this explanation, if the proximal transducer outputs a first portion of the audio data 1 and the distal transducer outputs a second portion of the audio data 1, and the first portion has an amplitude greater than the second amplitude, the feature extractor determines that the object 108M (FIG. 1A-1) in the environment 102 (FIG. 1A-1) is closer to the proximal transducer than the distal transducer. In this explanation, the feature extractor further determines that the object 108M is in a direction facing the proximal transducer. Meanwhile, as another explanation, if the second amplitude is greater than the first amplitude, the feature extractor determines that the object 108M is closer to the distal transducer than the proximal transducer. In this description, the feature extractor further determines that the object 108M is in a direction pointing towards the distal transducer. The direction in which the audio data set is sensed may be referred to herein as the sensing direction.

[0094] The classifier classifies parameters obtained from sounds related to the environments 102, 116, 152, and 158 based on the input datasets 1-3. For example, the classifier determines a combination of objects in an environmental system, such as the environment 102 (FIG. 1A-1) or the external environment 116 (FIG. 1A-1) or a combination of the environments 102 and 116, and establishes a correlation, such as a one-to-one correspondence or a unique relationship, with the parameters determined from the audio dataset 1. In this example, the classifier determines the object placement, or the object state, or a combination of two or more of the objects, object placement, and object state in the environment. For illustrative purposes, the classifier receives in the input dataset 1 the identities of the objects 108A-108O in the environment 102, and the identities of the objects 108P, child 1, and child 2 via the list 150 and the user account 1. In this illustration, the classifier receives in the input dataset 1 the states of the objects 108A-108O (FIG. 1A-1). Further, in this description, the classifier determines from the sensing direction that object 108M is located closer to microphone M1 than object 108L and compared to object 108K (FIG. 1A-1). Also, in this description, the classifier determines from the sensing direction that children 1 and 2 are located farther away from microphone M1 than objects 108M, 108L, and 108K, and determines the location of children 1 and 2 and objects 108M, 108L, and 108K. As another description, the classifier determines from the sensing direction or amplitude or frequency or a combination thereof that object 108P or child 1 or child 2 is located outside environment 102.

[0095] As another explanation, the classifier receives in the input data set 2 the identities of the objects 154A-154N, 108O, 154P, and 154Q in the environment 152, and the identity of the object 154R via a list, such as the list 150, and the user account 1. In this explanation, the classifier receives in the input data set 2 the states of the objects 154A-154N, 108O, 154P, and 154Q (FIG. 1B). In addition, in this explanation, the classifier determines from the sensing direction that the object 154D is located closer to the microphone M1 than the object 154F (FIG. 1B). In addition, in this explanation, the classifier determines from the sensing direction that the object 154R is flying above the environment 152, and determines the location of the object 154R relative to the environment 152. As another explanation, the classifier determines from the sensing direction or the amplitude or the frequency or a combination thereof that the object 154R is located outside the environment 152.

[0096] As yet another illustration, the classifier receives in the input dataset 3 the identities of objects 154A-154N, 108O, 154P, and 154Q in the environment 152, as well as the identity of object 154R via a list, such as list 150, and user account 2. In this illustration, the classifier receives in the input dataset 3 the states of objects 154A-154N, 108O, 154P, and 154Q (FIG. 1B). Further, in this illustration, the classifier determines from the sensing direction that object 154C is located closer to microphone M2 than object 154E (FIG. 1B).

[0097] The AI ​​model is trained based on parameters associated with the environments 102, 116, 152, and 158 (FIGS. 1A-1 and 1B) and correlations between the input datasets 1-3. For example, referring to FIG. 3, the AI ​​model is provided by the classifier with an indication of correlations 302 (e.g., links or one-to-one correspondences) between a set including amplitude 1, frequency 1, and sensing direction 1 and a set including a first type of environment, a first combination of objects in the first type of environment, a first arrangement of objects, and a first state of objects. In this example, the AI ​​model receives amplitude 1, frequency 1, and sensing direction 1 from the classifier. In this example, amplitude 1, frequency 1, and sensing direction 1 are determined by analyzing audio dataset 1 captured by microphone M1. In this example, the one or more amplitudes determined from audio dataset 1 are referred to herein as amplitude 1. Also, in this example, the one or more frequencies determined from audio dataset 1 are referred to herein as frequency 1, and the direction in which audio dataset 1 is sensed is referred to herein as sensing direction 1. For purposes of illustration, amplitude 1, frequency 1, and sensing direction 1 are examples of parameters of audio dataset 1. Also, in this example, the classifier provides the AI ​​model, via user account 1, with a first type of environment 102, a first combination of objects in the environment 102 and the external environment 116, a first state of the objects, and a first arrangement of the objects. For illustrative purposes, the first type of environment includes whether the environment 102 is an indoor environment, such as a room or an open space in a building, or an outdoor environment, such as a park, a concert, or a lake. As another illustration, the first combination of objects includes the identity of the objects 108A-108O, such as a can or a display device or a speaker or a window. In this illustration, the first arrangement of the objects includes the can being closer to the microphone M1 than the speaker, and the first state includes the can being open or closed. As another illustration, the first combination of objects includes the identity of the objects 108A-108O, such as a window with blinds or a carpet or a speaker.Also, in this example, the combination of environment 102 and external environment 116 is referred to as a first environmental system.

[0098] As another example, referring to FIG. 3, an AI model is provided with an indication of a correlation 304 (e.g., a link or one-to-one correspondence) between a set including amplitude 2, frequency 2, and sensing direction 2 and a set including a second type of environment, a second combination of objects in the second type of environment, a second arrangement of objects, and a second state of objects. In this example, the AI ​​model receives amplitude 2, frequency 2, and sensing direction 2 from a classifier. In this example, amplitude 2, frequency 2, and sensing direction 2 are determined by analyzing an audio dataset 2 captured by microphone M1. Also in this example, one or more amplitudes determined from audio dataset 2 are referred to herein as amplitude 2, one or more frequencies determined from audio dataset 2 are referred to herein as frequency 2, and a direction in which audio dataset 2 is sensed is referred to herein as sensing direction 2. For purposes of illustration, amplitude 2, frequency 2, and sensing direction 2 are examples of parameters of audio dataset 2. Also in this example, the classifier provides the AI ​​model with a second type of environment 152, a second combination of objects in environment 152 and external environment 158, a second state of objects, and a second arrangement of objects. In this example, the second type, the second combination, the second state, and the second arrangement are received via user account 1. For illustrative purposes, the second type of environment includes whether the environment 152 is an indoor environment, such as a room or an open space in a building, or an outdoor environment, such as a park, a concert, or a lake. As another illustration, the second combination of objects includes the identity of the objects 154A-154N, 108O, 154P-154T, such as a can, a display device, or a speaker, or a floor or a ceiling. In this illustration, the second arrangement of objects includes the object 154G being closer to the microphone M1 than the object 154F, and the second state includes the object 154G being open or closed. As another illustration, the second arrangement includes the object 154B being farther away from the microphone M1 than the objects 154F or 154D (FIG. 1B).In this illustration, object 154B is a display device emitting a sound from a video game or another application, and in this example, the combination of environment 152 and external environment 158 ​​is referred to as a second environment system.

[0099] As yet another example, referring to FIG. 3, the AI ​​model is provided by the classifier with an indication of a correlation 306 (e.g., a link or one-to-one correspondence) between a set including amplitude 3, frequency 3, and sensing direction 3 and a set including a third type of environment, a third combination of objects in the third type of environment, a third arrangement of objects, and a third state of objects. In this example, the AI ​​model receives amplitude 3, frequency 3, and sensing direction 3 from the classifier. In this example, amplitude 3, frequency 3, and sensing direction 3 are determined by analyzing an audio dataset 3 captured by microphone M2. Also, in this example, one or more amplitudes determined from the audio dataset 3 are referred to herein as amplitude 3, one or more frequencies determined from the audio dataset 3 are referred to herein as frequency 3, and a direction in which the audio dataset 3 is sensed is referred to herein as sensing direction 3. For purposes of illustration, amplitude 3, frequency 3, and sensing direction 3 are examples of parameters of the audio dataset 3. Also, in this example, the classifier provides the AI ​​model with a third type of environment 152, a third combination of objects in the environment 152 and the external environment 158, a third state of objects, and a third arrangement of objects. In this example, the third type, third combination, third state, and third arrangement are received via user account 2. For illustrative purposes, the third type of environment includes whether the environment 152 is an indoor environment, such as a room or an open space in a building, or an outdoor environment, such as a park, a concert, or a lake. As another illustration, the third combination of objects includes the identity of the objects 154A-154N, 108O, 154P-154T, such as a can, a display device, or a speaker, or a floor or a ceiling. In this illustration, the third arrangement of objects includes the object 154T being closer to the microphone M2 than the objects 154E or 154C, and the third state includes the object 154T being open or closed. Also, in this example, the combination of the environment 152 and the external environment 158 ​​is referred to as a third environment system.

[0100] In an embodiment, instead of or in addition to receiving identities, such as list 150 (FIGS. 1A-4), of objects in the environmental system, image data, such as image frames, of the environmental system are received from cameras C1 and C2. In this embodiment, the feature extractor identifies objects in the environmental system from the image data. For example, the feature extractor identifies objects 108A-108P from image frames of input dataset 1. In this example, the feature extractor determines that an outline of object 108A matches a pre-stored outline of a chair and determines that object 108A is a chair. In this example, the pre-stored outline is stored in one or more of memory devices 1-N. In this example, the feature extractor compares the size and shape of object 108O to a pre-stored size and a pre-stored shape of a pre-stored pair of glasses and determines that object 108O is a pair of glasses 120. In this example, the pre-stored size, the pre-stored shape, and the pre-stored identification of the glasses are stored in memory device 204. In this example, the pre-stored identification of the glasses includes alphanumeric characters. In this example, the feature extractor provides the AI ​​model with the identities of objects in the environmental system.

[0101] In one embodiment, the feature extractor identifies the location and graphical parameters, such as color, brightness, tone, and texture, of objects in the environmental system from image data received from cameras C1 and C2. For example, the feature extractor determines the positions of objects 108A-108P relative to one another and the graphical parameters of objects 108A-108P. The location and graphical parameters are stored in one or more of memory devices 1-N of server system 136.

[0102] In one embodiment, the processor is replaced by an application specific integrated circuit (ASIC) or a programmable logic device (PLD) or a central processing unit (CPU) or a combination of a CPU and a GPU.

[0103] In an embodiment, instead of a game engine, the engine of another application, such as a video conferencing application, is used.

[0104] In one embodiment, the classifier receives from the feature extractor the identities of the objects 108A-108O in the environment 102. The identities of the objects 108A-108O are determined by the feature extractor from images captured by the camera C1 and the user account 1. The images captured by the camera C1 are part of the input dataset 1. Similarly, the classifier receives from the feature extractor the identities of the objects 154A-154N, 108O, 154P-154Q, 154S, and 154T in the environment 152. The identities of the objects 154A-154N, 108O, 154P-154Q, 154S, and 154T are determined by the feature extractor from images captured by the camera C1 and the user account 1. The images captured by the camera C1 are part of the input dataset 2. The classifier also receives from the feature extractor the identities of the objects 154A-154N, 108O, 154P-154Q, 154S, and 154T in the environment 152. The identities of the objects 154A-154N, 108O, 154P-154Q, 154S, and 154T are determined by the feature extractor from images captured by camera C2 and user account 2. The images captured by camera C2 are part of the input dataset 3.

[0105] 3 is a diagram of an embodiment of a system 300 to illustrate correlations 302, 304, and 306. System 300 includes audio data sets 1-3.

[0106] 4A is a diagram of an embodiment of an object state 400. For example, the state of a floor is whether the floor is made of carpet, tile, or concrete. As another example, the state of a window blind is whether the window blind is open or closed. As yet another example, the state of a container is whether the container is open or closed.

[0107] FIG. 4B is a diagram of an embodiment of an object type 402. For example, the object type provides the identity of the object. For illustrative purposes, a vehicle is a type of object, a monitor is a type of object, a display device is a type of object, an external speaker is a type of object, a human being such as a child or an audience or a user is a type of object, a house is a type of object, a building is a type of object, a carpet is a type of object, a bare floor is a type of object, a wall is a type of object, and a window is a type of object. An example of each of the objects 108K (FIG. 1A-1), 108L (FIG. 1A-1), 154E (FIG. 1B), and 154F (FIG. 1B) is an external speaker. For illustrative purposes, the external speaker is not integrated into another device such as a display device, but is located outside the display device.

[0108] 5A is a diagram of an embodiment of an environment 500 to illustrate the use of an AI model to identify objects in the environment 500 and to determine the state of the objects in the environment 500 and the placement of the objects in the environment 500. The environment 500 is a room in a house 504. The environment 500 includes an example object, a user 3. The environment 500 further includes an object 502A, an object 502B, an object 502C, an object 502D, an object 502E, an object 502F, an object 502G, an object 502H, an object 502I, an object 502J, an object 502K, an object 502L, an object 502M, an object 502N, an object 502O, an object 502P, and an object 502Q. An example of a client device 3 (FIG. 2) includes glasses worn by the user 3, an object 502I, an object 502E, an object 502D, an object 502A, an object 502G, an input controller coupled to the glasses, and a combination of two or more thereof.

[0109] Object 502A is a display device including a computer. Object 502B is also a display device such as a monitor. Object 502C is a table on which objects 502A, 502B, 502D, 502E, 502F, and 502G are placed. Object 502D is a mouse coupled to object 502A, and object 502E is a keyboard coupled to object 502A. Object 502F is a stapler, and object 502G is a speaker coupled to object 502A. Object 502H is a robotic arm. Object 502I is a handheld controller used by user 3 to play game G1. Object 502J is a chair on which user 3 is sitting. Object 502K is a shelf on which objects 502L, 502M, and 502N are supported. Object 502L is a box, and object 502M is another box. Object 502N is a stack of containers. Object 502O is a pair of glasses worn by user 3. As an example, object 502O has the same structure and function as glasses 120 (FIG. 1A-2). For example, object 502O includes a camera C3 and a microphone M3. In this example, glasses 120, which is an example of object 502O, includes a camera C3 and a microphone M3 instead of camera C1 and microphone M1. Object 502P is a window without blinds. Object 502Q is an open soda can.

[0110] A user 3 accesses the game G1 from the game cloud via the computer network 142 (FIGS. 1A-2) and plays the game G1, which has a virtual scene represented on the display screen of the object 502A. For example, the user 3 selects one or more buttons on one or more of the objects 502I, 502D, and 502E to provide authentication information, such as a username and password. The object 502I or the objects 502D and 502E transmit the authentication information to the object 502A, which forwards the authentication information to the game cloud via the computer network 142. An authentication server of the game cloud determines whether the authentication information is authentic, and if so, provides access to the user account 3 and the game program executing on the game cloud. When the game program is executed by one or more of the processors 1-N of the game cloud, image frames of the game G1 are generated and encoded, and the encoded image frames are output. The encoded image frames are transmitted to the object 502A. Object 502A decodes the encoded image frames and provides them to a display screen of object 502A, which applies a rendering program to the image frames to display images of the virtual scene and further enables user 3 to play game G1.

[0111] During the play of the game G3, the sound of the game G3 is output from the object 502G. For example, if the virtual object in the virtual scene displayed on the object 502A is a car accelerating on the virtual ground in the virtual scene, the sound of the car running is output through the object 502G. As an example, the virtual object in the virtual scene displayed on the object 502A is controlled by the user 3 through the object 502I or the objects 502D and 502E. Also, during the play of the game G1, the object 502R, which is a car near the house 504, makes sounds such as engine noise. The car 502R is part of the external environment 506 located outside the house 504.

[0112] Microphone M3 captures audio data, such as audio frames, generated from sounds associated with, such as sounds reflected from, one or more of the objects located in environment 500. For example, microphone M3 captures audio data generated based on sounds emanating from object 502G and received from object 502G via path 508A. In this example, sound path 508A is a direct path from object 502G to microphone M3 and does not impinge on any other objects between object 502G and microphone M3. As another example, microphone M3 captures sounds emanating from object 502G and received from object 502G via path 508B. In this example, sound path 508B is an indirect path from object 502G to microphone M3. For purposes of illustration, sounds emanating from object 502G impinge on one or more other objects (e.g., objects 502A and 502C) in environment 500 and are reflected from the one or more other objects toward microphone M3.

[0113] Microphone M3 also captures audio data generated based on sounds, such as background noise, emanating from object 502R located outside environment 500. As an example, microphone M3 captures sounds emanating from object 502R and received through a wall or window of environment 500.

[0114] Microphone M3 captures sounds associated with one or more objects in environment 500, such as objects 502A-502Q, such as sounds emanating from or reflected from those objects, and background noise emanating from object 502R in external environment 506, to generate audio frames, such as audio dataset N, where N is a positive integer. Encoded audio frames are generated based on the audio frames and provided to one or more of processors 1-N of the game cloud via computer network 142 for processing and training of AI models. Note, by way of example, that there is no capture of images of environment 500 by camera C3.

[0115] The feature extractor extracts, e.g., determines, from the audio dataset N, parameters such as one or more amplitudes and one or more frequencies and one or more sensing directions in the same way as the parameters are determined from the audio datasets 1, 2, or 3. For example, the feature extractor determines the magnitude, or the peak-to-peak amplitude, or the zero-to-peak amplitude, and the frequency of the audio dataset N. For illustration, the feature extractor determines the absolute maximum power of the audio dataset N, or the absolute minimum power of the audio dataset N, to determine the magnitude of the audio dataset N. As another illustration, the feature extractor determines a local maximum of the magnitude and a local minimum of the magnitude of the audio dataset N. As another illustration, the feature extractor determines multiple local maxima of the magnitude and multiple local minima of the magnitude, and a best fit or an average or a median is applied by the feature extractor to the local maxima of the magnitude and the local minima to determine the maximum of the magnitude and the minimum of the magnitude. Alternatively, the feature extractor determines a first time when the audio dataset N reaches a predetermined magnitude and a second time when the audio dataset N reaches the same predetermined magnitude, and calculates the difference between the first and second times to determine the time interval. The feature extractor inverts the time interval to determine the absolute frequency of the audio dataset N. Alternatively, the feature extractor determines a local frequency of the audio dataset N. In this description, the local frequency is a frequency within a predetermined period of time, the predetermined period of time being less than the entire period during which the audio dataset N is generated. In this description, a number of local frequencies are determined from the audio dataset N, and the feature extractor applies a best fit or average or median to the local frequencies to determine the frequency.

[0116] As another explanation, the feature extractor determines the direction in which the audio data set N is sensed. In this explanation, the microphone M3 includes an array of transducers arranged in a linear direction (e.g., a linear array). The array includes a proximal transducer and a distal transducer. In this explanation, if the proximal transducer outputs a first portion of the audio data N and the distal transducer outputs a second portion of the audio data N, and the first portion has an amplitude greater than the second amplitude, the feature extractor determines that the object 502G (FIG. 5A) in the environment 500 (FIG. 5A) is closer to the proximal transducer than the distal transducer. In this explanation, the feature extractor further determines that the object 502G is in a direction facing the proximal transducer.

[0117] The one or more amplitudes determined from the audio data set N are referred to herein as amplitudes N. Also, the one or more frequencies determined from the audio data set N are referred to herein as frequencies N, and the direction in which the audio data set N is perceived is referred to herein as perceived direction N.

[0118] In one embodiment, user 3 plays a different game than game G1.

[0119] In an embodiment, the external environment 506 includes any other number of objects, such as two or three.

[0120] In an embodiment, object 502O excludes camera C3.

[0121] In one embodiment, parameters may be referred to herein as features.

[0122] In an embodiment, instead of or in addition to microphone M3, one or more additional microphones, such as stand-alone microphones, are present to capture sounds emanating from objects in environment 500 and from objects located in external environment 506. For example, a display device located on a table in environment 500 includes an additional microphone.

[0123] 5B is a diagram of an embodiment of model output 550. Model output 550 includes A% probability that environment 500 includes combination N of objects, B% probability that objects have configuration N, C% probability that objects have state N, and D% probability that environment 500 is of type N, where A, B, C, and D are positive real numbers. As an example, A, B, C, and D are equal. As another example, one of A, B, C, and D is not equal to at least one of the remaining A, B, C, and D.

[0124] When the AI ​​model is provided with the amplitude N, frequency N, and sensing direction N from the feature extractor, the AI ​​model provides a model output 500. For example, upon determining that the amplitude N is within a predetermined range from amplitude 1 and outside a predetermined range from amplitude 2, the AI ​​model indicates that there is a greater than 50% probability that the audio data set N was received from a room in a house, as opposed to a room in a building. In this example, a house is an example of environment type N. For purposes of illustration, a house is an indoor type environment. As another example, upon determining that the frequency N is within a predetermined range from frequency 2 and outside a predetermined range from frequency 1, the AI ​​model indicates that there is a greater than 50% probability that the audio data set N was received from a room in a building, as opposed to a room in a house. In this example, a building is an example of environment type N. For purposes of illustration, a building is an indoor type environment. As another example, a combination of two or more of the amplitude N, frequency N, and sensing direction N is used to determine an environment 500 of type N.

[0125] As yet another example, upon determining that sensing direction N is within a predetermined range from sensing direction 1 and outside a predetermined range from sensing direction 2, the AI ​​model dictates that there is a greater than 50% probability that audio data set N is output from a speaker behind the display device compared to a speaker in front of the display device of environment 500. Example placements of speakers and display devices N are provided depending on whether the speakers are behind or in front of the display device. As another example, a combination of two or more of amplitude N, frequency N, and sensing direction N is used to determine a placement N of an object in environment 500.

[0126] As another example, upon determining that the amplitude N is within a predetermined range from amplitude 2 and outside a predetermined range from amplitude 1, the AI ​​model may indicate that there is a greater than 50% probability that a window without blinds exists within environment 500. In this example, the window without blinds is an example of an object combination N or object within environment 500. As another example, a combination of two or more of the amplitude N, frequency N, and sensing direction N is used to determine the object combination N within environment 500.

[0127] As yet another example, upon determining that amplitude N is within a predetermined range from amplitude 1 and outside a predetermined range from amplitude 2, the AI ​​model may indicate that there is a greater than 50% probability that a predetermined number of speakers are present in environment 500. In this example, the predetermined number of speakers is an example combination N of objects in environment 500.

[0128] As yet another example, upon determining that the amplitude N is within a predetermined range from amplitude 1 and outside a predetermined range from amplitude 2, the AI ​​model indicates that there is a greater than 50% probability that a window in environment 500 has open blinds or that the window does not have blinds. In this example, open or closed blinds are examples of state N of an object in environment 500. As another example, a combination of two or more of the amplitude N, frequency N, and sensing direction N is used to determine the state N of an object in environment 500.

[0129] As yet another example, upon determining that frequency N is within a predetermined range from frequency 1 and outside a predetermined range from frequency 2, the AI ​​model may indicate that there is a greater than 50% probability that a soda can in environment 500 is open. In this example, a soda can being open in environment 500 is an example of state N of an object in environment 500.

[0130] In an embodiment, the server system 136 (FIG. 7) receives input via the user account 3 and either the object 502I or the objects 502D and 502E. When the user 3 makes one or more selections on the object 502I or the objects 502D and 502E or the input controller to indicate that the user 3 wants to identify an object in the environment 500, the input is generated by an input controller coupled to the object 502I or the objects 502D and 502E or the object 502O. In this embodiment, the user 3 is wearing an HMD. The input is transmitted from the object 502I or the objects 502D and 502E or the input controller to the server system 136 via the object 502O or the object 502A and the computer network 142. Upon receiving an input requesting the identities of objects in the environment 500, the server system 136 applies the AI ​​model to the object 502O or object 502A via the computer network 142 to obtain the probabilities A%, B%, C%, and D% for the combination N, arrangement N, state N, and type N of the environment 500, as well as the probabilities for the combination N, arrangement N, state N, and type N. The object 502O or object 502A displays these probabilities, as well as the combination N, arrangement N, state N, and type N of the objects in the environment 500 on the display screen of the object 502O or object 502A.

[0131] In one embodiment, the camera C3 captures image data of the environment 500 and transmits the image data to the server system 136 (FIG. 7) via the computer network 142. The image data is used to verify the combination N, arrangement N, state N, and type N of the environment 500 determined by the AI ​​model. One or more processors 1-N of the server system 136 verify the identity of the objects in the environment 500 determined by the AI ​​model based on the image data. For example, the processor N determines that a match exists between a first identity of the object 502A being a display device and a second identity of the object 502A being a display device. In this example, the first identity is determined by the AI ​​model and the second identity is determined based on the image data. On the other hand, upon determining that no match occurs, the processor N reapplies the AI ​​model to determine a third identity of the object 502A, or determines to ignore the first identity and use the second identity.

[0132] FIG. 6A is a diagram of an embodiment of an acoustic profile database 600 stored in one or more of memory devices 1-N (FIGS. 1A-2). The acoustic profile database 600 includes audio data set 1, audio data set 2, and audio data set N. The acoustic profile database 600 also includes environmental systems 1, 2, and N. The acoustic profile database 600 further includes correspondences between audio data sets 1, 2, and N and environmental systems 1, 2, and N. For example, the acoustic profile database 600 includes a first exclusive correlation between audio data set 1 and environmental system 1, a second exclusive correlation between audio data set 2 and environmental system 2, and an Nth exclusive correlation between audio data set N and environmental system N. An example of environmental system 1 is the first environmental system, an example of environmental system 2 is the second environmental system (FIG. 3), and an example of the Nth environmental system is type N of environment 500 or environment 500.

[0133] FIG. 6B is a diagram of an embodiment of a method 650 illustrating application of audio data to output sounds corresponding to an environmental system. The method 650 is executed by one or more of the processors 1-N (FIG. 1A-2), or a processor of the game console 112 (FIG. 1A-1), or a combination thereof. The method 650 includes an operation 652 of receiving, from the user 3, a selection of an environmental system to be simulated, e.g., presented. For example, the user 3 selects the environmental system N corresponding to the sounds the user 3 wants to hear, via the object 502I or a combination of the objects 502E and 502D. In this example, the user 3 is located in an environment different from the environment 500. For illustration, the object 502P has blinds or the object 502B (FIG. 5A) is removed from the environment 500 to generate a different environment. In this example, the user 3 accesses another application program, such as a game program of the game G1 or another application, from the game cloud. To illustrate, when a game program or other application is executed by one or more of processors 1-N, one or more image frames of one or more virtual scenes are generated and displayed on object 502A. Further, in this example, during or prior to execution of the game program or other application program, user 3 selects environmental system N. In this example, an indication of the selection of environmental system N is transmitted from object 502A over computer network 142 to server system 136 (FIGS. 1A-2). One or more of processors 1-N receive the indication of selection and access database 600 to determine that audio data set N is exclusively associated with environmental system N.

[0134] In operation 654 of method 650, one or more processors 1-N access, e.g., retrieve, audio data set N from database 600. In operation 656 of method 600, one or more processors 1-N apply audio data set N to output sounds corresponding to simulated environmental system N to user 3. For example, one or more processors transmit audio data set N to object 502A via computer network 142. Object 502A outputs sounds generated based on audio data set N while playing game G1 or during execution of other application programs. The sounds are output via a speaker of object 502A or via object 502G. Outputting sounds simulating environmental system N causes user 3 to feel as if he or she is in environmental system N rather than another environment.

[0135] FIG. 7 is a diagram of an embodiment of a system 700 for illustrating the method 600 (FIG. 6). The system 700 includes an input controller system 702, a display system 704, a server system 136, and a computer network 142. The display system 704 includes a communication device 706, a network transfer device 708, a CPU 712, a sound output system 710, and an audio memory device 714. An example of the display system 704 is the object 502O (FIG. 5A). Another example of the display system 704 is the object 502A (FIG. 5A). An example of the input controller system 702 is the input controller 122 (FIGS. 1A-2). Another example of the input controller system 702 is a handheld controller. Yet another example of the input controller system 702 is a combination of a mouse and a keyboard.

[0136] The communication device 706 has the same structure as the communication device 134 (FIG. 1A-2) and has similar functions as the communication device 134. Also, the network forwarding device 708 has the same structure as the network forwarding device 126 (FIG. 1A-2) and has similar functions as the network forwarding device 126. Furthermore, the CPU 712 has the same structure as the CPU 135 (FIG. 1A-2) and has similar functions as the CPU 135. An example of the audio memory device 714 is a buffer for storing audio data. The sound output system 710 includes a digital-to-analog converter (DAC), an amplifier, and a speaker.

[0137] The communication device 706 is coupled to the input controller system 702 and a CPU 712. The CPU 712 is coupled to a network transfer device 708, which is coupled to the server system 136 via the computer network 142. The CPU 712 is coupled to a DAC, which is coupled to an amplifier. The amplifier is coupled to a speaker. The CPU 712 is coupled to an audio memory device 714.

[0138] 6, in operation 652, the input controller system 702 generates an input signal, such as an indication, in response to a selection of environmental system N by user 3. User 3 uses the input controller system 702 to select one or more buttons on the input controller system 702. The communication device of the input controller 702 applies a communication protocol to the indication of the selection of environmental system N to generate one or more forwarded packets and transmits the forwarded packets to the communication device 706. The communication device 706 applies the communication protocol to the data packets to extract the indication of the selection of environmental system N from the forwarded packets and transmits the indication to the CPU 712.

[0139] The CPU 712 sends an indication of the selection of the environment system N to the network forwarding device 708. The network forwarding device 708 applies a network forwarding protocol to generate data packets including the indication of the selection of the environment system N and sends the data packets to the server system 136 via the computer network 142. The network forwarding device 138 of the server system 136 applies a network forwarding protocol to the data packets to extract the indication of the selection of the environment system N from the data packets and provides the indication to the one or more processors 1-N. The one or more processors 1-N perform an operation 654 ( FIG. 6B ) to identify and access, e.g., read, an audio data set N from one or more of the memory devices 1-N corresponding to the environment system N and transmit the audio data set N to the network forwarding device 138. As an example, in addition to performing the operation 654, the one or more processors 1-N may access an audio file (including audio information) of a game program or other application program and provide the audio information to the network forwarding device 138. An example of audio information is audio data of a virtual character jumping when the virtual character of game G1 jumps. Another example of audio information is a user providing commentary during a YouTube®™ video about a sporting event, such as a basketball game, when the other application program is YouTube®™. In this example, the sporting event is an example of a context.

[0140] The network forwarding device 138 applies a network forwarding protocol to generate data packets from the audio data set N and transmits the data packets to the display system 704 via the computer network 142 for application of the audio data set N in operation 656 (FIG. 6B). For example, the network forwarding device 138 embeds the audio data set N in data packets and transmits the data packets to the network forwarding device 708 via the computer network 142. In this example, the network forwarding device 708 applies a network forwarding protocol to the data packets to obtain the audio data set N and transmits the audio data set N to the CPU 712. The CPU 712 provides the audio data set N to a DAC, which converts the audio data set N from a digital format to an analog format to output an analog audio signal N. The DAC provides the analog audio signal N to an amplifier. The amplifier amplifies, e.g., increases or decreases, the amplitude of the analog audio signal N to output an amplified audio signal. The speaker converts the electrical energy of the analog audio signal N into sound energy to output the sound of the environmental system N based on the audio data set N, and simulates the environmental system N within the environment 500.

[0141] As another example, the network forwarding device 138 embeds the audio data set N and audio information associated with the game program or other application program in data packets and transmits the data packets to the network forwarding device 708 over the computer network 142. In this example, the network forwarding device 708 applies a network forwarding protocol to the data packets to obtain the audio data set N and the audio information and transmits the audio data set N and the audio information to the CPU 712. The CPU 712 provides the audio data set N and the audio information to a DAC, which converts the audio data set N and the audio information from a digital format to an analog format to output an analog audio signal. The DAC provides the analog audio signal to an amplifier. The amplifier amplifies, e.g., increases or decreases, the amplitude of the analog audio signal to output the amplified audio signal. The speaker converts the electrical energy of the analog audio signal into sound energy to output a first set of sounds of the game G1 or other application and a second set of sounds of the environmental system N as a background to the first set of sounds. In this example, the first and second sets are blended together when being output simultaneously. In this example, the context of the audio information associated with the game program or other application program and the context of the audio data set N are matched. To illustrate, if a YouTube®™ video includes a commentary of a sporting event, such as a baseball game, then the audio dataset N represents the sounds made during the sporting event, such as a baseball game. In this example, the baseball game is an example of a context.

[0142] In an embodiment, one or more of the processors 1-N determine not to provide audio information corresponding to a video of a virtual scene of the game G1 or other application program. For example, instead of accessing audio information of a virtual character jumping in a virtual scene outputted together with the virtual scene from one or more of the memory devices 1-N, one or more of the processors 1-N access an audio data set N from one or more of the memory devices 1-N and provide an audio data set N applied together with the virtual scene. As another example, instead of accessing audio information outputted together with a YouTube®™ video from one or more of the memory devices 1-N, one or more of the processors 1-N access an audio data set N from one or more of the memory devices 1-N and provide an audio data set N applied together with the YouTube®™ video. As yet another example, one or more of the processors 1-N stop applying audio information of a virtual character jumping in a virtual scene and apply the audio data set N instead. As yet another example, one or more of the processors 1-N stop applying audio information outputted together with the YouTube®™ video and apply the audio data set N instead.

[0143] In one embodiment, the display system 704 includes additional components, such as a microphone, a display screen, a GPU, an audio encoder, a video encoder, and a display screen. These components have similar structures and similar functions as the corresponding components of the glasses 120. For example, the microphone of the display system 704 has the same structure as the microphone M1, and if the display system 704 is a display device, the display screen of the display system 704 is larger than the display screen 132. As another example, if the display system 704 is glasses, such as the glasses 120 (FIGS. 1A-2), the display screen of the display system 704 has the same size as the display screen 132.

[0144] 8 is a diagram of an embodiment of a system 800 for illustrating a microphone 800. The microphone 800 is an example of a microphone M1, or M2, or M3. The microphone 800 includes a transducer, a converter, and an analog-to-digital converter (ADC). The transducer is coupled to a sound energy to electrical energy converter (SE converter), which is coupled to the ADC. An example of a transducer is a diaphragm. An example of an SE converter is a capacitor or a series of capacitors.

[0145] The transducer detects sound emanating from and / or reflected from objects in the environment and outputs vibrations. The vibrations are provided to an SE converter, which modifies the electric field generated in the SE converter to output an electrical signal, an audio analog signal. The audio analog signal is provided to an ADC, which converts the audio analog signal from an analog format to a digital format to output audio data, such as audio data set 1 or 2 or 3 or N.

[0146] 9 is a diagram of an embodiment of a system 900 illustrating a method for using direct audio data to create effects for an environmental system N and using reverberant audio data to determine object placement within the environmental system N as well as object properties such as the object's material type and the object's surface type. System 900 includes an audio data separator, direct audio data, reverberant audio data, a feature extractor, a classifier, and an AI model.

[0147] An example of direct audio data is audio data generated by a microphone based on sound received from a sound source via a direct path. Also, an example of reverberant audio data is audio data generated by a microphone based on sound received from a sound source via an indirect path. For the sake of explanation, the first direct audio data of the audio data set 1 is generated by the microphone M1 (FIG. A) based on sound received from the object 108K (FIG. 1A-1) via the path 106A. In this explanation, the first reverberant audio data of the audio data set 1 is generated by the microphone M1 based on sound received from the object 108K via the path 106B (FIG. 1A-1). As another explanation, the reverberant audio data is generated based on sound reflected or diffused from an object in the environmental system.

[0148] The audio data separator may be implemented as hardware or software, or a combination thereof. For example, the audio data separator may be a computer program, and the functions of the computer program may be executed by one or more of the processors 1-N (FIGS. 1A-2). As another example, the audio data separator may be an ASIC or a PLD.

[0149] The audio data separator is coupled to the feature extractor and to the audio decoder 144 of the server system 136 (FIG. 7). For example, the audio data separator is coupled between the feature extractor and the audio decoder 144.

[0150] The audio data separator receives audio data sets 1, 2, 3, and N from client devices 1 and 2 (FIG. 2) via computer network 142, network transport device 138, and audio decoder 144 (FIG. 7). The audio data separator determines parameters of audio data sets 1, 2, 3, and N, and identifies direct audio data and reverberated audio data in each of audio data sets 1, 2, 3, and N based on the parameters. For example, the audio data separator identifies first direct audio data and first reverberated audio data in audio data set 1, and identifies second direct audio data and second reverberated audio data in audio data set 2. For illustrative purposes, the audio data separator determines that a first portion of audio data set 1 has a first amplitude that is greater than a second amplitude of a second portion of audio data set 1. In this description, examples of amplitudes include peak-to-peak amplitude and zero-to-peak amplitude. Furthermore, in this description, the audio data separator determines that the first portion is first direct audio data and the second portion is first reverberated audio data. As another explanation, the audio data separator determines that a first portion of the audio data set 1 has a first frequency range and a second portion of the audio data set 1 has a second frequency range. Further in this explanation, the audio data separator determines that the first portion is first direct audio data and the second portion is first reverberant audio data. The audio data separator determines parameters for the audio data sets 1, 2, 3, and N in the same way that the feature extractor determines parameters for the audio data sets 1, 2, 3, and N.

[0151] The direct audio data output from the audio data separator is stored by one or more of the processors 1-N in one or more of the memory devices 1-N (FIG. 7). Based on the direct audio data, operations similar to operation 654 (FIG. 6) are performed. For example, upon receiving a selection of the environment system N to be simulated from the user 3, one or more of the processors 1-N access the direct audio data corresponding to the environment system N from one or more of the memory devices 1-N. As another example, the same operations as operation 654 are performed, except that the operations are performed on the direct audio data of the audio data set N instead of the audio data set N.

[0152] Further, an operation similar to operation 656 (FIG. 6B) is performed to apply the direct audio data to output a sound corresponding to the environmental system N. For example, the same operation as operation 656 is performed, except that instead of the audio data set N, the direct audio data of the audio data set N is applied. As another example, the direct audio data of the audio data set N is transmitted from the server system 136 (FIGS. 1A-2) to the object 502A or 502O (FIG. 5A). In this example, the object 502A or 502O outputs a sound generated based on the direct audio data of the audio data set N while playing the game G1 or during execution of other application programs. As yet another example, the direct audio data of the audio data set N is combined with the reverb audio data of the audio data set N, and a sound based on the audio data set N is output. For illustrative purposes, one or more of the processors 1-N combine the direct audio data of the audio data set N with the reverb audio data of the audio data set N to output the audio data set N. In this description, the audio data set N is then applied in the manner described above in operation 656 to simulate the environmental system N to the user 3. As yet another example, the direct audio data of the audio dataset N is combined by one or more of the processors 1-N with reverberant audio data of another audio dataset, such as audio dataset 1 or 2 or 3, to generate an additional audio dataset. In this example, the additional audio dataset is applied in the same manner as audio dataset N is applied in operation 656. For illustrative purposes, the additional audio dataset is transmitted from the server system 136 via the computer network 142 to the object 502A or 502O, and sound is output based on the additional audio dataset.

[0153] The reverberant audio data output from the audio data separator is sent to the feature extractor, for example, the first reverberant audio data and the second reverberant audio data are sent from the audio data separator to the feature extractor.

[0154] The feature extractor determines parameters of the reverberated audio data of any of the audio datasets 1, 2, 3, and N in the same manner as the feature extractor determines parameters of the audio datasets. For example, the feature extractor determines an amplitude 1a of the reverberated audio data of the audio dataset 1, an amplitude 2a of the reverberated audio data of the audio dataset 2, an amplitude 3a of the reverberated audio data of the audio dataset 3, and an amplitude Na of the reverberated audio data of the audio dataset N. As another example, the feature extractor determines a frequency 1a of the reverberated audio data of the audio dataset 1, a frequency 2a of the reverberated audio data of the audio dataset 2, a frequency 3a of the reverberated audio data of the audio dataset 3, and a frequency Na of the reverberated audio data of the audio dataset N. The feature extractor transmits the parameters of the reverberated audio data of the audio datasets 1, 2, and 3 to the classifier. The feature extractor also transmits the parameters of the reverberated audio data of the audio dataset N to the AI ​​model.

[0155] The classifier classifies parameters of the reverberant audio data of the audio datasets 1-3 based on the input datasets 1-3. For example, the classifier determines or identifies a combination of objects in an environmental system, such as the environment 102 (FIG. 1A-1) or the external environment 116 (FIG. 1A-1), or a combination thereof, from the input dataset 1, and establishes a correlation (such as a one-to-one correspondence) between the combination of objects and parameters of the reverberant audio data of the audio dataset 1. In this example, the classifier further determines or identifies an arrangement of the objects relative to each other, or a state of the objects, or a surface type of the objects, or a material type of the objects, or a combination of two or more thereof. In this example, the input dataset 1 includes a material type of the objects and a surface type of the objects. For illustrative purposes, the classifier receives in the input dataset 1 the material types of the objects 108A-108O (FIG. 1A-1) and the surface types of the objects 108A-108O.

[0156] Examples of material types include wood or plastic or glass or marble or stainless steel or leather or wool or cloth or cotton or polyester or tile or granite. For illustrative purposes, the list 150 (FIGS. 1A-4) includes an entry indicating that the desktop table is made of wood, and another entry indicating that the chair is made of leather. In this illustration, in the same way that the user 1 selects a chair and a desktop table in the list 150, the user 1 selects the type of material used to manufacture the desktop table, and the type of material used to manufacture the chair. In another illustration, the list 150 includes an entry indicating that the desktop table has a textured or smooth surface, and an entry indicating that the surface of the chair seat is intact, or worn, such as torn or torn. In this illustration, in the same way that the user 1 selects a chair and a desktop table in the list 150, the user 1 selects the type of surface of the desktop table and the type of surface of the chair.

[0157] In another description of the classification, the classifier receives in the input dataset 2 the identities of objects 154A-154N, 108O, 154P, and 154Q in the environment 152, as well as the identity of object 154R via a list, such as list 150, and user account 1. In this description, the classifier receives in the input dataset 2 the state, material type, and surface type of objects 154A-154N, 108O, 154P, and 154Q (FIG. 1B). Further, in this description, the classifier determines from a sensing direction of reverberant audio data in the audio dataset 2 that object 154D is located closer to microphone M1 than object 154F (FIG. 1B).

[0158] As yet another illustration, the classifier receives in the input dataset 3 the identities of the objects 154A-154N, 108O, 154P, and 154Q in the environment 152, as well as the identity of the object 154R via a list, such as list 150, and the user account 2. In this illustration, the classifier receives in the input dataset 3 the state, material type, and surface type of the objects 154A-154N, 108O, 154P, and 154Q (FIG. 1B). Further, in this illustration, the classifier determines from the sensing direction that the object 154C is located closer to the microphone M2 than the object 154E (FIG. 1B). In this illustration, the sensing direction is determined from parameters of the reverberant audio data of the audio dataset 2.

[0159] The AI ​​model is trained based on correlations between the reverberated audio data of the audio datasets 1-3 associated with the environments 102, 116, 152, and 158 (FIGS. 1A-1 and 1B) and parameters of the input datasets 1-3. For example, the AI ​​model is provided by a classifier with an indication of a first correlation, such as a link or one-to-one correspondence, between a set including an amplitude 1a of the reverberated audio data of the audio dataset 1, a frequency 1a of the reverberated audio data of the audio dataset 1, and a sensing direction 1a of the reverberated audio data of the audio dataset 1, and a set including a first type of environment, a first combination of objects in the first type of environment, a first arrangement of objects, a first state of the objects, a first set of material types of the objects, and a first set of surface types of the objects. In this example, the one or more amplitudes determined from the reverberated audio data of the audio dataset 1 are referred to herein as amplitude 1a. Also in this example, the one or more frequencies determined from the reverberated audio data of the audio dataset 1 are referred to herein as frequency 1a, and the direction in which the reverberated audio data of the audio dataset 1 is sensed is referred to herein as sensing direction 1a. For purposes of illustration, amplitude 1a, frequency 1a, and perceived direction 1a are example parameters of reverberant audio data of audio dataset 1. Also, in this example, the classifier provides the AI ​​model via user account 1 a first type of environment 102, a first combination of objects in environment 102 and external environment 116, a first state of the objects, a first arrangement of the objects, a first set of material types of the objects, and a first set of surface types of the objects.

[0160] As another example, the AI ​​model is provided with an indication of a second correlation, such as a one-to-one correspondence, between a set including the amplitude 2b, the frequency 2b, and the sensing direction 2a of the reverberant audio data of the audio dataset 2 and a set including a second type of environment, a second combination of objects in the second type of environment, a second arrangement of objects, a second state of objects, a second set of material types of objects, and a second set of surface types of objects. In this example, the AI ​​model receives the amplitude 2b, the frequency 2b, and the sensing direction 2b from the classifier. In this example, the amplitude 2b, the frequency 2b, and the sensing direction 2b are determined by analyzing the reverberant audio data of the audio dataset 2 captured by the microphone M1. Also, in this example, the one or more amplitudes determined from the reverberant audio data of the audio dataset 2 are referred to herein as amplitude 2b, the one or more frequencies determined from the reverberant audio data of the audio dataset 2 are referred to herein as frequency 2b, and the direction in which the reverberant audio data of the audio dataset 2 is perceived is referred to herein as sensing direction 2b. For purposes of illustration, the amplitude 2b, the frequency 2b, and the sensing direction 2b are examples of parameters of the reverberant audio data of the audio dataset 2. In this example, the classifier also provides the AI ​​model with a second type of environment 152, a second combination of objects within environment 152 and external environment 158, a second state of the objects, a second arrangement of the objects, a second set of material types of the objects, and a second set of surface types of the objects. In this example, the second type, second combination, second state, and second arrangement, the second set of material types of the objects, and the second set of surface types of the objects are received via user account 1.

[0161] As yet another example, the AI ​​model is provided by the classifier with an indication of a third correlation (e.g., a link) between a set including the amplitude 3a, the frequency 3a, and the sensing direction 3a of the reverberant audio data of the audio dataset 3 and a set including a third type of environment, a third combination of objects in the third type of environment, a third arrangement of objects, a third state of objects, a third set of material types of objects, and a third set of surface types of objects. In this example, the AI ​​model receives the amplitude 3c, the frequency 3c, and the sensing direction 3c from the classifier. In this example, the amplitude 3c, the frequency 3c, and the sensing direction 3c are determined by analyzing the reverberant audio data of the audio dataset 3 captured by the microphone M2. Also, in this example, the one or more amplitudes determined from the reverberant audio data of the audio dataset 3 are referred to herein as the amplitude 3c, the one or more frequencies determined from the reverberant audio data of the audio dataset 3 are referred to herein as the frequency 3c, and the direction in which the reverberant audio data of the audio dataset 3 is perceived is referred to herein as the sensing direction 3c. For purposes of illustration, the amplitude 3c, the frequency 3c, and the sensing direction 3c are examples of parameters of the reverberant audio data of the audio dataset 3. In this example, the classifier also provides to the AI ​​model a third type of environment 152, a third combination of objects within environment 152 and external environment 158, a third state of the objects, a third arrangement of the objects, a third set of material types of the objects, and a third set of surface types of the objects. In this example, the third type, the third combination, the third state, the third arrangement, the third set of material types of the objects, and the third set of surface types of the objects are received via user account 2.

[0162] When the AI ​​model is provided with the amplitude Na, frequency Na, and sensing direction Na from the feature extractor, the AI ​​model provides a model output 902. For example, when determining that the amplitude Na is within a predetermined range from the amplitude 1a and outside a predetermined range from the amplitude 2a, the AI ​​model indicates that there is a greater than 50% probability that the reverberant audio data of the audio data set N is generated based on sound reflected from a plastic table or a table with a smooth top. In this example, the probability that the table is made of plastic or the table has a smooth top is an example of the model output 902. As another example, when determining that the frequency Na is within a predetermined range from the frequency 2a and outside a predetermined range from the frequency 1a, the AI ​​model indicates that there is a greater than 50% probability that the reverberant audio data of the audio data set N is generated based on sound reflected from a table with an uneven surface or a table with a marble top. In this example, the probability that the table is made of marble or the table has an uneven surface is an example of the model output 902. As another example, a combination of two or more of the amplitude Na, frequency Na, and sensing direction Na is used to determine the material type of any of the objects in the environment 500, or the surface type of any of the objects in the environment 500.

[0163] Note that the type of material an object is made of and the type of surface an object is made of are examples of properties of an object.

[0164] In one embodiment, during operation 654, a visual mapping of the scene of the environment N is created on the display screen 132 (FIG. 1A-2) based on the image data captured by the cameras C1 and C2 (FIG. 1B). For example, in addition to directly applying the audio data to simulate the environment system N, the arrangement of the objects in the environment system N and the graphical parameters of the objects in the environment system N are applied to simulate the environment system N. For illustration, the arrangement of the objects in the environment system N and the graphical parameters of the objects in the environment system N are accessed by one or more of the processors 1-N from one or more of the memory devices 1-N of the server system 136 and transmitted to the glasses 120 (FIG. 1A-2) via the computer network 142. Upon receiving the arrangement of the objects in the environment system N and the graphical parameters of the objects in the environment system N, the GPU 130 (FIG. 1A-2) displays the graphical parameters of the objects according to the arrangement to simulate the environment system N when the user 3 is in a different environment.

[0165] It should be noted that in various embodiments, one or more features of some of the embodiments described herein may be combined with one or more features of one or more of the remaining embodiments described herein.

[0166] The embodiments described in this disclosure may be implemented with a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. In one implementation, the embodiments described in this disclosure are practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wired or wireless network.

[0167] With the above embodiments in mind, it should be understood that in one implementation, the embodiments described in this disclosure employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the embodiments described in this disclosure are useful machine operations. Some embodiments described in this disclosure also relate to devices or apparatus for performing these operations. The apparatus may be specially constructed for the required purpose, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, in one embodiment, various general-purpose machines are used with computer programs written in accordance with the teachings herein. Alternatively, it may be more convenient to construct more specialized apparatus to perform the required operations.

[0168] In an embodiment, some embodiments described in this disclosure are embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that stores data which is subsequently read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), ROM, RAM, compact disc ROM (CD-ROM), CD-recordable (CD-R), CD-rewritable (CD-RW), magnetic tape, optical data storage devices, non-optical data storage devices, etc. As an example, the computer readable medium includes computer readable tangible media distributed over network-coupled computer systems such that the computer readable code is stored and executed in a distributed fashion.

[0169] Additionally, although some of the above embodiments are described with respect to a gaming environment, in some embodiments other environments are used in place of a game, such as, for example, a video conferencing environment.

[0170] Although the method operations have been described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be adjusted to occur at slightly different times, or may be distributed within a system that allows processing operations to occur at various intervals related to processing, so long as the processing of the overlay operation is performed in the desired manner.

[0171] Although the foregoing embodiments described in this disclosure have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Thus, the present embodiments should be considered as illustrative and not restrictive, and the present embodiments should not be limited to the details set forth herein, but may be modified within the scope of the appended claims and their equivalents.

Claims

1. 1. A method for determining an environment in which a first user is located, comprising: receiving a plurality of audio data sets based on sounds originating in a plurality of environments, each of the plurality of environments having a different combination of objects; receiving input data relating to the plurality of environments; training an artificial intelligence model (AI model) based on the plurality of audio data sets and the input data; applying the AI ​​model to audio data captured from an environment surrounding the first user to determine a type of the environment; A method comprising:

2. 2. The method of claim 1, further comprising extracting a plurality of features from the plurality of audio data sets, the plurality of features comprising a plurality of amplitudes of the plurality of audio data sets, a plurality of frequencies of the plurality of audio data sets, and a plurality of sensed directions from which the sound is perceived.

3. classifying the plurality of features and outputting correlations between the plurality of features and a plurality of types of the plurality of environments, the object within the plurality of environments, a plurality of configurations of the object, and a plurality of states of the object within the plurality of environments; Training the AI ​​model includes: providing the correlations between the features and the types of environments, the objects within the environments, the configurations of the objects, and the states of the objects within the environments to the AI ​​model; determining by the AI ​​model a plurality of probabilities based on the correlations between the plurality of features and the plurality of types of the plurality of environments, the object within the plurality of environments, the plurality of configurations of the object, and the plurality of states of the object within the plurality of environments, wherein the plurality of probabilities provide a contingency that the audio data captured from the environment indicates the type of environment, a contingency that the type of environment includes a plurality of items, a contingency that the plurality of items have a plurality of states, and a contingency that the plurality of items have a configuration; The method of claim 2 , comprising:

4. receiving an indication of the type of the environment to be simulated; accessing the audio data captured from the environment based on the type of the environment being simulated; providing the audio data captured from the environment to a client device to output a sound corresponding to the type of the environment; The method of claim 1 further comprising:

5. The method of claim 1 , wherein the plurality of audio data sets comprises audio data generated from sounds emanating from one or more of the objects in the plurality of environments and sounds reflected from remainders of the objects.

6. The method of claim 1 , wherein the input data comprises data identifying the objects in the multiple environments, or image data captured by a camera in the multiple environments, or a combination thereof.

7. The method of claim 1 , wherein the plurality of audio data sets are captured as the second user moves from one location to another.

8. The method of claim 1 , wherein the multiple audio data sets are captured when multiple users, including a second user and a third user, are present at a location.

9. 2. The method of claim 1, further comprising applying the AI ​​model to the audio data captured from the environment surrounding the first user to identify one or more objects, or one or more states of the one or more objects, or a positioning of the one or more objects, or a combination thereof, in the environment surrounding the first user.

10. A server for determining an environment in which a user is located, comprising: configured to receive, via a computer network, a plurality of audio data sets based on sounds emitted in a plurality of environments, each of the plurality of environments having a different combination of objects; configured to receive, via the computer network, input data relating to the plurality of environments; configured to train an artificial intelligence (AI) model based on the plurality of audio data sets and the input data; a processor configured to apply the AI ​​model to audio data captured from an environment surrounding a user to determine a type of the environment; a memory device coupled to the processor; Including, the server.

11. The processor, 11. The server of claim 10, configured to extract a plurality of features from the plurality of audio data sets, the plurality of features comprising a plurality of amplitudes of the plurality of audio data sets, a plurality of frequencies of the plurality of audio data sets, and a plurality of sensing directions from which the sound is perceived.

12. the processor is configured to classify the features and output correlations between the features and a plurality of types of the environments, the object within the environments, a plurality of configurations of the object, and a plurality of states of the object within the environments; To train the AI ​​model, the processor: providing the correlations between the features and the types of environments, the objects within the environments, the configurations of the objects, and the states of the objects within the environments to the AI ​​model; 12. The server of claim 11, configured to use the AI ​​model to determine a plurality of probabilities based on the correlations between the plurality of features and the plurality of types of the plurality of environments, the objects within the plurality of environments, the plurality of configurations of the objects, and the plurality of states of the objects within the plurality of environments, wherein the plurality of probabilities provide a contingency that the audio data captured from the environment indicates the type of environment, a contingency that the type of environment includes a plurality of items, a contingency that the plurality of items have a plurality of states, and a contingency that the plurality of items have a configuration.

13. The processor, configured to receive an indication of a type of environment to be simulated; configured to access the audio data captured from the environment based on the type of the environment being simulated; 13. The server of claim 12, configured to provide the audio data captured from the environment to a client device to cause it to output sounds corresponding to the type of the environment.

14. The server of claim 10 , wherein the plurality of audio data sets comprises audio data generated from sounds emanating from one or more of the objects in the plurality of environments and sounds reflected from remainders of the objects.

15. The server of claim 10 , wherein the input data includes data identifying the objects in the multiple environments, or image data captured by a camera in the multiple environments, or a combination thereof.

16. 11. The server of claim 10, wherein the processor is configured to apply the AI ​​model to the audio data captured from the environment surrounding the user to identify one or more objects, or one or more states of the one or more objects, or a position of the one or more objects, or a combination thereof, in the environment surrounding the user.

17. 1. A system for determining an environment in which a user is located, comprising: configured to generate a plurality of audio data sets based on sounds emitted in a plurality of environments, each of the plurality of environments having a different combination of objects; a plurality of client devices configured to receive input data relating to the plurality of environments; a server coupled to the plurality of client devices, configured to receive the plurality of audio data sets from the plurality of client devices over a computer network; configured to receive the input data regarding the plurality of environments from the plurality of client devices via the computer network; configured to train an artificial intelligence (AI) model based on the plurality of audio data sets and the input data; the server configured to apply the AI ​​model to audio data captured from an environment surrounding a user to determine a type of the environment; Including, the system.

18. 20. The system of claim 17, wherein the server is configured to extract a plurality of features from the plurality of audio data sets, the plurality of features comprising a plurality of amplitudes of the plurality of audio data sets, a plurality of frequencies of the plurality of audio data sets, and a plurality of sensing directions for sensing the sound.

19. the server is configured to classify the features and output correlations between the features and a plurality of types of the environments, the objects within the environments, a plurality of configurations of the objects, and a plurality of states of the objects within the environments; To train the AI ​​model, the server: providing the correlations between the features and the types of environments, the objects within the environments, the configurations of the objects, and the states of the objects within the environments to the AI ​​model; 20. The system of claim 18, configured to use the AI ​​model to determine a plurality of probabilities based on the correlations between the plurality of features and the plurality of types of the plurality of environments, the object within the plurality of environments, the plurality of configurations of the object, and the plurality of states of the object within the plurality of environments, wherein the plurality of probabilities provide a contingency that the audio data captured from the environment is indicative of the type of environment, a contingency that the type of environment includes a plurality of items, a contingency that the plurality of items have a plurality of states, and a contingency that the plurality of items have a configuration.

20. The server, configured to receive an indication of a type of environment to be simulated; configured to access the audio data captured from the environment based on the type of the environment being simulated; 20. The system of claim 19, configured to provide the audio data captured from the environment to a client device to cause it to output sounds corresponding to the type of the environment.

Citation Information

Patent Citations

  • Acoustic program, acoustic device, and acoustic system

    EP3799035A1

  • System and method to convert two-dimensional video into three-dimensional extended reality content

    US11551407B1

  • Stereophonic apparatus for blind and visually-impaired people

    US20200258422A1

  • Room acoustics simulation using deep learning image analysis

    WO2020139588A1

  • Systems and methods for generating audio presentations

    WO2021216060A1