Room Acoustic Simulation Using Deep Learning Image Analysis

Abstract

The method includes receiving an image of a real-world environment, classifying the image using a machine learning classifier to generate classifications associated with sound presets for simulating an acoustic environment, the sound presets each including acoustic parameters that describe sound reverberation, and selecting a sound preset from the sound presets based on the classification.

Description

[Technical field]

[0001] Claiming priority This application claims priority to U.S. Provisional Patent Application No. 62 / 784,648, filed December 24, 2018, which is incorporated by reference herein in its entirety.

[0002] This disclosure relates to classifying images for augmented reality. [Background technology]

[0003] Augmented reality (AR) is the blending of real and virtual worlds to generate new environments and visualizations in which real or real physical objects and digital or virtual objects can coexist and interact in real time. AR brings a virtual world with lifelike sights and sounds into the user's real-world environment. AR mixes virtual sounds from virtual sound objects with real sounds in a real acoustic environment. The virtual sounds from the virtual sound objects must match the equivalent real-world sounds when played to the user through headphones to ensure a pleasant AR experience. Otherwise, the user experiences a degradation of the AR experience. Conventional techniques use complex multi-step processes to match the virtual sounds with the equivalent real-world sounds. Such complexity introduces noticeable auditory delays into the AR simulation, which can degrade the user experience. Moreover, the complexity undesirably increases the processing demands for the AR device, and therefore its cost. Summary of the Invention

[0004] [Brief description of the drawings]

[0005] [Figure 1A] FIG. 1 is a high-level block diagram of an example extended reality (XR) system configured to provide an AR experience to a user. [Figure 1B]FIG. 1 is a perspective view of an exemplary AR device or system that may be worn by a user and configured to convey an AR experience to the user. [Diagram 2] FIG. 13 is a diagram of an example sound response for an environment that may be simulated by an XR system based on an acoustic preset. [Diagram 3] FIG. 1 illustrates an example method of machine learning (ML)-based classification of images combined with rendering of sounds from virtual sound objects that may be performed by an XR system. [Figure 4] 1 is a flowchart of an exemplary method for using ML classification to directly classify images of real-world environments into acoustic presets of acoustic parameters for acoustic environment simulation. [Diagram 5] 1 is a flowchart of an example method for using ML classification to classify images of real-world environments into room types from which acoustic presets for acoustic environment simulation may then be derived. [Figure 6] 1 is a flowchart of an example method for using ML classification to directly classify images of real-world environments into acoustic presets of acoustic parameters for acoustic environment simulation based on a cache of real-world environment scenes and their associated acoustic presets. [Figure 7] 1 is a flowchart of an exemplary method for using ML classification to directly classify images of real-world environments into general / primary acoustic presets and secondary acoustic modifiers for acoustic environment simulation. [Figure 8] 8 is a flowchart of an example method that extends the method of FIG. 7 to generate a three-dimensional (3D) mesh and acoustic parameters derived from material properties of the 3D mesh. [Figure 9A] FIG. 1 illustrates an example method for training an ML classifier for an XR system based on training images according to a first training scenario and classifying images using the previously trained ML classifier. [Figure 9B] FIG. 9B is a diagram of the preparation of pre-training data for the training method of FIG. 9A. [Figure 10]FIG. 13 is a diagram of an example method for training an ML classifier according to the second and third training scenarios. [Figure 11A] FIG. 13 is a diagram of an example method for training an ML classifier according to a fourth training scenario. [Figure 11B] FIG. 1 is a diagram of an exemplary image sensor that includes back-to-back 180° fisheye lenses to capture a 360° image of a room within the two camera fields of view. [Figure 12] FIG. 1B is a block diagram of an example XR processor that includes additional functionality compared to the embodiment of the XR processor of FIG. 1A. [Figure 13] A block diagram of an example interactive audio engine (IAE) that may be used within or in conjunction with an XR processor to perform audio environment simulation based on acoustic parameters is provided. [Figure 14] FIG. 13 is a flow diagram of an exemplary acoustic parameter refinement process that may be performed by the XR processor of FIG. [Figure 15] 13 is a table of exemplary acoustic parameters generated by the XR processor of FIG. 12. [Figure 16] FIG. 1 is a block diagram of an example XR processor that uses cached audio presets. [Figure 17] 1 is a flow chart of an exemplary method of transitioning between sound presets. [Figure 18] 11 is a flow chart of an example method of transitioning between audio presets and performing an enabled image safety check. [Figure 19] 11 is a flow chart of an exemplary method of transitioning between sound presets in a calibrated scene embodiment. [Figure 20] FIG. 1 illustrates an example method of deep neural network (DNN) training of an ML classifier, where the ML classifier starts in an untrained condition. [Figure 21]FIG. 1 is a block diagram of an example computing device in which the XR processor and / or interactive audio engine may be implemented. [Figure 22] 1 is a flow chart of an exemplary summarized method for directly classifying an image into an acoustic preset. [Diagram 23] 23 is a flowchart of an exemplary method for directly classifying a subsequent or second image to an image classified in the method of FIG. 22. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] Exemplary embodiments Extended reality (XR) generally encompasses virtual reality (VR) and augmented reality (AR), and is sometimes referred to as mixed reality (MR). Audio signal reproduction systems have evolved to deliver three-dimensional (3D) audio to listeners. In 3D audio, sound is produced by headphones or earphones (collectively referred to herein as "headphones" for brevity) and may involve or include a virtual placement of sound sources in a real or theoretical 3D space or environment that is audibly perceived by the listener. For example, virtualized sound may be provided above, below, or even behind the listener who hears the 3D audio processed sound. Conventional audio reproduction through headphones tends to provide sound that is perceived to originate or emanate from inside the listener's head. In one example, audio signals delivered by headphones, including using a conventional stereo pair of headphones, may be specially processed to achieve a 3D audio effect, such as to provide the listener with a perceived spatial sound environment.

[0007] 3D audio headphone systems can be used for VR applications, such as to provide a listener with the perception of a sound source at a particular location in a local or virtual environment where no actual sound source is present. Similarly, 3D audio headphone systems can be used for AR applications, such as to provide a listener with the perception of a sound source at a location where no actual sound source is present, yet in a manner in which the listener still remains at least partially aware of one or more actual sounds in the local environment. Computer-generated audio rendering for VR or AR can leverage signal processing technology developments in gaming and virtual reality audio rendering systems and application programming interfaces, such as building on and extending previous developments in the fields of computer music and architectural acoustics. Various binaural techniques, simulated reverberation, physical room acoustic modeling, and auralization techniques can be applied to provide users with an enhanced listening experience. VR or AR signal processing systems can be configured to reproduce some sounds such that they are perceived by the listener to be emanating from an external source in the local environment, rather than from headphones or from a location inside the listener's head.

[0008] Compared to VR 3D audio, AR audio includes the additional challenge of encouraging participants to suspend disbelief, such as by providing simulated environmental acoustics and source-environment interactions that are substantially consistent with the acoustics of the local listening environment. This presents the challenge of providing audio signal processing for the virtual or added signal in a manner that includes or represents the user's environment, and such that the signal is easily distinguishable from other sounds that occur naturally or are played over headphones in the environment. Such audio signal processing provides accurate sound sources within the virtual sound field by matching and applying reverberation characteristics, including decay times, reverberation loudness characteristics, and / or reverberation equalization characteristics (e.g., spectral content of reverberation) for a given listening environment. In audiovisual AR applications, computer-generated sound objects (referred to as "virtual sound objects") can be rendered to mix with the physical environment as naturally heard by an observer / listener through acoustically transparent headphones. Such mixing may include or use binaural pseudo-reverberation processing to match or approximate the local environmental acoustics.

[0009] The embodiments presented herein provide a practical and efficient approach to extend 3D audio rendering algorithms or simulations to closely match or approximate the physical local environment acoustics. The embodiments provide solutions to the problems and / or challenges mentioned above, and also provide advantages that will become apparent from the following description. The embodiments may be used in 3D audio applications, such as VR and AR. The embodiments use machine learning (ML) techniques to infer acoustic properties of the local environment, such as reverberation properties, directly from images of the local environment captured by an image sensor. The embodiments may then use the inferred acoustic properties in an acoustic simulation of the environment that matches or approximates the actual acoustics of the local environment. Based on the inferred acoustic properties, the acoustic environment simulation seamlessly blends the virtual sound with the local environment when perceived by a listener through headphones.

[0010] More specifically, the embodiments presented herein use ML techniques to train one or more neural networks of ML classifiers to accurately infer acoustic properties of an unknown environment using image sensors. The inferred acoustic properties are then used to create an acoustic context for virtual sound objects in the form of an acoustic environment simulation created in real time within the environment. The embodiments advantageously: utilize camera sensors that are commonly available on XR devices, enable the use of typical voice plug-ins used in machine learning engines such as Unity and Unreal engines, reduce the complexity, processing demands, and delays associated with matching virtual sounds to equivalent real-world sounds in a real-time AR environment compared to conventional techniques, provide scalable implementation depending on the availability of image sensors, and can be implemented as a deep learning inference engine.

[0011] System Level Description At a high level, the embodiments presented herein utilize ML techniques to classify images of real-world (i.e., actual) environments directly into acoustic presets that represent a set of acoustic parameters for acoustic environment simulation (AES). The set of acoustic parameters represents a set of characteristics sufficient to perform AES. The AES simulates or models the sound response of the real-world environment based on the set of acoustic parameters of the acoustic preset. The acoustic preset is a parametric representation of the sound response. The AES applies the sound response to sounds from virtual sound objects (virtually) placed in the real-world environment, transforming the sounds into realistic sounds that feel as if they are actually emanating from the virtual sound objects when played back to the user through headphones. The above-mentioned real-world environments include any real-world environment or space that has reverberant properties, such as, but not limited to, rooms, auditoriums, concert halls, amphitheaters, etc. The rooms may also include rooms in a residence, such as kitchens, living rooms, dining rooms, bathrooms, etc. The rooms may also include office spaces, and the like.

[0012] 1A, there is a high-level block diagram of an exemplary XR system 100 configured to provide an AR experience to a user, according to embodiments presented herein. Although the embodiments are primarily described in the context of AR applications, the embodiments apply equally to VR applications. In FIG. 1A, the dashed lines generally represent the flow of parameters, e.g., the flow of acoustic parameters, while the solid lines generally represent the flow of image and sound signals.

[0013] The XR system 100 includes an image sensor 102 for capturing a sequence of images, or footage (collectively, "images") 103, an AR display 104, a headset 106 including left and right headphones, an optional position sensor 107, and an XR processor or processor 108 coupled to and in communication with the image sensor, the AR display, the headset, and the position sensor. The XR processor 108 includes (i) an ML-based acoustic environment classifier 120 (simply referred to as "ML classifier" 120) including one or more neural networks for classifying the images 103 into acoustic presets 122 according to the embodiments presented herein, and an interactive audio engine (IAE) 124. The IAE 124 may be implemented as part of the XR processor 108 as shown in FIG. 1A, or may be separate from the XR processor. In one example, the ML classifier 120 may include one or more convolutional neural networks (CNNs), such as AlexNet, GoogLeNet, and ResNet50. In other examples, the ML classifier 120 includes a non-CNN neural network suitable for classifying images as described herein. The IAE 124 generates or performs the AES 126 based on the acoustic presets 122 from the ML classifier 120, and also generates one or more virtual sound objects 128 for virtual placement within a scene of a real-world environment.

[0014] The image sensor 102 may include a video camera for capturing a series of images 103 of the real-world environment. The image sensor 102 may be positioned at different positions and orientations (collectively, "viewpoints") in the real-world environment to capture images 103 of different scenes of the real-world environment from the different viewpoints. For example, the image sensor 102 may include a video camera worn by a user who is the subject of the AR experience, such that the video camera operates to capture different scenes of the real-world environment as the user moves around in the real-world environment. The position sensor 107 senses or determines the position and orientation of one or more objects, including the user, in the environment and provides position information 114 to the XR processor 108 that indicates the object's position and orientation.

[0015] At a high level, in operation, the XR processor 108 processes (i) the image 103 of the real-world environment, (ii) the sound (i.e., sound signal) from the virtual sound object 128, and (iii) the position information 114, when available, to generate a video signal 136 and a sound signal 138 that represent a scene of the real-world environment augmented with the virtual sound object and other virtual information. The AR display 104 converts the video signal 136 into a video and plays the video to the user. The headphones of the headset 106 convert the sound signal 138 into a sound and plays the sound to the user. More specifically, the ML classifier 120 of the XR processor 108 utilizes deep learning neural network techniques to classify the image 103 into a sound preset 122. Each of the sound presets 122 represents a respective set of sound parameters, such as reverberation ("reverb") parameters, that represent real-world environmental sound characteristics. The IAE 124 performs the AES 126 based on the acoustic presets 122 to simulate or model the acoustic response, including reverberation, for the real-world environment. The IAE 124 also generates one or more virtual sound objects 128 located in various virtual locations within the scene of the real-world environment. The AES 126 applies the sound response to the sound signals generated by the virtual sound objects 128 and converts the sound signals from the virtual sound objects into sound signals 118 that convey a realistic sound for the virtual sound objects. That is, the AES 126 models at least the sound reverberation, for example, for the virtual sound objects.

[0016] 1B, there is a perspective view of an AR device 150 that may be worn by a user and used to communicate an AR experience to the user. The device 150 includes a wide-angle tracking camera 160, a red, green, blue (RGB) camera 164, a microphone array 168, and stereo headphones 170, all coupled to an embedded XR processor, not shown in FIG. 1B. The AR device 150 may also include a time-of-flight depth sensor and additional stereo cameras coupled to the XR processor.

[0017] Referring to FIG. 2, there is an illustration of a typical sound response 200 for a real-world environment that may be simulated by the AES based on the acoustic preset 122. The sound response 200 occurs in response to a sound impulse emanating from a sound source in the real-world environment. The sound response 200 may be recorded at a listener's position in the real-world environment that is spaced apart from the sound source. The sound response 200 includes a direct path (DP) sound, a reflection (REF) including an early reflection following the direct path sound, and a reverberant energy or reverberation (REV) following the reflection. The reflection REF starts after the reflection delay that the direct path sound DP occurs, and the reverberant REV starts after the reverberant delay that the reflection REF starts. The amplitude of the reverberant REV decays according to the decay time of the reverberation. In the embodiment presented herein, the AES 126 employs the acoustic parameters of the acoustic preset 122 in addition to other acoustic parameters to simulate / represent the direct path sound, early reflection, and reverberation for a sound from a virtual sound object 128.

[0018] FIG. 3 is a diagram of an exemplary method 300 of ML-based direct classification of an image 103 into an acoustic preset 122 and rendering of sounds from a virtual sound object 128, which may be performed by the ML classifier 120 and the IAE 124. The method 300 assumes that the ML classifier 120 has already been trained in a pre-training operation to directly classify images into acoustic presets. Briefly, the training involves training the ML classifier 120 with different images labeled with different ones of the acoustic presets. In one example, the acoustic presets may include M, e.g., 10, 20, etc., acoustic presets P1-PM associated with labels L1-LM, respectively. Each acoustic preset Pi represents a set of acoustic parameters AP1-APN used for acoustic environment simulation, e.g., AES 126. The labels L1-LM and acoustic presets P1-PM (and their associated sets of acoustic parameters AP1-APN) may be stored in a memory of the XR processor 108. The acoustic parameters AP1-APN have respective values ​​for a given acoustic preset Pi, and the values ​​vary depending on the acoustic presets P1-PM. The acoustic parameters AP1-APN may include at least acoustic reverberation parameters. In general, the reverberation parameters may include, for example, reverberation decay time, reverberation loudness characteristics, and / or reverberation equalization characteristics (e.g., spectral content of reverberation). More specifically, the reverberation parameters may include high frequency decay and death time, low frequency decay and death time, damping, diffusion, density, room size, etc. The reverberation parameters may include parameters as defined in any known or later developed acoustic-related standard, such as Interactive 3D Audio Rendering Guidelines Level 2.0 (I3DL2).

[0019] The acoustic parameters AP1 to APN may include additional acoustic parameters, such as one or more sound reflection parameters / coefficients, one or more sound absorption parameters / coefficients, and so on.

[0020] At 302, the XR processor 108 selects or establishes one of the acoustic presets P1-PM as a default or initial acoustic preset for the AES. The acoustic parameters AP1-APN of the default acoustic preset represent the initial acoustic parameters.

[0021] At 304, the ML classifier 120 receives an image in the sequence of images 103 captured by the image sensor 102. In steady state operation, the image may be a current image in the sequence of images 103 and a future image in the sequence of images 103 to be processed sequentially through the method 300.

[0022] In 306, referred to as "inference", the (pre-trained) ML classifier 120 directly classifies the image into a set of multiple (current) classifications corresponding to the acoustic presets P1-PM. The set of classifications may simply include labels L1-LM indicating the acoustic presets P1-PM with confidence levels C1-CN associated with each of the labels. The labels L1-LM may be used to access the (known) acoustic presets P1-PM and thus the (known) acoustic parameters AP1-APN of the acoustic presets. For example, the acoustic presets P1-PM may be indexed and therefore stored to be retried based on the labels L1-LM. The confidence levels Ci represent the probability that the associated labels Li / acoustic presets Pi are correct for the image, i.e., that the image has been correctly classified into the labels Li / acoustic presets Pi. In this way, the classification may be considered as a soft decision rather than a hard decision.

[0023] At 308, the XR processor 108 selects the label / acoustic preset associated with the highest confidence level among the confidence levels C1-CN in the classification to generate a (current) selected label / acoustic preset. The selected acoustic preset replaces the default acoustic preset from operation 302. The selected acoustic preset is retrieved from memory (i.e., the acoustic parameters AP1-APN of the selected preset are retrieved from memory).

[0024] At 310, the XR processor 108 updates the IAE 124 with the selected acoustic preset, ie, with the parameters AP1-APN of the selected acoustic preset.

[0025] The method 300 repeats sequentially as the next image in the sequence of images 103 arrives for classification, generating a sequence of classification results corresponding to the sequence of images and which are passed sequentially to the IAE 124 for the AES 126 .

[0026] A variation of method 300 conditions the acoustic preset update to IAE 124 on a predetermined confidence level threshold, which may introduce hysteresis in the updates provided to the IAE as the method iterates through classifying successive images. More specifically, the variation updates IAE 124 only when one or more (current) classifications have a confidence level that exceeds the confidence level threshold, in which case operations 308 and 310 proceed as described above. Otherwise, the variation does not update IAE 124, i.e., the variation simply maintains the last previous update to IAE that exceeded the confidence level threshold. Assuming that the classifications represent confidence levels as probabilities or include softmax values ​​(i.e., soft decisions) associated with them, the confidence level threshold may be set equal to, for example, a probability of 0.7. In that case, the update occurs only when the corresponding probability exceeds >0.7. To add hysteresis, the update may occur only when the average confidence level over a predetermined number >1 of successive classifications (through operation 306) exceeds 0.7.

[0027] Classification Flowchart Next, various methods of classifying images using ML techniques will be described with reference to the flowcharts of Figures 4 to 7. The methods may be performed to classify images into categories indicative of the sound presets P1 to PM described above. The methods may be performed by the XR system 100.

[0028] 4 is a flow chart of an example method 400 for using ML classification to directly classify images of real-world environments to a "best match" acoustic preset of acoustic parameters for an AES (e.g., AES 126). Method 400 summarizes the work described above in connection with method 300. Method 400 assumes an ML classifier (e.g., ML classifier 120) that has been trained using many images of different real-world environments that have been labeled with various of the acoustic presets P1-PM, such that as a result of the training, the ML classifier is configured to directly classify images to acoustic presets (i.e., to the acoustic parameters of the acoustic preset) without any intervening classification or work.

[0029] At 402, an initial sound preset is established among the sound presets P1 to PM.

[0030] At 404, an image of a scene of a real-world environment is captured.

[0031] At 406, deep learning neural network inference is used to directly classify the image (received from 404) into M classifications indicating acoustic presets P1-PM and their respective confidence levels C1-CN. The acoustic preset among the acoustic presets P1-PM associated with the highest confidence level among the confidence levels C1-CN is considered the "best match" acoustic preset to the real-world environment depicted in the image. That is, the simulated sound response generated by the AES 126 based on the best match acoustic preset is closer to the actual sound response of the real-world environment than would be generated based on any of the other acoustic presets. At 408, the best match acoustic preset may be identified / selected based on the confidence level associated with the classification / acoustic preset.

[0032] At 408, it is determined whether to update the AES 126 with the best match acoustic preset, e.g., as described above in connection with Figure 3. If it is determined to update the AES 126, the best match acoustic preset is provided to the AES, thus replacing the previous acoustic preset. Otherwise, the AES 126 is not updated with the best match acoustic preset, and the AES uses the previous best match acoustic preset (i.e., the previous acoustic preset is not replaced).

[0033] From 408, flow control returns to 404 and the process repeats for the next image.

[0034] 5 is a flow chart of an exemplary method for using ML classification to classify images of real-world environments into room types from which acoustic presets for acoustic environment simulation may then be derived. Method 500 assumes an ML classifier trained with images of different real-world environments labeled with room types (e.g., kitchen, bathroom, living room, etc.), such that, as a result of the training, the ML classifier is configured to classify images into room types. Method 500 also assumes that each one of the acoustic presets P1-PM may correspond to or be derived from a room type resulting from the above classification.

[0035] In 502, an initial sound preset is established among the sound presets P1 through PM.

[0036] At 504, an image of a scene of a real-world environment is captured.

[0037] At 506, deep learning neural network inference is used to classify the image (received from operation 504) into a room type, e.g., a kitchen.

[0038] In 508, an acoustic preset is obtained from among the acoustic presets P1 to PM associated / corresponding to the room type.

[0039] At 510, the AES may be updated with the acoustic presets from 508.

[0040] From 510, flow control returns to 504 and the process repeats for the next image.

[0041] In the method 500, the inference operation 506 does not directly classify into acoustic presets, so after classification is performed, a special operation 508 is used to identify the acoustic preset, i.e., the room type is translated into an acoustic preset.

[0042] 6 is a flowchart of an exemplary method for classifying images of real-world environments based on a cache of scenes of real-world environments and their associated acoustic presets directly into acoustic presets of acoustic parameters for AES 126 using ML classification. Method 600 is similar to method 400, except that method 600 includes additional operations 602 and 604, described below. Method 600 assumes that the XR processor 108 determines which real-world environments, e.g., rooms, the user has been in and records in the cache the (best) acoustic prefix for each of the rooms.

[0043] Flow proceeds from 402 and 404 to 602. At 602, the user determines whether the user has previously been in the room in which the user is currently located. If the user has previously been in the room, flow proceeds to 604 where an acoustic prefix for the room is obtained from a cache. From 604, flow proceeds to 408 where 408 uses the acoustic prefix obtained from the cache. If the user has not previously been in the room, flow proceeds to 406 and operation continues as described above. An example of an XR processor configured to perform method 600 is described below in connection with FIG. 16.

[0044] 7 is a flow chart of an exemplary method for directly classifying images of real-world environments into general / primary acoustic presets and secondary acoustic modifiers for AES 126 using ML classification. More specifically, method 700 uses a first neural network of an ML classifier (e.g., ML classifier 120) trained to directly classify images of real-world environments into general acoustic presets (also referred to as "primary acoustic presets"), as in method 400. Each of the general acoustic presets includes a respective set of general acoustic parameters. For example, the general acoustic parameters can be reverberation parameters. Method 700 also uses a second neural network of an ML classifier trained to further classify images into additional or secondary acoustic parameters, such as absorption and / or reflection parameters or coefficients, room volumes, etc., that can be used to modify the general acoustic presets.

[0045] In 702, an initial sound preset is established among the sound presets P1 through PM.

[0046] At 704, an image of a scene of a real-world environment is captured.

[0047] At 706, the first neural network is used to directly classify the image into generic acoustic presets and then select the best generic acoustic preset, i.e., the acoustic preset associated with the highest confidence level is selected as the best acoustic preset.

[0048] At 708, a second neural network is used to classify the images directly into secondary acoustic parameters.

[0049] At 710, one or more of the general acoustic parameters of the general acoustic preset selected at 706 are modified / adjusted based on one or more of the secondary acoustic parameters to generate a modified general acoustic preset. For example, the value of the general acoustic parameter of the general acoustic preset may be increased or decreased based on the value of the secondary acoustic parameter. Alternatively, one or more of the general acoustic parameters may be replaced by one or more of the secondary acoustic parameters.

[0050] In a simple example, an absorption coefficient α in the decimal range 0<α<1 may be used as the secondary acoustic parameter, in which case operation 710 may multiply one or more of the general acoustic parameters by the absorption coefficient α to generate one or more modified general acoustic parameters. In practice, such modification based on absorption may be more complicated for the following reasons: Since each material has its own absorption coefficient, early reflections from a material are typically directly affected by the absorption coefficient of the material. Therefore, reverberation in an acoustic environment that includes many different materials may be affected by the sum of the materials in the environment, which collectively produce a total absorption. The total absorption may affect the delay rate of the reverberation differently in different frequency bands, which may be taken into account in operation 710.

[0051] At 712, the AES may be updated with the modified general audio preset.

[0052] From 712, flow returns to 704 and the process repeats.

[0053] 8, there is a flowchart of an example method 800 that is similar to method 700, except that method 800 includes a 3D mesh processing operation 802 linked with operation 708 of method 700. Only the 3D mesh processing operation 802 will be described. In general, the 3D mesh processing operation 802 maps one or more of the secondary acoustic parameters generated in operation 708 to components of a 3D mesh generated, for example, using a depth camera.

[0054] At 804, a depth camera captures a depth map (image) of the same real-world environment whose image was captured in operation 704.

[0055] At 806, a 3D mesh is created from the depth map.

[0056] At 808, the secondary acoustic parameters (eg, material sound absorption) generated in operation 708 are associated with a 3D mesh.

[0057] At 810, the 3D mesh and secondary acoustic parameters are exported.

[0058] The training and real-time operation of the ML classifier 120 will now be described in further detail with respect to FIGS. 9-11B.

[0059] ML training 9A is a diagram of an example method of training an ML classifier 120 based on training images according to a first training scenario and, once trained, using the ML classifier to classify images. Once trained, the ML classifier 120 classifies images, typically in real time, in what is referred to as an "inference" stage or operation. In the example of FIG. 9A, the ML classifier 120 is constructed using a CNN. For example, the ML classifier 120 includes a convolutional layer 902 coupled to a fully connected layer 904. In practice, the ML classifier 120 may include many convolutional layers leading to a fully connected layer.

[0060] For training, and after training, for the inference phase, the ML classifier 120 receives images 906 and generates classifications 908 in the form of labels representing acoustic presets. In the inference phase, it selects 910 the acoustic preset with the highest confidence based on the labels and their confidence levels, as described above. During training, the images 906 represent training images on which the ML classifier 120 trains.

[0061] In a first training scenario, training the ML classifier 120 may include the following operations. Establish M distinct labels L1 through LM, for example labels 0 through 20. The labels can be numeric, alphanumeric, human readable labels, etc. b. Create / design M acoustic presets P1-PM representing each typical user environment, e.g., a room. Each preset includes a respective set of acoustic presets AP1-APN as described above. Each acoustic preset Pi is associated with a corresponding label Li. c. Perform pre-training data preparation and label a number of training images (e.g., training pictures) of the room with their most likely acoustic presets among the acoustic presets P1-PM. FIG. 9B is an illustration of pre-training data preparation, which is described below. i. Divide the training pictures into M groups G1-GM of training pictures (individually designated "I" in FIG. 9B), such that the scenes (rooms) in the training pictures in each group Gi have similar acoustic characteristics to the scenes (rooms) in that group, but different acoustic characteristics from the scenes (rooms) in the other groups. For example, the scenes in the training pictures of group G1 will be perceived as having similar acoustic characteristics to each other, the scenes in the training pictures of group G2 will be perceived as having similar acoustic characteristics to each other, but different acoustic characteristics from the scenes in the training pictures of group G1, and so on for all M groups G1-GN of training pictures. ii. Label the groups of training pictures with each of the most likely acoustic presets among the acoustic presets P1-PM. The most likely acoustic preset is the acoustic preset that is most likely to match the acoustic characteristics of the scene in the training pictures. That is, the training pictures are labeled with each of the acoustic presets that are the same within each group, but different between the groups. For example, assume that acoustic preset P1 is the best match to the training pictures of group G1, acoustic preset P2 is the best match to the training pictures of group G2, etc. Then, label all of the training pictures of group G1 with the label L1 for acoustic preset P1, label all of the training pictures of group G2 with the label L2 for acoustic preset P2, etc. This operation produces labeled training pictures, i.e., training pictures labeled with their most likely acoustic presets. d. Using the labeled training pictures, train the ML classifier 120 to directly classify images to the labels / acoustic presets, i.e., the labeled training pictures are fed to the training input of the ML classifier 120, which then trains using the labeled training pictures. The ML classifier is trained to classify images into soft-decision (e.g., softmax) classifications that associate classifications with confidence levels, as described above.

[0062] Tasks (a)-(c) may be performed based on subjective sound design, i.e., substantially manually, by a sound designer. The sound designer uses his experience of room acoustics to design each sound preset with a respective set of most likely sounding sound parameters for a corresponding one of the scenes depicted in a training picture among many training pictures in the training database. That is, the sound designer designs each respective set of sound parameters to best represent or match the acoustic characteristics of the corresponding scene depicted in one of the training pictures, based on the designer's subjective design experience. For example, the designer selects a first set of reverberation parameters of a first sound preset for a "live" room (e.g., a highly reverberant kitchen), selects a second set of reverberation parameters for a "dead" room (e.g., a fully carpeted bedroom with fabric-covered furniture), selects a third set of reverberation parameters of a third sound preset for a room having intermediate reverberation characteristics between those of a "reverberant" room and a "non-reverberant" room, and so on. The designer then labels the training pictures with their most likely acoustic preset (each representing a respective set of acoustic parameters). For example, the designer might label training pictures of rooms that appear to be similarly reverberant with a first acoustic preset, training pictures of rooms that appear to be similarly non-reverberant with a second acoustic preset, training pictures of similar rooms that appear to have intermediate reverberation with a third acoustic preset, and so on.

[0063] An alternative to relying primarily on the experience of a sound designer to establish acoustic presets for training is to use actual acoustic measurements of rooms with different reverberation characteristics and then algorithmically derive the acoustic presets from the acoustic measurements. For example, an acoustic impulse response for each room may be measured using any known or later developed technique for measuring the acoustic impulse response of a real-world environment. A set of acoustic parameters for the acoustic presets is then algorithmically derived from the measured acoustic impulse responses, for example, using any known or later developed technique for deriving reverberation parameters from the acoustic impulse responses.

[0064] In one simplified example, the absolute value of the impulse response may be normalized and converted to dB magnitude. The time from the initial pulse where the dB magnitude falls below 60 dB (normalized to 0 dB) is taken as the RT60 decay time (i.e., how long it would take for the sound to decay 60 dB in the room). Using additional frequency domain analysis, such methods may be extended to multi-band analysis of the RT60 time. Similarly, values ​​such as initial spectral energy, time of occurrence, early reflection timing, and density may be observed directly within the impulse response or a windowed section thereof. It will be appreciated that this particular technique is provided by way of example only, and any additional or alternative method of impulse analysis may be used.

[0065] Once trained, the ML classifier 120 may be validated by determining that any room model "sounds as expected."

[0066] For the inference stage, the ML classifier 120 (or logic external to the ML classifier) ​​may be configured to apply a smoothing function to the softmax (output) classification produced by the ML classifier, such that the classification transitions from its previous state (i.e., the previous acoustic preset provided to AES 126) only if the softmax classification exceeds a softmax threshold, with some built-in hysteresis to avoid spurious classifications, similar to the thresholding process described above in connection with method 300 of Figure 3. For example, the acoustic preset may transition smoothly using appropriate delay line interpolation and gain crossfading.

[0067] Training may also utilize transfer learning, leveraging a pre-trained neural network that already performs traditional room type classification. This approach freezes the convolutional layers of the pre-trained neural network (on feature extraction) and continues to adapt the fully connected layers (classification) with the labels described above.

[0068] 10 is a diagram of an example workflow for the ML classifier 120, showing training the ML classifier 120 according to a second training scenario and a third training scenario, and once trained, using the ML classifier to classify images. In the inference stage, the example of FIG. 10 is similar to the example of FIG. 9A, except that in 1002, acoustic parameters are updated from the labels output by the classification.

[0069] In a second training scenario, the labels may be based on lower level acoustic parameters, such as reverberation parameters. The reverberation parameters may include, for example, I3DL2 acoustic parameters. First, a sound designer uses his or her experience of room acoustics to design each acoustic preset with a set of most likely sounding acoustic parameters for a corresponding one of the scenes depicted in a training picture among many training pictures in the training database. That is, each respective set of acoustic parameters is designed to best represent or match the acoustic characteristics of the corresponding scene depicted in one of the training pictures. Then, during inference, the acoustic parameters are updated based on the labels, as shown at 1002.

[0070] In a third training scenario, the labels are based on lower level acoustic parameters derived from acoustic measurements of real acoustic properties obtained in the same room depicted in the training images. The acoustic measurements may include, for example, measurements of the room (sound) impulse response. The preparation of the pre-training data then involves analyzing the room impulse response and automatically adjusting the appropriate acoustic parameters, i.e., performing auto-tuning. The auto-tuning itself may be based on an ML neural network.

[0071] Both the second and third training scenarios may utilize ML neural networks.

[0072] FIG. 11A is a diagram of an example workflow for an ML classifier, showing training an ML classifier 120 according to a fourth training scenario and, once trained, using the ML classifier to classify images. The example of FIG. 11A is similar to the examples of FIG. 9A and FIG. 10, except for the following differences. In the example of FIG. 11A, the ML classifier 120 includes a long short-term memory (LSTM) 1102 followed by a convolutional layer 902. A neural network based on the LSTM 1102 is suitable for image descriptors. In the flow of FIG. 11A, the LSTM 1102 classifies the acoustic descriptors 1104. An operation 1106 translates the acoustic descriptors 1104 into reverberation parameters that are used in an update operation 1106.

[0073] In a fourth training scenario, the ML classifier 120 is trained using descriptive features of pictures that have acoustic relevance. Preparing the data for pre-training involves labeling pictures of room scenes with a given acoustic vocabulary. The example of FIG. 11A includes an LSTM 1102, but the LSTM may be replaced by a fully connected layer (e.g., fully connected layer 904) if the focus is on the most likely combinations of labels (e.g., loud, reverberant, tile, etc.). Training involves a large number of training pictures labeled with acoustic-specific descriptors.

[0074] Referring to FIG. 11B, there is a diagram of an example of an image sensor 102 (e.g., a camera) including back-to-back 180° fisheye lenses that jointly capture a 360° image of a room in two camera views (images) I180-1 and I180-2, i.e., capture the entire room scene. In this example, the XR processor 108 may stitch the different camera views into a single rectangular image frame using any known or later developed equirectangular projection. The XR processor 108 provides the single rectangular image frame to the ML classifier 120 for processing during training or during the inference / real-time phase. In another example, a traditional rectangular image may be used to train the ML classifier 120. Alternatively, the traditional rectangular image may be stitched into a larger image, e.g., a composite image, based on tracing the room as the user / observer points the camera to different views of the room. The resulting stitched image may be applied to the ML classifier 120 even when the ML classifier was trained using a 360 image. In another example, the image sensor 102 captures a rectangular image of a room, the XR processor 108 maps the rectangular image to an area on an equirectangular space to generate a mapped image, and the ML classifier 120 classifies the mapped image. Any known or later developed technique for mapping a rectangular image to an equirectangular space may be used.

[0075] Example of XR processor FIG. 12 is a block diagram of an embodiment of an XR processor 108 including additional functionality compared to that provided by the embodiment of the XR processor of FIG. 1A. In the example of FIG. 12, the XR processor 108 includes an image classification path 1202, a material estimation path 1204, acoustic parameter consolidation (APC) logic 1206, and an IAE 124. The image classification path 1202 processes the image 103 from the image sensor 102, generates a general / primary acoustic preset 1210 and a secondary acoustic modifier 1212, and provides the general acoustic preset and the secondary acoustic modifier to the APC logic 1206. The material estimation path 1204 processes the image 103 from the image sensor 102 in parallel with the image classification path 1202, generates early reflection model data (ERE) (also referred to as "early reflection parameters") 1214, and provides the early reflection model data to the APC logic 1206. The APC logic 1206 jointly processes the general acoustic presets 1210 , the secondary acoustic modifiers 1212 , and the early reflection model data 1214 to generate final acoustic adjustment parameters 1220 and provide them to the AES 126 of the IAE 124 .

[0076] The image classification path 1202 includes an image pre-processor 1222 (for acoustic analysis), followed by the ML classifier 120. The image pre-processor 1222 processes the images 103, i.e., raw image data, and generates images in a format suitable for consumption by the ML classifier 120. The image pre-processor 1222 formats the raw image data and / or selects, retrieves, or sums the raw image data to match training assumptions for the ML classifier 120. For example, the image pre-processor 1222 may stitch together successive ones of the images 103 and generate a stitched image for classification, as described above.

[0077] Assuming the ML classifier 120 has been trained to directly classify images into both the general acoustic presets (with their confidence levels) and the secondary acoustic modifiers, the ML classifier directly classifies each of the images from the image pre-processor 1222 into a general acoustic preset 1210 and an acoustic modifier 1212. In one example, the general acoustic preset 1210 may include initial reverberation parameters, and the secondary acoustic modifiers 1212 may include one or more of acoustic absorption parameters, acoustic reflection parameters, acoustic diffuseness parameters, and specific environment (e.g., room) dimensions.

[0078] Provided there is sufficient image information, and sufficient ML classifier (e.g., neural network) processing power, for both types of classification to proceed simultaneously, the ML classifier 120 may simultaneously generate the general acoustic preset 1210 and the secondary acoustic modifiers 1212. Alternatively, the ML classifier 120 may (i) initially generate only the general acoustic preset 1210 based on the first received image and / or initially constrained processing power, and (ii) simultaneously generate both the general acoustic preset 1219 and the secondary acoustic modifiers 1212 as more images arrive and / or more processing power becomes available.

[0079] The APC logic 1206 modifies the (initial) reverberation parameters of the general acoustic preset 1210 based on the acoustic modifiers 1212, generates a modified general acoustic preset including the modified reverberation parameters, and provides the modified general acoustic preset to the AES 126 in the final acoustic adjustment parameters 1220.

[0080] The material estimation path 1204 includes an image pre-processor 1232 (for geometric analysis), followed by an architectural mesh and material estimator (simply referred to as the "material estimator") 1234. The image pre-processor 1232 processes the raw image data in the image 103 and generates an image for consumption by the material estimator 1234. The material estimator 1234 builds a (digital) architectural 3D mesh for the scene depicted in the image, estimates the types of materials depicted in the scene based on the architectural 3D mesh, estimates the acoustic properties of the materials, and generates early reflection model data (e.g., parameters) 1214 including the acoustic properties. The image pre-processor 1232 and material estimator 1234 may use any known or later developed technique to perform the geometric image analysis, generate the architectural mesh, and estimate the material properties from the mesh.

[0081] The APC logic 1206 combines the early reflection model data 1214 with the modified generic acoustic preset to produce final acoustic adjustment parameters 1220. Alternatively and / or additionally, the APC logic 1206 may use various parameters in the early reflection model data 1214 to further modify the modified generic acoustic preset.

[0082] In an embodiment in which the material estimation path 1204 is omitted, the early reflectance model data 1214 is still used but may be set to a default value, for example.

[0083] In FIG. 13, there is a block diagram of a portion of the IAE 124 that is used to perform the AES 126 based on the acoustic parameters of the final acoustic adjustment parameters 1220. The IAE 124 includes sound channels 1300(1)-1300(O) that receive respective sound signals S1-SO from respective ones of the virtual sound objects 128. Each channel 1300(i) provides a respective direct path for the sound signal S i to a multi-channel output bus 1302 through a series of adjustable gain (G), delay, and panoramic potentiometer (pan) stages. The resulting direct path sound signals per channel are mixed to the multi-channel output bus 1302. Each channel 1300(i) also provides a respective reflection path for the sound signal S i to the multi-channel output bus through an adjustable reflection stage (refl) that controls the reflection in response to a reflection control signal 1310. The reflection control signal 1310 may include one or more acoustic parameters of the final acoustic adjustment 1220, as described above. The resulting per-channel reflections are also mixed into the multi-channel output bus 1302. The IAE 124 also includes a reverberation generator (rev) 1312 configured to reverberate the combined direct path sound signal and reflections fed by the per-channel reflections and combined on the multi-channel output bus 1302 in response to a reverberation control signal 1314. The reverberation control signal 1314 may include acoustic parameters (e.g., reverberation parameters) of the final acoustic adjustment parameters 1220.

[0084] FIG. 14 is a diagram of an exemplary acoustic parameter refinement process 1400 that may be performed, for example, by the XR processor 108 of FIG. 12. At 1402, initial image data flows into the ML classifier 120, which directly classifies the initial image data and generates soft-decision labels for generic acoustic presets P1-PM (e.g., generic acoustic preset 1210). Each acoustic preset Pi includes a respective set of N acoustic parameters AP1-APN (Param 1-Param N). Select one of the generic acoustic presets P1-PM based on a confidence level to generate a selected generic acoustic preset. Early reflection model data (ERE) default parameters (e.g., default values ​​for early reflection model data 1214) may be added to the selected generic acoustic preset.

[0085] At 1404, further image data flows into the ML classifier 120 and based on the further image data, the ML classifier generates a secondary acoustic modifier (eg, a secondary acoustic modifier) ​​1212 in addition to the general acoustic presets P1-PM.

[0086] At 1406, the acoustic parameter safety check logic performs an acoustic parameter safety check on the selected general acoustic preset and the secondary acoustic modifiers to ensure that the above-mentioned acoustic parameters are within reasonable ranges, given the (current) selected general acoustic preset and additional information useful for performing the safety check. Following the safety check, the APC logic 1206 modifies the selected general acoustic preset based on the secondary acoustic modifiers to generate a modified / aggregated acoustic preset that includes the N acoustic parameters as modified. The ERE default parameters are retained along with the modified / aggregated acoustic preset.

[0087] At 1410, the material estimation pathway 1204 generates early reflectance model data 1214 based on the initial image data and the further image data.

[0088] At 1412, the acoustic parameter safety check logic performs an acoustic parameter safety check on the modified / aggregated acoustic preset and the early reflection model data 1214. The APC logic 1206 further modifies the modified / aggregated acoustic preset based on the early reflection model data 1214 or simply adds the early reflection data to the modified preset to generate final acoustic tuning parameters 1220.

[0089] FIG. 15 is a table of exemplary acoustic parameters generated by the XR processor 108 of FIG. 12 for a room image. The table maps the general acoustic parameters of the general acoustic preset in the first row of the table, the secondary acoustic parameters in the second row of the table, and the early reflection data parameters in the third row of the table to various characteristics of the above-mentioned parameters shown in the columns of the table. The first column of the table identifies the three types of acoustic parameters mentioned above. The next or middle two columns contain information associated with the (descriptive) labels of the tags generated by the ML classifier 120 and the material estimation path 1204 as indicated in the columns. The last column provides examples of reverberation and ERE parameters.

[0090] Cache embodiment FIG. 16 is a block diagram of the XR processor 108 in an embodiment using cached acoustic presets associated with known real-world environments as described above in connection with the method 600 of FIG. 6. During the calibration operation, the image pre-processor 1601 and the ML classifier 120 jointly operate to store acoustic parameters corresponding to previous "scenes" in a calibrated preset memory or cache 1602. After calibration, in real time, the image pre-processor 1601 (performing image matching) together with logic 1604 (which may include the ML classifier 120) identifies acoustic parameters from known real-time images and retrieves the identified acoustic parameters without complete image analysis as described above. In the embodiment of FIG. 16, machine learning focuses on the easier task of matching a room to one of the pre-calibrated scenes.

[0091] 16 embodiment, the APC logic 1206 may be omitted when the calibrated preset memory or cache 1602 stores general acoustic presets, secondary acoustic parameters, and early reflection model data, and when such data has already undergone acoustic parameter safety checks. The IAE 124 finalizes the reflections based on the position information for the virtual sound objects 128 and the position information 114.

[0092] Flowchart for Acoustic Preset Migration Method 17-19 respectively relate to a method for transitioning between acoustic presets based on confidence levels of softmax classification, a method for adding enabled image safety checks to the transition, and a method for performing transitions between acoustic presets in a calibrated scene embodiment. In general, the methods of Figures 17-19 prevent the general / primary acoustic preset and secondary acoustic modifier classifiers from being forced to make guesses about images or real-world scenes that do not have meaningful / recognizable features.

[0093] 17 is a flow chart of an example method 1700 of transitioning between acoustic presets, e.g., reverberation presets. In a loop including 1702 and 1704, the method 1700 establishes a default acoustic preset for the IAE 124 and checks for a likely acoustic environment change. If there is a likely acoustic environment change, flow proceeds to 1706 to update the acoustic preset, from which flow proceeds to 1708. Otherwise, flow returns to the loop. In a loop including 1708, 1710, and 1712, the method 1700 receives an environment-based acoustic preset at 1708 and checks for a likely acoustic environment change at 1710. If there is a likely acoustic environment change, flow proceeds to 1712 to update the acoustic preset. Otherwise, flow returns to 1708.

[0094] 18 is a flow chart of an example method 1800 of transitioning between audio presets and performing a usable image safety check. Method 1800 is similar to method 1700, except that method 1800 further includes operations 1802 and 1804 that validate the current sensor data (i.e., image) as a usable image. For example, operation 1802, inserted between operations 1702 and 1704, validates the current sensor data as a usable image. If yes, flow proceeds from 1702 to 1704, while if no, flow returns to 1702. Similarly, operation 1804 is inserted between operations 1708 and 1710, conditioning the flow between those operations similar to how operation 1804 conditions the flow between operations 1702 and 1704.

[0095] FIG. 19 is a flow chart of an exemplary method 1900 of transitioning between sound presets in a calibrated scene embodiment, such as the embodiment shown in FIG. 16. In a loop including 1902 and 1904, method 1900 establishes a default sound preset and checks the current scene against a scene associated with a calibrated scene. If a calibrated scene is not found, flow returns to 1902. Otherwise, when a calibrated scene is found, flow proceeds to 1906, where method 1900 updates the default sound preset with the calibrated sound preset. Flow proceeds from 1906 to 1908, 1910, and 1912, which repeat the tasks performed in 1902, 1904, and 1906, but starting with the updated calibrated sound preset from 1906 instead of the default sound preset.

[0096] Training process FIG. 20 is a block diagram illustrating an example training process 2000 used for deep neural network (DNN) training of the ML classifier 120 when it is initially untrained, which represents the training method described above in connection with FIGS. 9, 10, and 11A. In 2002, the image pre-processor 1222 formats training images of the real-world environment from the image sensor 102 for consumption by the untrained ML classifier 120 and provides the (formatted) training images to the training input of the ML classifier. In 2004, the image pre-processor function presents the training images to a user / human operator on a display in a user-friendly, easy-to-view format. In 2006, the user applies labels / acoustic adjustment parameters corresponding to acoustic presets, secondary parameters, etc. to each of the training images such that the acoustic parameters match the real-world environment / scene in the training images. The user associates labels / acoustic adjustment parameters to each of the training images input to the untrained ML classifier 120, thereby causing the training images to correlate with their labels. At 2010, one or more neural networks of ML classifier 120 are trained using the labeled training images. The training at 2010 updates the filter coefficients of the neural network of ML classifier 120 based on features in the training images, resulting in a trained version of ML classifier 120. The training process thus configures one or more neural networks of ML classifier 120 to directly classify images to their corresponding acoustic presets.

[0097] Computer Devices Referring to FIG. 21, there is a block diagram of an exemplary computing device 2100 in which the XR processor 108 and the IAE 124 may be implemented. There are numerous possible configurations for the device 2100, and FIG. 21 is intended to be one example. Examples of the device 2100 include tablet computers, personal computers, laptop computers, mobile phones such as smartphones, and the like. The device 2100 may include an output 2104 for driving a display and headphones. The device 2100 may also include one or more network interface units (NIUs) 2108, and memory 2114, each coupled to the processor 2116. The one or more NIUs 2108 may include wired and / or wireless connection capabilities that enable the processor 2116 to communicate over a communication network. For example, the NIU 2108 may include an Ethernet card for communicating over an Ethernet connection, a wireless RF transceiver for wirelessly communicating with a cellular network in the communication network, an optical transceiver, and the like, as would be understood by one of ordinary skill in the art.

[0098] The processor 2116 may include, for example, a collection of microcontrollers and / or microprocessors, each configured to execute respective software instructions stored in the memory 2114. The processor 2116 may be implemented in one or more programmable application specific integrated circuits (ASICs), firmware, or a combination thereof. Portions of the memory 2114 (and the instructions therein) may be integrated with the processor 2116. As used herein, the terms "acoustic," "audio," and "sound" are synonymous and interchangeable.

[0099] The memory 2114 may include read only memory (ROM), random access memory (RAM), magnetic disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible (e.g., non-transitory) memory storage device. Thus, generally, the memory 2114 may include one or more computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions that, when executed (by the processor 2116), are operable to perform the operations described herein with respect to the ML classifier 120, the IAE 124, the image preprocessors 1222 and 1232, the APC logic 1206, the material estimation path 1204, and the methods described above.

[0100] Additionally, the memory 2114 stores data / information 2122 used and generated by the logic 2120, such as images, acoustic parameters, neural networks, and the like.

[0101] Summary method flow chart Referring to FIG. 22, there is a flow chart of an example method 2200 for classifying images that includes the various tasks described above.

[0102] At 2202, the method receives an image of the real-world environment. To do this, the method may capture the image with an image sensor or access the image from a file or a pre-stored image.

[0103] In 2204, the method receives the images captured in operation 2202 using an ML classifier already or previously trained as described herein, and classifies the images directly into classifications that are associated with and point to (known) acoustic presets for the AES. The classifications include respective confidence levels. The acoustic presets each include (known) acoustic parameters that represent sound reverberation for the AES.

[0104] At the time of classification in operation 2204, the acoustic presets and their respective parameters are already known from pre-training of the ML classifier. Therefore, the ML classifier "directly" classifies the images into the classifications associated with and indicating the acoustic presets, without first classifying them into room types, which would then require further work to derive the acoustic parameters from the room types. The direct classification of operation 2204 is essentially a single classification operation that flows from image to classification, providing direct access to the known / predetermined acoustic parameters associated with the classification, without any intervening parameter translation. Furthermore, the AES uses the acoustic presets directly, i.e., as is. In one embodiment, the ML classifier was trained using (labeled) training images of a real-world environment that were divided into different groups of training images. The training images of the different groups of training images are labeled with respective ones of the acoustic presets that are the same within each of the different groups, but different between the different groups. The training images may be further labeled with additional (secondary) acoustic parameters that are utilized in further operations 2210-2214, described below.

[0105] At 2206, the method selects an acoustic preset among the acoustic presets (i.e., a particular one of the acoustic presets) based on the confidence level of the classification. The method accesses / obtains the acoustic preset.

[0106] At 2208, the method performs AES based on the acoustic parameters of the acoustic preset. The AES models sound reverberation for one or more virtual sound objects virtually located in the real-world environment based on the acoustic parameters of the acoustic preset.

[0107] At 2210, the method further classifies the image directly or classifies one or more additional images using a machine learning classifier to generate one or more acoustic parameter modifiers. The further classification may be simultaneous with the classification of operation 2204. Alternatively, the further classification may result from receiving and classifying additional or subsequent images.

[0108] At 2212, the method modifies acoustic parameters of the acoustic preset from 2206 based on the one or more acoustic parameter modifiers from 2210 to generate a modified acoustic preset including modified acoustic parameters for the AES.

[0109] At 2214, the method performs AES using the altered acoustic parameters.

[0110] Different combinations of operations 2202-2206 of method 2200 may represent separate and independent embodiments. For example, operations 2202-2206 collectively represent separate and independent embodiments.

[0111] 23, there is a flow chart of an exemplary method 2300 of classifying a subsequent or second image to the image classified in method 2200. Method 2300 includes various operations as discussed above.

[0112] At 2302, the method captures / receives a second image of the real-world environment.

[0113] At 2304, using a machine learning classifier, the method directly classifies the second image to generate second classifications having respective second confidence levels.

[0114] At 2306, the method determines whether one or more of the second classifications have a respective second confidence level that exceeds a confidence level threshold.

[0115] At 2308, if one or more of the second classifications have a respective second confidence level that exceeds the confidence level threshold, the method selects a second acoustic preset among the acoustic presets (a second particular one of the acoustic presets) based on the second confidence level of the second classification and updates / replaces the acoustic preset with the second acoustic preset for the acoustic environment simulation.

[0116] At 2310, if one or more of the second classifications does not have a corresponding second confidence level that exceeds the confidence level threshold, the method does not select a second acoustic preset and does not update / replace the acoustic preset for the acoustic environment simulation.

[0117] In methods 2200 and 2300, each classification may be based on one image or more than one image. For example, considering the situation of classifying a series of images (or a series of image frames), the method may classify one image at a time and generate a separate classification for each image (or image frame), but the classification preset (i.e., the acoustic preset presented to the AES) changes or updates when there is a significant / significant difference in the "running average" of the confidence levels for the classification from several such images (or image frames). The image being classified may also be augmented with multiple images from the image sensor, for example by stitching together multiple views to generate a less cropped view of the environment.

[0118] In summary, in one embodiment, a method is provided that includes receiving an image of a real-world environment; classifying the image using a machine learning classifier to generate a classification associated with an acoustic preset for an acoustic environment simulation, the acoustic presets each including acoustic parameters that represent sound reverberation; and selecting an acoustic preset from among the acoustic presets based on the classification.

[0119] In another embodiment, an apparatus is provided, the apparatus comprising a processor, the processor being configured to receive an image of a real-world environment, classify the image directly into a classification associated with an acoustic preset for an acoustic environment simulation using a trained machine learning classifier including one or more neural networks, the acoustic presets each including acoustic parameters representing sound reverberation, select an acoustic preset among the acoustic presets based on the classification, and perform the acoustic environment simulation based on the acoustic parameters of the acoustic preset.

[0120] In a further embodiment, a non-transitory computer-readable medium is provided that is encoded with instructions that, when executed by a processor, cause the processor to perform the method presented herein, including receiving an image of a real-world environment, directly classifying the image into a classification associated with an acoustic preset for an acoustic environment simulation using a machine learning classifier previously trained with training images of the real-world environment labeled with a respective one of the acoustic presets, each of the acoustic presets including an acoustic parameter representative of sound reverberation, selecting an acoustic preset from among the acoustic presets based on the classification, and performing the acoustic environment simulation based on the acoustic parameters of the acoustic preset.

[0121] In another embodiment, a system is provided that includes an image sensor for capturing an image of a real-world scene; a processor coupled to the image sensor, the processor configured to implement and use a previously trained machine learning classifier to directly classify the image into a classification associated with an acoustic preset for an acoustic environment simulation, the acoustic presets each including acoustic parameters representing sound reverberation; selecting an acoustic preset among the acoustic presets based on the classification; and performing an acoustic environment simulation based on the acoustic parameters of the acoustic preset to generate a sound signal representing the acoustic environment simulation; and one or more headphones coupled to the processor and configured to convert the sound signal into sound.

[0122] Although the technique has been illustrated and described herein as embodied in one or more specific examples, it is not intended to be limited to the details shown, since various modifications and structural changes may be made within the scope and realm of equivalents of the claims.

[0123] Each claim presented below represents a separate embodiment, and embodiments that combine different claims and / or different embodiments are within the scope of this disclosure and will be apparent to one of ordinary skill in the art after reviewing this disclosure.

Claims

1. 1. A method comprising: Receiving an image of a real world environment; classifying the images using a machine learning classifier to generate classifications associated with acoustic presets for simulating an acoustic environment, the acoustic presets each including acoustic parameters describing sound reverberation; selecting an audio preset from among the audio presets based on the classification; receiving a second image of the real world environment; and classifying the second image using the machine learning classifier to generate second classifications having respective second confidence levels; determining whether one or more of the second classifications have a respective second confidence level that exceeds a confidence level threshold; selecting a second acoustic preset among the acoustic presets based on the second classification if one or more of the second classifications have a respective second confidence level that exceeds the confidence level threshold and replacing the acoustic preset with the second acoustic preset for the acoustic environment simulation; Including, said classifying includes classifying the images to generate classifications, each of the classifications having a respective confidence level; and selecting the acoustic preset such that the acoustic preset is associated with a classification among the classifications having the highest of the respective confidence levels; The method, wherein the machine learning classifier was trained using training images of a real-world environment labeled with each of the acoustic presets.

2. The method of claim 1 , further comprising performing the acoustic environment simulation based on the acoustic parameters of the acoustic preset.

3. 3. The method of claim 2, wherein the performing the acoustic environment simulation includes modeling sound reverberation for one or more virtual sound objects virtually placed in the real-world environment based on the acoustic parameters of the sound preset.

4. classifying the image, or one or more further images, using the machine learning classifier to generate one or more acoustic parameter modifiers; modifying the acoustic parameters of the acoustic preset based on the one or more acoustic parameter modifiers to generate a modified acoustic preset including modified acoustic parameters for the acoustic environment simulation; Further comprising: The method of claim 2 , wherein said performing said acoustic environment simulation comprises performing said acoustic environment simulation with said altered acoustic parameters.

5. The method of claim 1 , wherein the acoustic parameters include one or more of a frequency dependent decay time, a room size, a reflection delay, a time of early onset, and a diffuseness level.

6. The method of claim 1 , wherein the classifying comprises directly classifying the image into the classification and into the acoustic preset.

7. 2. The method of claim 1 , further comprising: not selecting the second acoustic preset and not replacing the acoustic preset for the acoustic environment simulation if one or more of the second classifications do not have a corresponding second confidence level that exceeds the confidence level threshold.

8. The method of claim 1 , wherein the receiving the image of the real world environment comprises receiving the image as a 360 degree image of the real world environment.

9. The method of claim 1 , wherein the image represents a composite of different images.

10. The method of claim 1 , wherein the acoustic parameters of the acoustic preset are algorithmically derived from acoustic impulse responses representing a real-world environment represented in training images on which the machine learning classifier was trained.

11. The method of claim 1 , wherein the acoustic parameters of the sound presets are developed based on a subjective sound design.

12. The method of claim 1 , further comprising performing an audio parameter safety check on the audio presets.

13. Receiving a digital three-dimensional (3D) mesh; estimating acoustic properties of physical materials represented in the 3D mesh to generate early reflection parameters for the acoustic environment simulation; The method of claim 1 further comprising:

14. An apparatus, comprising: a processor, the processor comprising: Receiving an image of a real world environment; - classifying the images directly into classifications associated with acoustic presets for simulating an acoustic environment using a trained machine learning classifier including one or more neural networks, the acoustic presets each including acoustic parameters describing sound reverberation; selecting an audio preset from among the audio presets based on the classification; performing the acoustic environment simulation based on the acoustic parameters of the acoustic preset; receiving a second image of the real world environment; and classifying the second image using the machine learning classifier to generate second classifications having respective second confidence levels; determining whether one or more of the second classifications have a respective second confidence level that exceeds a confidence level threshold; selecting a second acoustic preset among the acoustic presets based on the second classification if one or more of the second classifications have a respective second confidence level that exceeds the confidence level threshold and replacing the acoustic preset with the second acoustic preset for the acoustic environment simulation; configured to: said classifying includes classifying the images to generate classifications, each of the classifications having a respective confidence level; and selecting the acoustic preset such that the acoustic preset is associated with a classification among the classifications having the highest of the respective confidence levels; The apparatus, wherein the machine learning classifier is trained using training images of a real-world environment labeled with each of the acoustic presets.

15. 15. The apparatus of claim 14, wherein the processor is configured to perform the acoustic environment simulation by modeling sound reverberation for one or more virtual sound objects virtually placed in the real-world environment based on the acoustic parameters of the sound preset.

16. The processor, further classifying the image using the machine learning classifier to generate one or more acoustic parameter modifiers; modifying the acoustic parameters of the acoustic preset based on the one or more acoustic parameter modifiers to generate a modified acoustic preset including modified acoustic parameters for the acoustic environment simulation; further performing the acoustic environment simulation using the altered acoustic parameters; and The apparatus of claim 15 , further configured to:

17. the classifications having respective confidence levels; 15. The apparatus of claim 14, wherein the processor is configured to select by selecting the acoustic preset such that the acoustic preset is associated with a classification among the classifications having the highest of the respective confidence levels.

18. A non-transitory computer-readable medium encoded with instructions that, when executed by a processor, cause the processor to: Receiving an image of a real world environment; using a machine learning classifier previously trained with training images of real-world environments labeled with respective ones of the acoustic presets, each of the acoustic presets including acoustic parameters representative of sound reverberation, to directly classify the images into classifications associated with the acoustic presets for simulating an acoustic environment; selecting an audio preset from among the audio presets based on the classification; performing the acoustic environment simulation based on the acoustic parameters of the acoustic preset; receiving a second image of the real world environment; and classifying the second image using the machine learning classifier to generate second classifications having respective second confidence levels; determining whether one or more of the second classifications have a respective second confidence level that exceeds a confidence level threshold; selecting a second sound preset from among the sound presets based on the second classification if one or more of the second classifications have a respective second confidence level that exceeds the confidence level threshold, and replacing the sound preset with the second sound preset for the acoustic environment simulation; said classifying includes classifying the images to generate classifications, each of the classifications having a respective confidence level; and selecting the acoustic preset such that the acoustic preset is associated with a classification among the classifications having the highest of the respective confidence levels; A non-transitory computer-readable medium, wherein the machine learning classifier is trained using training images of real-world environments labeled with each of the acoustic presets.

19. 20. The non-transitory computer-readable medium of claim 18, wherein the instructions for causing the processor to perform the acoustic environment simulation include instructions for causing the processor to model sound reverberation for one or more virtual sound objects virtually placed in the real-world environment based on the acoustic parameters of the sound preset.

20. The processor, further classifying the image using the machine learning classifier to generate one or more acoustic parameter modifiers; modifying the acoustic parameters of the acoustic preset based on the one or more acoustic parameter modifiers to generate a modified acoustic preset including modified acoustic parameters for the acoustic environment simulation; further performing the acoustic environment simulation using the altered acoustic parameters; and 20. The non-transitory computer readable medium of claim 19, further comprising instructions for causing:

21. the classifications having respective confidence levels; 20. The non-transitory computer-readable medium of claim 18, wherein the instructions for causing the processor to select include instructions for causing the processor to select the acoustic preset such that the acoustic preset is associated with a classification among the classifications having the highest of the respective confidence levels.

Images