Calibrating acoustic instrument for physical environment

Abstract

A method includes displaying, on a display, a virtual acoustic instrument overlying on a pass-through of a physical environment. The method includes performing an acoustic simulation based on respective characteristics of the virtual acoustic instrument to generate estimated acoustic parameters for respective locations within the physical environment. The method includes displaying an indication of the estimated acoustic parameter on the display. A method includes measuring an environmental parameter indicative of an organoleptic condition at a location of an electronic device within a physical environment. The method includes determining whether the environmental parameter is within an acceptable range. The method includes, in response to determining that the environmental parameter is not within the acceptable range, triggering presentation of enhancement content to enhance the sensory condition at the location of the electronic device.

Description

Technical Field

[0001] The present disclosure generally relates to calibrating acoustic instruments to a physical environment. Background Art

[0002] Some equipment can be used to calibrate instruments that produce sound. Calibrating an instrument often requires moving the instrument around the environment. Some instruments are heavy and difficult to move. Some instruments are delicate and can be damaged when moved. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0004] Figures 1A to 1N is a diagram of an example operating environment according to some implementations.

[0005] Figure 2 is a diagram of an acoustic configuration system according to some specific implementations.

[0006] Figure 3 is a flowchart representation of a method of configuring an acoustic instrument according to some specific implementations.

[0007] Figure 4 is a block diagram of an apparatus for utilizing a virtual acoustic instrument to configure a corresponding physical acoustic instrument according to some specific implementations.

[0008] FIG. 5A to FIG. 5D is a diagram of another example operating environment according to some implementations.

[0009] Figure 6 is a diagram of an enhanced content presentation system according to some specific implementations.

[0010] Figure 7 is a flowchart representation of a method of presenting enhanced content according to some specific implementations.

[0011] Figure 8 is a block diagram of a device for presenting enhanced content according to some specific implementations.

[0012] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, for clarity, the sizes of the various features may be arbitrarily expanded or reduced. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention

[0013] Various embodiments disclosed herein include devices, systems, and methods for configuring corresponding physical acoustic instruments using virtual acoustic instruments. In various embodiments, a method is performed at an electronic device including a non-volatile memory, one or more processors, a display, and an image sensor. In some embodiments, the method includes displaying the virtual acoustic instrument on a display as an overlay on a physical environment. In some embodiments, the method includes performing an acoustic simulation based on corresponding characteristics of the virtual acoustic instrument to generate estimated acoustic parameters for corresponding positions within the physical environment. In some embodiments, the method includes displaying an indication of the estimated acoustic parameters on the display.

[0014] Various embodiments disclosed herein include devices, systems, and methods for enhancing a portion of a physical environment based on a sensory condition of the portion of the physical environment. In various embodiments, a method is performed at an electronic device including a non-transitory memory, one or more processors, a display, and an image sensor. In some embodiments, the method includes measuring an environmental parameter indicating a sensory condition at a location of the electronic device within a physical environment. In some embodiments, the method includes determining whether the environmental parameter is within an acceptable range. In some embodiments, the method includes triggering the presentation of enhanced content in response to determining that the environmental parameter is not within the acceptable range to enhance the sensory condition at the location of the electronic device.

[0015] According to some specific implementations, the device includes one or more processors, multiple sensors, non-transitory memory, and one or more programs. In some specific implementations, one or more programs are stored in non-transitory memory and executed by one or more processors. In some specific implementations, one or more programs include instructions for executing or causing the execution of any method described herein. According to some specific implementations, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of a device, cause the device to execute or cause the execution of any of the methods described herein. According to some specific implementations, a device includes one or more processors, non-transitory memory, and a device for executing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION

[0016] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.

[0017] Calibrating an acoustic instrument for a live performance prior to a live performance may be difficult. For example, some acoustic instruments may be heavy (e.g., a 200-pound loudspeaker), and it may be difficult to move the heavy acoustic instrument to different locations within a physical environment in order to determine the proper position of the heavy acoustic instrument. Some acoustic instruments may be delicate (e.g., an instrument such as a cello), and it may be difficult to move the delicate acoustic instrument to different locations within a physical environment in order to determine the proper position of the delicate instrument.

[0018] Additionally, calibrating an acoustic instrument without an audience may result in a calibration that does not take into account the audience. For example, when the physical environment has audience members, the audience members may make sounds (e.g., by clapping, talking to each other, singing, screaming, etc.), and the calibration must take into account the sounds that the audience members may make. Furthermore, the auditory signals generated by some of these acoustic instruments may be reflected from the audience members, and calibrating the acoustic instrument without audience members may not take into account the reflections of the auditory signals from the audience members.

[0019] The present disclosure provides methods, systems and / or devices for providing a user interface that allows a user to overlay a virtual acoustic instrument onto a transparent transmission of a physical environment, perform an acoustic simulation based on the virtual acoustic instrument, and view the results of the acoustic simulation in order to correctly calibrate the corresponding physical acoustic instrument. Calibrating a physical instrument based on an acoustic simulation requires fewer adjustments to the calibration of the physical instrument, thereby reducing the number of user inputs corresponding to adjusting the calibration of the physical instrument. Reducing the number of calibration adjustment user inputs often enhances the operability of an electronic device by reducing the utilization of resources (e.g., processing resources, memory resources, and / or power resources) associated with receiving, interpreting, and processing calibration adjustment user inputs. For example, if a user is using a battery-powered device to calibrate a physical instrument, providing fewer calibration adjustment user inputs may extend the battery life of the battery-powered device.

[0020] When presenting a pass-through of the physical environment, the device provides a user interface that allows the user to overlay a virtual representation of an acoustic instrument onto the pass-through of the physical environment. The user interface allows the user to place virtual instruments, virtual microphones, virtual displays, and / or virtual speakers throughout the pass-through of the physical environment. Additionally, the user interface allows the user to overlay virtual people (e.g., virtual audience members and / or virtual performers) onto the pass-through of the physical environment.

[0021] After the user overlays the virtual acoustic instruments and virtual audience members onto the pass-through of the physical environment, the device performs an acoustic simulation to generate estimated acoustic parameters for various locations within the physical environment. The estimated acoustic parameters may include estimated sound levels, estimated frequency responses, estimated sound quality values, etc. at various different locations within the physical environment. Performing the acoustic simulation may include generating an acoustic mesh of the physical environment. The acoustic mesh takes into account the acoustic properties of the physical environment (e.g., the absorption level or reflection level of the material of the physical environment). The acoustic simulation is a function of the corresponding positions of the virtual acoustic instruments, the corresponding positions of the virtual audience members, and the number of virtual audience members.

[0022] The device displays an indication of the estimated acoustic parameters. For example, when the estimated acoustic parameters of the area of ​​the physical environment are within an acceptable range, the device may virtually color the area of ​​the physical environment green, when the estimated acoustic parameters of the area of ​​the physical environment are close to the edge of the acceptable range, the device may virtually color the area of ​​the physical environment yellow, and when the estimated acoustic parameters of the area of ​​the physical environment are outside the acceptable range, the device may virtually color the area of ​​the physical environment red. For example, an area where the estimated sound level is below the acceptable sound level may be shown in red. As another example, an area where the estimated reverberation is greater than the acceptable reverberation may be shown in red.

[0023] The device may provide calibration recommendations to improve the estimated acoustic parameters. The calibration recommendations may include recommended placement locations for some of the acoustic instruments. For example, the calibration recommendations may include recommended placement locations for speakers to increase the sound level in a particular area of ​​the physical environment from an unacceptable sound level to an acceptable sound level. As another example, the calibration recommendation may include reducing the gain of one of the microphones to reduce the estimated reverberation in the area. As another example, the calibration recommendation may include recommended EQ. For example, the calibration recommendation may include a recommendation to recommend EQ processing, such as a recommendation to apply a filter (e.g., a low-pass filter, a high-pass filter, a band-pass filter, etc.) to reduce the effects of interfering frequencies.

[0024] Figure 1A 1 is a diagram illustrating an example physical environment 10 according to some implementations. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the example implementations disclosed herein. To this end, as a non-limiting example, the physical environment 10 includes a stage 12, an audience area 14, and pillars 16 (e.g., stanchions). Figure 1AAlso illustrated are various acoustic instruments 30 that need to be placed in the physical environment 10 for a live performance (e.g., a concert or a show). The acoustic instruments 30 include microphones 40 (e.g., a first microphone 40a, a second microphone 40b, and a third microphone 40c), musical instruments 50 (e.g., a first musical instrument 50a and a second musical instrument 50b), displays 60 (e.g., a first display 60a and a second display 60b), and speakers 70 (e.g., a first speaker 70a and a second speaker 70b). In some implementations, the acoustic instruments 30 may include Figure 1A 1 and 2. The acoustic instrument 30 may be a keyboard-based instrument (such as a piano), a string-based instrument (such as a guitar or a cello), and / or a mouth-based instrument (such as a clarinet, a saxophone, etc.). In some implementations, the acoustic instrument 30 includes a mixing board for setting and / or adjusting settings (e.g., configuration parameter values) of other acoustic instruments 30 (e.g., for setting and adjusting gain values ​​of the microphone 40 and the speaker 70, for setting and adjusting the resolution and text size of the display 60, etc.).

[0025] Figure 1A Also illustrated are an electronic device 100, a user 102 of the electronic device 100, and an acoustic configuration system 200. In various implementations, the user 102 configures the physical environment 10 for a live performance, such as a concert, a show, or a demonstration. Therefore, the user 102 may be an audio engineer who must determine the placement of an acoustic instrument and various settings of the acoustic instrument (e.g., gain values, EQ settings, lighting parameters, etc.). In various implementations, the acoustic configuration system 200 provides a user interface that enables the user 102 to determine the configuration parameters of the acoustic instrument 30 before placing the acoustic instrument 30 in the physical environment 10. In some implementations, the user interface allows the user 102 to overlay a virtual representation of the acoustic instrument 30 onto a transparent representation of the physical environment 10, and perform an acoustic simulation that generates estimated acoustic parameters that guide the user 102 to select the configuration parameters of the acoustic instrument 30 without significant trial and error. In some implementations, the acoustic configuration system 200 resides at the electronic device 100. Alternatively, in some implementations, the acoustic configuration system 200 resides at another device (e.g., a remote device) that communicates with the electronic device 100. In some implementations, a portion of the acoustic configuration system 200 resides at the electronic device 100 (e.g., a user-facing portion that presents a user interface), while the remainder of the acoustic configuration system 200 resides at another device (e.g., a back-end portion that performs acoustic simulations and generates estimated acoustic parameters).

[0026] In some implementations, the physical environment 10 represents an enclosed space, such as a room, a banquet hall, a concert venue, a stadium, etc. In some implementations, the physical environment 10 represents an open space, such as a park, a theater, a home backyard, etc. In some implementations, the electronic device 100 includes a portable electronic device that the user 102 can carry throughout the physical environment 10 to assess the local acoustic conditions at different parts of the physical environment 10. For example, when the user 102 carries the electronic device 100 within the physical environment 10, the acoustic configuration system 200 can display estimated acoustic parameters specific to the current location of the electronic device 100 within the physical environment 10. In some implementations, the electronic device 100 includes a smart phone, a tablet computer, a laptop computer, or a desktop computer. In some implementations, the electronic device 100 includes a head mounted device (HMD) that the user 102 wears around his / her head.

[0027] In some implementations, the electronic device 100 communicates electronically with a mixing board that controls various configuration parameters of the acoustic instrument 30. In such implementations, the electronic device 100 may send a control signal to the mixing board, and the mixing board may set and / or change the configuration parameters of the acoustic instrument 30 based on the control signal. As an example, the acoustic configuration system 200 may determine an EQ process and send an indication of the EQ process to the mixing board, and the mixing board applies the EQ process. In some implementations, the electronic device 100 implements a mixing board. For example, the electronic device 100 displays a graphical user interface corresponding to (e.g., emulating) a mixing board.

[0028] refer to Figure 1B , the electronic device 100 displays an environmental representation 110 of the physical environment 10. In some implementations, the environmental representation 110 includes a transparent representation of the physical environment 10 (e.g., an optical perspective or video perspective of the physical environment 10). In some implementations, the environmental representation 110 includes a representation of Figure 1A The stage 12 shown in FIG. 112 shows the stage 12. Figure 1A The audience area representation 114 of the audience area 14 is shown, and the Figure 1A A pillar representation 116 of the pillar 16 is shown. In some implementations, the electronic device 100 detects a user input 118 corresponding to a request to overlay a virtual instrument corresponding to one of the acoustic instruments 30 on the environmental representation 110. In some implementations, the electronic device 100 displays the environmental representation 110 on a touch screen display, and the user input 118 includes a touch input on the touch screen display.

[0029] refer to Figure 1C In some implementations, the electronic device 100 responds to Figure 1BThe user input 118 is shown to display a menu 120. The menu 120 includes various affordances (e.g., user-selectable options or buttons) that the user 102 can select to trigger the display of a corresponding virtual object. Figure 1C In the example of , the menu 120 includes a virtual microphone enable representation 120a, a virtual musical instrument enable representation 120b, a virtual display enable representation 120c, a virtual speaker enable representation 120d, and a virtual human enable representation 120e. The virtual microphone enable representation 120a, when selected, causes the electronic device 100 to display a virtual microphone corresponding to the microphone 40. The virtual musical instrument enable representation 120b, when selected, causes the electronic device 100 to display a virtual musical instrument corresponding to the musical instrument 50. The virtual display enable representation 120c, when selected, causes the electronic device 100 to display a virtual display corresponding to the display 60. The virtual speaker enable representation 120d, when selected, causes the electronic device 100 to display a virtual speaker corresponding to the speaker 60. The virtual human enable representation 120e, when selected, causes the electronic device 100 to display a virtual human (e.g., a virtual presenter / performer / musician and / or a virtual audience member) that simulates a real person. Figure 1C In the example of FIG. 1 , the electronic device 100 detects user input 122 directed toward a virtual instrument indicator 120 b.

[0030] refer to Figure 1D , in response to detecting Figure 1C The user input 122 shown, the electronic device 100 displays a virtual musical instrument 150. Specifically, the virtual musical instrument 150 includes a first virtual musical instrument 150a representing the first musical instrument 50a. The first virtual musical instrument 150a has the same instrument type as the first musical instrument 50a. For example, if the first musical instrument 50a is an electric guitar, the first virtual musical instrument 150a is a virtual electric guitar representing an electric guitar.

[0031] Figure 1E An environment representation 110 is illustrated with an additional virtual instrument corresponding to the acoustic instrument 30. Figure 1EIn the example of , the environment representation 110 includes a virtual microphone 140 representing a microphone 40 (e.g., a first virtual microphone 140a representing a first microphone 40a, a second virtual microphone 140b representing a second microphone 40b, and a third virtual microphone 140c representing a third microphone 40c). The environment representation 110 includes a virtual musical instrument 150 representing a musical instrument 50 (e.g., a first virtual musical instrument 150a representing a first musical instrument 50a and a second virtual musical instrument 150b representing a second musical instrument 50b). The environment representation 110 includes a virtual display 160 representing a display 60 (e.g., a first virtual display 160a representing a first display 60a and a second virtual display 160b representing a second display 60b). The environment representation 110 includes a virtual speaker 170 representing a speaker 70 (e.g., a first virtual speaker 170a representing a first speaker 70a and a second virtual speaker 170b representing a second speaker 70b). In Figure 1E In the example of FIG. 1 , the environment representation 110 includes virtual performers 124 representing musicians (e.g., a first virtual performer 124a representing a first musician playing a first instrument 50a, a second virtual performer 124b representing a second musician using a second microphone 40b, and a third virtual performer 124c representing a third musician playing a second instrument 50b). The environment representation 110 also includes virtual audience members 126 that simulate real audience members. In some specific implementations, the virtual audience members 126 are programmed to make sounds similar to real audience members (e.g., clapping, screaming, singing along with the second musician, etc.).

[0032] like Figure 1E As can be seen in FIG. 1 , the user interface presented by the electronic device 100 allows the user 102 to visualize the placement of the acoustic instrument 30 without actually placing the acoustic instrument 30 in the physical environment 10. Because some acoustic instruments 30 may be heavy and / or delicate, reducing the need to move the acoustic instrument in order to determine the placement of the acoustic instrument 30 within the physical environment 10 tends to speed up the process of setting up the acoustic instrument 30 in the physical environment 10. In some implementations, the user 102 places each of the virtual objects in the virtual object. Figure 1E Alternatively, in some implementations, the electronic device 100 (eg, the acoustic configuration system 200) automatically determines the placement of the virtual object based on the corresponding characteristics of the virtual object's corresponding real object.

[0033] refer to Figure 1FIn various implementations, the acoustic configuration system 200 obtains characteristic values ​​128 characterizing the acoustic instruments 30 and their corresponding virtual representations displayed within the environment representation 110. In some implementations, the characteristic values ​​128 indicate corresponding target positions (e.g., corresponding desired placement positions) of the acoustic instruments 30. For example, the user 102 may specify where the user 102 wants to place each of the acoustic instruments 30. In some implementations, the characteristic values ​​128 indicate corresponding instrument types of the acoustic instruments 30. For example, the characteristic values ​​128 may indicate that the second microphone 40b is a dynamic microphone, the first musical instrument 50a is an electric guitar, and the speaker 70 is a three-way speaker of a specific size. In some implementations, the characteristic values ​​128 indicate corresponding functions of the acoustic instruments 30. For example, the characteristic values ​​128 may indicate corresponding pickup patterns or corresponding pickup sensitivities of the microphone 40. As another example, the characteristic values ​​128 may indicate corresponding gain values, bass values, or treble values ​​of the speaker 70. As yet another example, characteristic value 128 may indicate a corresponding brightness value, a corresponding resolution, or a corresponding text size of display 60 .

[0034] In some implementations, the characteristic value 128 characterizes the ambient lighting of the physical environment 10. For example, the characteristic value 128 can indicate the intensity and / or color of the ambient lighting. In some implementations, the characteristic value 128 indicates the material of the physical environment 10. For example, the characteristic value 128 can indicate the sound reflectivity or absorption of various materials in the physical environment 10.

[0035] In various implementations, the acoustic configuration system 200 generates an acoustic simulation 130 based on the characteristic values ​​128. The acoustic simulation 130 outputs estimated acoustic parameters 132 for various locations within the physical environment 10. In various implementations, the estimated acoustic parameters 132 indicate how auditory signals generated by various entities in the physical environment 10 might sound at various locations in the physical environment 10. When the acoustic instruments 30 are placed and configured in a manner similar to the corresponding virtual representations of the acoustic instruments 30, the estimated acoustic parameters 132 provide an indication of how music produced by the musicians represented by the virtual performers 124 will sound at various locations within the physical environment 10.

[0036] In some implementations, the estimated acoustic parameters 132 include estimated sound intensity values ​​for various locations within the physical environment 10. For example, the estimated acoustic parameters 132 may include estimated sound amplitude values ​​for various locations within the physical environment 10. In some implementations, the estimated acoustic parameters 132 may indicate an estimated echo level and / or an estimated reverberation level at various different locations within the physical environment 10.

[0037] In various implementations, in addition to setting up the acoustic instrument 30, the user 102 is also responsible for setting up lighting instruments such as focus lights, flashlights, stage lights, etc. In such implementations, the electronic device 100 allows the user 102 to place virtual lighting instruments representing lighting instruments in the entire environment representation 110. In addition, in addition to performing the acoustic simulation 130, the electronic device 100 also performs a lighting simulation that generates estimated lighting parameters. The estimated lighting parameters indicate the estimated lighting levels at various locations within the physical environment 10. For example, the estimated lighting level may indicate the estimated light intensity and / or estimated light color at various different locations within the physical environment 10. The user 102 can use the estimated lighting level to adjust the configuration settings of the lighting instrument. For example, the user 102 can use the estimated lighting level to determine the placement position and / or light intensity of the lighting instrument. In some implementations, in addition to performing the acoustic simulation 130, the acoustic configuration system 200 also performs a lighting simulation. Therefore, in addition to estimating the acoustic parameters 132, the acoustic configuration system 200 can also generate estimated lighting parameters. In some implementations, the lighting simulation is part of the acoustic simulation 130. Thus, estimating acoustic parameters 132 may include estimating lighting parameters.

[0038] In various implementations, the virtual representation of the acoustic instrument 30 has the same or similar configuration settings as the acoustic instrument 30. For example, the virtual microphone 140 has the same or similar pickup pattern and gain settings as the corresponding microphone 40. In this example, the user 102 can adjust the gain settings of the virtual microphone 140 based on the estimated acoustic parameters 132. For example, if the estimated acoustic parameters 132 indicate that the first virtual microphone 140a does not adequately pick up the sound generated by the first virtual musical instrument 150a, the user 102 can increase the gain value of the first virtual microphone 140a and use the increased gain value for the first microphone 40a. Therefore, when the user 102 sets the first microphone 40a in the physical environment 10, the first microphone 40a will be configured with a suitable gain value that allows the first microphone 40a to properly capture the sound generated by the first musical instrument 50a. More generally, in various implementations, user 102 may utilize estimated acoustic parameters 132 to adjust configuration settings of a virtual acoustic instrument, and utilize the adjusted configuration settings of acoustic instrument 30 , rather than using trial and error to determine the configuration settings of acoustic instrument 30 .

[0039] refer to Figure 1G In various implementations, the acoustic configuration system 200 presents the indication 134 of the estimated acoustic parameter 132. Figure 1GIn the example of , the indication 134 includes a loudness indication 134a and reverberation indications 134b and 134c. The loudness indication 134a indicates that, based on the estimated acoustic parameters 132, the sound level in a portion of the physical environment 10 covered by the loudness indication 134a is below a threshold sound level. Figure 1G In the example of , the loudness indication 134a is relatively far from the virtual speaker 170. Therefore, in order to increase the sound level in the portion of the physical environment 10 covered by the loudness indication 134a, it may be necessary to increase the volume setting of the virtual speaker 170. The electronic device 100 may display the loudness indication 134a by coloring the portion of the environment representation 110 (e.g., by overlaying a red mask over the portion of the environment representation 110).

[0040] The reverberation indications 134b and 134c indicate that reverberation exists on the left and right sides of the pillar representation 116. When the estimated acoustic parameter 132 indicates that the estimated reverberation level exceeds the acceptable reverberation level, the reverberation indications 134b and 134c are displayed. The user 102 can adjust the configuration settings of the virtual acoustic instrument to reduce the reverberation on both sides of the pillar representation 116, for example, by performing EQ processing (such as causing the virtual speaker 170 to output an auditory signal with the reverberation removed). The electronic device 100 can display the reverberation indications 134b and 134c by coloring these parts of the environment representation 110 (for example, by overlaying a yellow mask on these parts of the environment representation 110).

[0041] refer to Figure 1H In various implementations, the acoustic configuration system 200 recommends changes to the configuration settings of the virtual acoustic instrument to address the problem of the estimated acoustic parameters 132 being outside of an acceptable range. Figure 1HIn the example of , the electronic device 100 displays a menu 136, which includes current sound configuration parameters 136a, suggested sound configuration parameters 136b, and a suggested change enable indication 136c for changing the current sound configuration parameters 136a to the suggested sound configuration parameters 136b. In some specific implementations, the current sound configuration parameters 136a include the current gain value of the virtual microphone 140, the current tuning parameters of the virtual musical instrument 150 (e.g., the current string tightness of a string-based musical instrument), the current display settings of the virtual display 160 (e.g., the current resolution, text size, etc.), the current speaker settings of the virtual speaker 170 (e.g., the current volume level, bass, treble, etc.), and / or the current light settings of the virtual lighting instrument (e.g., the current brightness, color, pattern, frequency, etc.). In some embodiments, the recommended sound configuration parameters 136b include recommended gain values ​​for the virtual microphone 140, recommended tuning parameters for the virtual musical instrument 150 (e.g., recommended string tightness for a string-based instrument), recommended display settings for the virtual display 160 (e.g., recommended resolution, text size, etc.), recommended speaker settings for the virtual speakers 170 (e.g., recommended volume level, bass, treble, etc.), and / or recommended light settings for the virtual lighting instrument (e.g., recommended brightness, color, pattern, frequency, etc.).

[0042] In some embodiments, user input directed to the make suggested change enable representation 136c causes the electronic device 100 to change the current sound configuration parameters 136a to the suggested sound configuration parameters 136b. For example, selecting the make suggested change enable representation 136c triggers a change in the virtual microphone 140 from a current gain value to a suggested gain value, a change in the virtual musical instrument 150 from current tuning parameters to suggested tuning parameters, a change in the virtual display 160 from current display settings to suggested display settings, a change in the virtual speaker 170 from current speaker settings to suggested speaker settings, and / or a change in the virtual light from current light settings to suggested light settings. In some embodiments, changing the current sound configuration parameters 136a to the suggested sound configuration parameters 136b includes moving some of the virtual instruments from current positions to suggested positions.

[0043] Fig. 1IThe information indicators 180 are illustrated as being overlaid onto various portions of the environment representation 110. For example, the first information indicator 180a is overlaid toward the left portion of the environment representation 110, the second information indicator 180b is overlaid toward the right portion of the environment representation 110, and the third information indicator 180c is overlaid toward the bottom portion of the environment representation 110. When the electronic device 100 detects a user input directed to one of the information indicators 180, the electronic device 100 displays the estimated acoustic parameters 132 associated with the corresponding portion of the physical environment 10. For example, when the electronic device 100 detects a user input directed to the first information indicator 180a, the electronic device 100 displays a portion of the estimated acoustic parameters 132 associated with the left portion of the physical environment 10.

[0044] Fig. 1I Also illustrated is a warning indicator 182 overlaid onto the front portion of the environment representation 110. The warning indicator 182 indicates that at least some of the estimated acoustic parameters 132 corresponding to the front portion of the physical environment 10 are outside of an acceptable range. When the electronic device 100 detects a user input directed to the warning indicator 182, the electronic device 100 may display which of the estimated acoustic parameters 132 are outside of the acceptable range and to what extent. For example, the electronic device 100 may display a warning indicator 182 in response to detecting a user input directed to the warning indicator 182. Figure 1G Loudness indicator 134a is shown.

[0045] Figure 1J A user input 184 pointing to the warning indicator 182 is illustrated. Figure 1K As shown, the electronic device 100 responds to the detection Figure 1J The menu 186 is displayed based on the user input 184 shown. The menu 186 displays an estimate of the loudness level 186a in the front portion of the physical environment 10, the interfering frequency 186b, the suggested EQ processing 186c, and the make change enable indication 186d. In some embodiments, the estimated acoustic parameter 132 indicates the expected frequency interference in the front portion of the physical environment 10. The suggested EQ processing 186c may include applying a filter (e.g., a low pass filter, a high pass filter, a band pass filter) to reduce the effects of the interfering frequency. When the electronic device 100 detects the user input selecting the make change enable indication 186d, the electronic device 100 applies the suggested EQ processing 186c to reduce the effects of the interfering frequency. In some embodiments, the electronic device 100 triggers (e.g., instructs) the mixing board to apply the suggested EQ processing 186c.

[0046] refer to Figure 1LIn some implementations, the acoustic configuration system 200 provides suggestions for replacing some of the virtual instruments in order to improve the estimated acoustic parameters 132. Figure 1L In the example of , the acoustic configuration system 200 displays microphone suggestions 188 to, for example, replace the second virtual microphone 140b with a dynamic microphone to better capture the sound generated by the second virtual performer 124b. In some implementations, replacing some of these virtual instruments with the replacements suggested by the acoustic configuration system 200 results in the generation of modified estimated acoustic parameters 132 that are within an acceptable range.

[0047] refer to Figure 1M In some implementations, the acoustic configuration system 200 provides recommendations to move some of the virtual instruments to different locations within the environment representation 110 in order to improve the estimated acoustic parameters 132. Figure 1M In an example of , the acoustic configuration system 200 displays a display suggestion 190 to move the virtual display 160 closer to the stage or to increase the text size of the text displayed on the virtual display 160, for example so that the virtual performer 124 can better view the text (e.g., song lyrics or demonstration points) displayed on the virtual display 160. In some examples, the acoustic configuration system 200 may recommend moving the virtual microphone 140 to a different location within the environment representation 110 to better capture the sound generated by the virtual performer 124. In other examples, the acoustic configuration system 200 may recommend moving the virtual speaker to a different location within the environment representation 110 so that the sound emitted by the corresponding speaker 70 is fully propagated throughout the physical environment 10.

[0048] refer to Figure 1N In some implementations, the electronic device 100 (eg, the acoustic configuration system 200 or the lighting configuration system) recommends changes to the lighting instrument. Figure 1N In the example of FIG. 1 , the electronic device 100 displays a low light indication 192 to indicate that a portion of the physical environment 10 surrounded by the low light indication 192 is not adequately illuminated. The low light indication 192 includes a light adjustment enable indication 194. When the electronic device 100 detects a user input directed to the light adjustment enable indication 194, the electronic device 100 adjusts the configuration settings of the virtual lighting instrument. For example, the electronic device 10 increases the intensity of the virtual lights that provide light to the portion of the environment representation 110 surrounded by the low light indication 192.

[0049] In various implementations, the acoustic configuration system 200 displays an indication of the estimated acoustic parameters 132 to guide the user 102 to adjust the configuration settings of the various virtual instruments. After the user 102 has finished adjusting the configuration settings for the virtual instruments, the acoustic configuration system 200 may generate a report including all configuration settings selected by the user 102 based on the estimated acoustic parameters 132. The user 102 may use the report to configure the acoustic instrument 30 in the physical environment 10, thereby reducing the need to use trial and error to determine the settings of the acoustic instrument 30. As an example, the report may include the placement positions of the various virtual instruments, and the user 102 may place the corresponding physical instruments at the same placement positions, thereby reducing the need to use trial and error to determine the placement positions of the physical instruments. As another example, the report may include an EQ process that results in the least amount of echo and reverberation, and the user 102 may apply the same EQ process after the physical instrument has been placed in the physical environment 10 without resorting to trial and error.

[0050] Figure 2 is a block diagram of an acoustic configuration system 200 according to some implementations. In various implementations, the acoustic configuration system 200 includes a data acquirer 210, an environment renderer 220, a simulation generator 230, and an instrument configurator 240.

[0051] In various implementations, the data acquirer 210 acquires data related to the physical environment (e.g., Figure 1A In some implementations, the environmental data 212 includes image data 212a. In some implementations, the image data 212a includes an image of the physical environment and an instrument to be placed in the physical environment (e.g., Figure 1A In some implementations, the environment data 212 includes depth data 212b. For example, the electronic device 100 ( Figure 1A 212b of the physical environment 10. In some implementations, the environment data 212 includes a visual grid 212c of the physical environment. In some implementations, the data acquirer 210 generates the visual grid 212c using the image data 212a and / or the depth data 212b. In some implementations, the environment data 212 indicates environment dimensions 212d (e.g., Figure 1A212a, the physical dimensions of the physical environment 10 shown). In some implementations, the data acquirer 210 determines the environmental dimensions 212d based on the image data 212a, the depth data 212b, or the visual grid 212c. In some implementations, the environmental data 212 indicates materials 212e found in the physical environment, acoustic properties 212f of the materials 212e, and / or visual properties 212g of the materials 212e. For example, the environmental data 212 may indicate whether the materials 212e in the physical environment reflect sound / light and to what extent. In some implementations, the data acquirer 210 identifies the material 212e based on the image data 212a and retrieves the acoustic properties 212f and visual properties 212g of the material 212e from a data store storing properties of various materials.

[0052] In some implementations, the data acquirer 210 obtains instrument data 214 that characterizes various instruments to be placed in the physical environment. For example, the instrument data 214 includes Figure 1F to Figure 1N . In some implementations, the instrument data 214 indicates the number of instruments 214a to be placed in the physical environment (e.g., the number of microphones, speakers, displays, musical instruments, and / or lighting instruments). In some implementations, the instrument data 214 indicates an instrument type 214b of the instrument to be placed in the physical environment. As an example, the instrument type 214b may indicate the type of microphone to be placed in the physical environment, the size of the display, the size of the speaker, the type of musical instrument (e.g., acoustic, electric, string-based, keyboard-based, etc.), and the type of lighting instrument (e.g., floodlight, spotlight, LED light, mood light, stage light, etc.). In some implementations, the instrument data 214 indicates an instrument characteristic 214c of the instrument to be placed in the physical environment. As an example, the instrument characteristic 214c indicates the resolution of the display, the power value of the speaker, and / or the intensity / color of the light emitted by the lighting instrument. In some implementations, the instrument data 214 includes a microphone pickup pattern 214d of the microphone (e.g., cardioid, supercardioid, figure-of-eight, omnidirectional, shotgun, etc.). In some implementations, the instrument data 214 indicates an instrument loudness 214e of the instrument. In some implementations, the instrument data 214 indicates display characteristics 214f ​​(e.g., size and resolution) of the display. In some implementations, the instrument data 214 indicates speaker characteristics 214g (e.g., size, power rating, speaker class, frequency response, etc.) of the speaker. In some implementations, the data acquirer 210 determines the instrument data 214 based on the environment data 212. For example, the data acquirer 210 utilizes the image data 212a to identify the brand and model of the instrument and retrieves various characteristics of the instrument from a data store storing information about the instrument.

[0053] In some implementations, the environment renderer 220 renders a pass-through 222 of the physical environment. For example, the environment renderer 220 renders Figures 1B to 1N . In some implementations, the pass-through 222 is an optical see-through of the physical environment, and the environment renderer 220 renders the optical see-through by allowing light from the physical environment to pass through the optical see-through display and into the user's eyes. In some implementations, the pass-through 222 is a video see-through of the physical environment, and the environment renderer 220 renders the video see-through by capturing image data with a camera of the electronic device and displaying the captured image data on an opaque display.

[0054] In various implementations, the environment renderer 220 overlays the virtual instrument 224 on the transparent transmission 222. In some implementations, the virtual instrument 224 includes a virtual microphone, a virtual musical instrument, a virtual display, a virtual speaker, and / or a virtual lighting instrument. Figure 1E The virtual microphone 140, the virtual instrument 150, the virtual display 160, and the virtual speaker 170 are shown. In some implementations, the environment renderer 220 places the virtual instrument 224 based on user input (e.g., in response to receiving Figure 1C Alternatively, in some implementations, the environment renderer 220 automatically overlays the virtual instrument 224. For example, the environment renderer 220 automatically determines the placement of the virtual instrument 224 based on the environment data 212 and the instrument data 214. In some implementations, the environment renderer 220 overlays the virtual person (e.g., Figure 1E , virtual performers 124 and virtual audience members 126 are shown).

[0055] In various implementations, after the environment renderer 220 overlays the virtual instrument 224 onto the passthrough 222, the simulation generator 230 generates the simulation 232. For example, the simulation generator 230 generates Figure 1F to Figure 1N In various implementations, simulation 232 results in generation of estimated parameters 234 (e.g., Figure 1F to Figure 1N ), which provide an indication of how the physical environment might sound or appear when the physical instrument is configured (eg, placed and set up) in a manner similar to the corresponding virtual instrument 224.

[0056] In some implementations, the estimated parameters 234 include estimated loudness values ​​234a for various locations within the physical environment. The estimated loudness values ​​234a indicate how loud various locations within the physical environment might sound when the physical instrument is configured in a manner similar to the virtual instrument 224. As an example, referring to Figure 1G, the estimated loudness value 234a can indicate that a portion of the physical environment located in front of the stage 12 may not be loud enough when the speaker 70 is placed at the location indicated by the virtual speaker 170 and the speaker 70 is configured in a similar manner to the virtual speaker 170 (e.g., has the same settings). In some implementations, the estimated loudness value 234a includes a sound duration value (e.g., how long the sound lingers), a sound frequency (e.g., pitch), and a sound intensity value (e.g., in decibels).

[0057] In some implementations, the estimated parameters 234 include estimated frequency responses 234b for various locations within the physical environment. The frequency responses 234b can indicate locations within the physical environment that have undesirable frequencies (e.g., unacceptable levels of frequency interference). In some implementations, the estimated parameters 234 include estimated echo occurrences 234c, which indicate areas within the physical environment where an amount of echo is expected to be greater than a threshold echo amount. In some implementations, the estimated parameters 234 include estimated reverberation occurrences 234d, which indicate areas within the physical environment where a reverberation level is expected to be greater than a reverberation threshold level. For example, Figure 1G As shown, the acoustic configuration system 200 displays reverberation indications 134b and 134c. In some implementations, the estimated parameters 234 include estimated sound quality values ​​234e for various locations within the physical environment. Fig. 1I , the user 102 can tap the information indicator 180 or the warning indicator 182 to view the estimated sound quality value 234e. In some implementations, the estimated parameters 234 include estimated ambient light values ​​234f for various locations within the physical environment. The estimated ambient light values ​​234f indicate how bright various areas within the physical environment are expected to be when a physical lighting instrument is placed at the location indicated by the virtual lighting instrument and the physical lighting instrument is configured with the same settings as the virtual lighting instrument (e.g., the same brightness level and color). For example, Figure 1N As shown, the electronic device 100 displays a low light indication 192 .

[0058] In various implementations, the simulation generator 230 provides the estimated parameters 234 to the environment renderer 220 and / or the instrument configurator 240. In some implementations, the instrument configurator 240 displays a visual indicator 242 based on the estimated parameters 234. In some implementations, the visual indicator 242 includes a loudness indicator 242a (e.g., Figure 1G In some implementations, the visual indicator 242 includes a frequency interference indicator 242b (e.g., Figure 1KIn some implementations, the visual indicator 242 includes an echo indicator 242c based on the estimated echo occurrence 234c. In some implementations, the visual indicator 242 includes a reverberation indicator 242d based on the estimated reverberation occurrence 234d (e.g., Figure 1G In some implementations, the visual indicator 242 includes a sound quality indicator 242e (e.g., Fig. 1I In some implementations, the visual indicator 242 includes an ambient light indicator 242f based on the estimated ambient light value 234f (e.g., Figure 1N Low light indicator 192 shown).

[0059] In some implementations, the instrument configurator 240 determines a recommended configuration 244 for some of the instruments based on the estimated parameters 234 being outside of the acceptable range. In some implementations, the recommended configuration 244 includes a recommended equipment position 244a. For example, Figure 1M As shown, the electronic device 100 displays a display suggestion 190 to move the virtual display 160 closer to the stage. In some implementations, moving the equipment according to the suggested equipment position 244a often results in a revised estimated parameter within an acceptable range. For example, moving some of the instruments may result in a revised estimated loudness value within an acceptable loudness range.

[0060] In some implementations, the suggested configuration 244 includes a suggested equipment replacement 244b. Figure 1L As shown, the electronic device 100 displays a microphone suggestion 188 to replace the second virtual microphone 140b with a dynamic microphone to better capture the voice of the second virtual actuator 124b. In some implementations, replacing the current equipment with the suggested equipment results in a revised estimated parameter within an acceptable range. For example, switching to a dynamic microphone can result in a revised estimated frequency response within an acceptable frequency response range.

[0061] In some implementations, the suggested configuration 244 includes a suggested gain value 244c for the microphone and / or speaker. In some implementations, changing the current gain value to the suggested gain value 244c often results in a revised estimated parameter that is within an acceptable range. For example, switching to the suggested gain value 244c may result in a revised estimated echo occurrence that is below a threshold number of echo occurrences. In some implementations, the instrument configurator 240 displays an indication that, when selected, triggers the mixer to change the current gain value to the suggested gain value 244c. For example, Figure 1HAs shown, a user selection to make a suggested change indication 136c triggers the electronic device 100 to change the current sound configuration parameters 136a to the suggested sound configuration parameters 136b.

[0062] In some implementations, the suggested configuration 244 includes a suggested EQ process 244d. In some implementations, applying the suggested EQ process 244d results in generating revised estimated parameters within an acceptable range. For example, applying the suggested EQ process 244d can result in generating revised estimated sound quality values ​​within an acceptable sound quality range. In some implementations, the instrument configurator 240 displays an affordance that, when selected, triggers the mixing board to apply the suggested EQ process 244d. For example, Figure 1K As shown, a user selection to make a change indication 186d triggers the electronic device 100 to apply the suggested EQ processing 186c to reduce the effects of the interfering frequency 186b.

[0063] In some implementations, the suggested configuration 244 includes suggested lighting parameters 244e. The suggested lighting parameters 244e may include a suggested intensity, a suggested light color emission setting, and / or a suggested frequency for the lighting instrument. In some implementations, changing the current lighting parameters to the suggested lighting parameters 244e results in a revised estimated ambient light value that is within the acceptable ambient lighting range. In some implementations, the instrument configurator 240 displays an affordance that, when selected, triggers a change from the current lighting parameters to the suggested lighting parameters 244e. For example, Figure 1N As shown, the light adjustment indicator 194 indicates that a user selection triggers the electronic device 100 to adjust the lighting so that the rear area of ​​the physical environment 10 is adequately illuminated.

[0064] In various implementations, the instrument configurator 240 provides the suggested configuration 244 to the environment renderer 220, and the environment renderer 220 displays the suggested configuration 244 as an overlay on the passthrough 222 (eg, Figure 1H The menu shown is 136 or Figure 1K In some implementations, the instrument configurator 240 causes the environment presenter 220 to display a user-selectable affordance that, when selected, triggers the electronic device 100 ( Figures 1A to 1N ) or the mixer board to change the current configuration to the suggested configuration 244 (e.g., Figure 1H The suggested changes shown indicate 136c or Figure 1K The suggested changes making indicator 186d is shown. In some implementations, the instrument configurator 240 provides the suggested configuration 244 to the mixing board by causing a change in the appearance of some of the mixing board's physical controls (e.g., by flashing a gain knob when the suggested configuration 244 requires a gain adjustment).

[0065] Figure 3 is a flowchart representation of a method 300 for configuring a physical instrument based on a simulation of a corresponding virtual instrument. In various specific implementations, the method 300 comprises Figures 1A to 1N The electronic device 100 shown in FIG. 1 and / or Figure 1A to Figure 2 The method 300 is performed by the acoustic configuration system 200 shown in . In some specific implementations, the method 300 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the method 300 is performed by a processor executing a code stored in a non-transitory computer readable medium (e.g., a memory).

[0066] As represented by block 310, in various implementations, method 300 includes displaying the virtual acoustic instrument on a display as an overlay onto a pass-through of the physical environment. Figure 1E As shown, the electronic device 100 overlays the virtual instruments 140, 150, 160, and 170 onto the environment representation 110. As represented by block 310a, in some implementations, displaying the virtual acoustic instrument includes receiving user input indicating a corresponding placement position of the virtual acoustic instrument. Figure 1C As shown, the electronic device 100 detects a user input 122 corresponding to a request to place a virtual musical instrument on the stage representation 112. As represented by block 310b, in some implementations, displaying the virtual acoustic instrument includes automatically placing the virtual acoustic instrument based on the dimensions of the physical environment. Figure 1E As discussed, in some implementations, the electronic device 100 automatically determines the placement of the virtual instruments 140, 150, 160, and 170, the virtual performer 124, and the virtual audience member 126 based on the layout of the physical environment 10. As represented by block 310c, in some implementations, the virtual acoustic instrument represents a physical acoustic instrument. Figure 1E As shown, the virtual microphone 140 represents the microphone 40 , the virtual musical instrument 150 represents the musical instrument 50 , the virtual display 160 represents the display 60 , and the virtual speaker 170 represents the speaker 70 .

[0067] As represented by block 320, in various implementations, method 300 includes performing an acoustic simulation based on the corresponding characteristics of the virtual acoustic instrument to generate estimated acoustic parameters for the corresponding location within the physical environment. Figure 1F As shown, the acoustic configuration system 200 performs an acoustic simulation 130 based on the characteristic value 128 to generate an estimated acoustic parameter 132. As another example, Figure 2As shown, the simulation generator 230 utilizes the environment data 212 and / or the instrument data 214 to perform a simulation 232, thereby obtaining estimated parameters 234. In various implementations, performing an acoustic simulation allows a user of an electronic device (e.g., an audio engineer) to evaluate the planned configurations (e.g., planned placement and planned settings) of various physical instruments before placing the physical instruments in a physical environment.

[0068] As represented by block 320a, in some implementations, performing an acoustic simulation includes obtaining an acoustic mesh for the physical environment and performing an acoustic simulation based on the acoustic mesh. In some implementations, the electronic device generates an acoustic mesh by modifying a visual mesh based on acoustic properties of materials in the physical environment. The acoustic mesh indicates the acoustic properties of the materials in the physical environment. For example, the acoustic mesh indicates a sound absorption level and a sound reflection level of various parts of the physical environment.

[0069] As represented by block 320b, in some implementations, method 300 includes displaying virtual audience members overlaid onto a pass-through of the physical environment. Figure 1E As shown, the electronic device 100 displays a virtual audience member 126. In some implementations, performing acoustic simulation includes simulating sound absorbed or reflected by the virtual audience member. Figure 1F , the virtual audience members 126 may absorb and reflect the sound generated by the musical instrument to varying degrees. In some implementations, performing the acoustic simulation includes simulating the virtual audience members to emit sound. For example, referring to Figure 1F , virtual audience members 126 are programmed to talk, scream and / or sing along with virtual performers 124.

[0070] As represented by block 320c, in some implementations, method 300 includes measuring actual acoustic parameters when a physical acoustic instrument is placed at a location corresponding to the virtual acoustic instrument, and adjusting the acoustic simulation based on a difference between the actual acoustic parameters and the estimated acoustic parameters. Figure 1F When the acoustic instrument 30 is placed at the position indicated by the virtual instrument, the electronic device 100 measures actual acoustic parameters, and the electronic device 100 updates the acoustic simulation 130 using the difference between the actual acoustic parameters and the estimated acoustic parameters 132 (e.g., to correct / compensate for the difference between the virtual instrument and the physical instrument).

[0071] As represented by block 320d, in some implementations, performing the acoustic simulation includes playing a pre-recorded sound of the musical instrument. In some implementations, the electronic device measures (e.g., estimates) acoustic parameters at various locations within the environment after playing the pre-recorded sound to assess how the pre-recorded sound sounds at various locations within the environment. For example, referring to Figure 1F, the electronic device 100 may determine the estimated acoustic parameters 132 by evaluating the detection of auditory signals at various locations after playing the pre-recorded sound of the musical instrument.

[0072] As represented by block 330, in various implementations, method 300 includes displaying an indication of the estimated acoustic parameter on a display. Figure 1G As shown, the electronic device 100 displays an indication 134 based on the estimated acoustic parameters 132. In various specific implementations, displaying the indication of the estimated acoustic parameters reduces the need to actually measure the acoustic parameters multiple times during the trial-and-error-based setup of the acoustic instrument. For example, displaying the indication of the estimated acoustic parameters helps the user select an initial configuration setting for a physical instrument that requires less adjustment and therefore requires less measurement. Performing fewer actual measurements tends to save power for battery-powered devices and extend the use time of battery-powered devices. In addition, the actual measurement of the acoustic parameters may need to be performed on the device, which may cause excessive heating, particularly on wearable electronic devices such as HMDs, and may cause health hazards to the user, while the acoustic simulation can be offloaded to another device such as a smart phone or a cloud computing platform, where the heating can be controlled without causing health hazards to the user.

[0073] As represented by box 330a, in some embodiments, method 300 includes indicating areas in the physical environment where the estimated acoustic parameters are not within an acceptable range. In some embodiments, the electronic device overlays an augmented reality (AR) mask onto the areas in the physical environment where the estimated acoustic parameters are outside of the acceptable range. The AR mask may include colored masks, for example, a green AR mask for areas within the acceptable range, a yellow AR mask for areas close to an upper or lower limit of the acceptable range (e.g., within a threshold of the upper or lower limit of the acceptable range), and a red mask for areas outside of the acceptable range. As an example, the electronic device 100 Figure 1G A loudness indication 134a is displayed in the image to indicate that the area in front of the stage may not be loud enough.

[0074] As represented by block 330b, in some implementations, method 300 includes recommending changes to configuration values ​​of the virtual acoustic instrument. Figure 1H As shown, the electronic device 100 displays the suggested sound configuration parameters 136b. In some implementations, the method 300 includes performing another acoustic simulation after the configuration value change has been performed. For example, referring to Figure 1H After the user 102 selects to make the suggested change affordance 136c, the acoustic configuration system 200 can update the acoustic simulation 130 to generate a revised version of the estimated acoustic parameters 132. In some implementations, the recommended changes include recommending that some of the virtual acoustic instruments be moved to different locations. For example, Figure 1M As shown, the acoustic configuration system 200 recommends moving the virtual display 160 closer to the stage. In some implementations, the recommended change includes recommending changing the settings of some of the virtual acoustic instruments. As an example, the electronic device may recommend reducing the gain of the virtual microphone in order to reduce the estimated reverberation. In some implementations, the method 300 includes recommending a location within the physical environment for placing a physical acoustic instrument corresponding to the virtual acoustic instrument based on the estimated acoustic parameters. For example, Figure 1M As shown, the electronic device 100 recommends moving the virtual display 60 closer to the stage representation 112. As a result, the display 60 will be placed closer to the stage 12.

[0075] As represented by block 330c, in some implementations, displaying the indication includes displaying a visualization of sound rays propagating through the physical environment. The visualization of the sound rays may allow the user of the electronic device to see which parts of the physical environment may have unacceptable sound quality. For example, the lack of sound rays in a portion of the physical environment indicates that the sound may not be loud enough in that portion of the physical environment. In some implementations, sound rays of different colors may represent sounds with different frequencies. In such implementations, sound rays of multiple colors in a portion of the physical environment may indicate frequency interference that requires appropriate EQ processing (e.g., a filter for reducing the effects of interfering frequencies). The density of sound rays within a portion of the physical environment may indicate whether the sound is loud enough (e.g., too many sound rays may indicate that the sound is too loud, while too few sound rays may indicate that the sound is not loud enough).

[0076] As represented by block 330d, in some implementations, displaying the indication includes indicating areas in the physical environment where the estimated reverberation is greater than an acceptable reverberation level or the estimated echo is greater than an acceptable echo level. Figure 1G As shown, the electronic device 100 displays reverberation indications 134b and 134c to indicate that the reverberation level adjacent to the pillar 16 may be greater than an acceptable reverberation level.

[0077] Figure 4 is a block diagram of a device 400 according to some implementations. In some implementations, the device 400 implements Figures 1A to 1N The electronic device 100 shown in FIG. 1 and / or Figure 1A to Figure 2400. Although some specific features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the device 400 includes one or more processing units (CPUs) 401, a network interface 402, a programming interface 403, a memory 404, one or more input / output (I / O) devices 408, and one or more communication buses 405 for interconnecting these and various other components.

[0078] In some implementations, a network interface 402 is provided to, among other purposes, establish and maintain a metadata tunnel between a cloud-hosted network management system and at least one private network including one or more compatible devices. In some implementations, one or more communication buses 405 include circuits for interconnecting and controlling communications between system components. Memory 404 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 404 optionally includes one or more storage devices located remotely from one or more CPUs 401. Memory 404 includes non-transitory computer-readable storage media.

[0079] In some implementations, one or more I / O devices 408 include a display Figure 1B The environment shown represents a display 110. In some implementations, the display comprises an extended reality (XR) display. In some implementations, the display comprises an opaque display. Alternatively, in some implementations, the display comprises an optically transparent display. In some implementations, one or more I / O devices 408 include a display for capturing Figure 2 The environmental sensors shown in environmental data 212 and / or instrument data 214 are shown. For example, one or more I / O devices 408 include an image sensor (e.g., a visible light camera and / or an infrared camera) for capturing image data 212a and / or an image sensor (e.g., a visible light camera and / or an infrared camera) for capturing Figure 2 A depth sensor (eg, a depth camera) is shown for the depth data 212b. In some implementations, one or more I / O devices 408 include an audio sensor (eg, a microphone) for receiving an auditory signal and converting the auditory signal into electronic signal data.

[0080] In some implementations, the memory 404 or a non-transitory computer-readable storage medium of the memory 404 stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 406 , a data acquirer 210 , an environment renderer 220 , a simulation generator 230 , and an instrument configurator 240 .

[0081] In various specific implementations, the data acquirer 210 includes a Figure 2 Instructions 210a and heuristics and metadata 210b for rendering environment data 212 and / or instrument data 214 are shown. In some implementations, environment renderer 220 includes instructions for rendering passthrough 222 and overlaying virtual instruments 224 onto Figure 2 Instructions 220a and heuristics and metadata 220b are shown on the passthrough 222. In some implementations, the simulation generator 230 includes a processor for executing the simulation 232 to generate Figure 2 Instructions 230a for estimating parameters 234 and heuristics and metadata 230b are shown. In some implementations, instrument configurator 240 includes instructions for displaying visual indicators 242 and providing Figure 2 Instructions 240a and heuristics and metadata 240b for a suggested configuration 244 are shown.

[0082] It should be understood that Figure 4 Serves as a functional description of various features that may be present in a particular implementation, as opposed to a structural schematic diagram of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 4 Some functional blocks shown separately in the figure may be implemented as a single block, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of blocks and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a specific specific implementation.

[0083] During a live performance, such as a concert, show, or demonstration, an audience member may be located in a portion of the physical environment where a sensory perception parameter (e.g., an acoustic parameter, a visual parameter, a tactile parameter, and / or an olfactory parameter) is outside of an acceptable range. For example, the sound level may be below a threshold sound level. As another example, the reverberation at the location may be greater than a threshold reverberation level. As yet another example, the lighting level may be outside of an acceptable lighting range. Thus, the audience member's experience may be adversely affected by the sensory perception parameter being outside of an acceptable range.

[0084] The present disclosure provides methods, systems and / or devices for enhancing a portion of a physical environment with augmented content when a local perception parameter is outside an acceptable range. The local perception parameter may include a local environmental parameter that provides an indication of how a person perceives (e.g., auditorily, visually, tactilely and / or olfactorily) the environment around him / her. When the device is within a portion of the physical environment where the local environmental parameter exceeds a threshold, the device augments the physical environment with augmented content. The device may measure the local environmental parameter using an on-device sensor.

[0085] The enhanced content may include acoustic content. The device may measure local acoustic parameters (e.g., sound levels, such as amplitude and / or frequency) based on the auditory signal data captured via the microphone. The device may play additional sounds to enhance the desired sound reaching the device and / or eliminate the undesired sound detected at the device. For example, the device may generate and play echo-cancelled sounds while still allowing the user to listen to live music being played.

[0086] The enhanced content may include visual content. The device may use an ambient light sensor to measure local lighting parameters (e.g., ambient light level). The device may display visual content to enhance desired visual effects and / or reduce the impact of undesired visual effects. For example, if the device is located in a portion of the physical environment that is too dark due to insufficient lighting, the device may increase the brightness of the display.

[0087] The enhanced content may include tactile content. The device may use a tactile sensor to measure a local tactile parameter (e.g., frequency or intensity of vibration). The device may generate a tactile response to enhance a desired tactile effect and / or eliminate an undesired tactile effect. For example, if a user sits on a cushion that vibrates, the device may increase the vibration to provide the effect of more bass or reduce the vibration to compensate for the excess bass.

[0088] Figure 5A is a diagram illustrating a physical environment 10, in which various physical instruments are placed throughout the physical environment 10. The physical instruments are placed with Figure 1E For example, microphone 40 is placed toward the front of stage 12, similar to Figure 1E The virtual microphone 140 is placed toward the front of the stage representation 112. The musical instrument 50 is placed with Figure 1E The display 60 is placed at the same position as the virtual instrument 150 shown. Figure 1E The speaker 70 is placed at the same position as the virtual display 60 shown. Figure 1E The virtual speaker 170 is shown at the same position. Figure 5A Also illustrated with Figure 1EThe virtual performers 124 are shown in the same position as the performers 24. For example, the first performer 24a is in the position of playing the first instrument 50a, similar to Figure 1E In the figure, the first virtual performer 124a is in a position to play the first virtual musical instrument 150a, the second performer 24b is in a position to sing at the second microphone 40b, similar to the position of the second virtual performer 124b singing at the second virtual microphone 140b, and the third performer 24c is in a position to play the second musical instrument 50b, similar to the position of the third virtual performer 124c playing the second virtual musical instrument 150b. Figure 5A Additionally, the physical environment 10 is illustrated with Figure 1E Virtual audience member 126 is shown located at the same location as audience member 26 .

[0089] Figure 5A An electronic device 500 is illustrated that is being used by a user 502. In some implementations, many audience members 26 have electronic devices. In some implementations, user 502 represents a specific one of audience members 26, and electronic device 500 represents the electronic device of that specific audience member 26. Alternatively, in some implementations, electronic device 500 operates as a master device, and the electronic device of audience member 26 operates as a slave device, and the master device controls the presentation of content on the slave device after obtaining informed consent during a live performance in physical environment 10. In some implementations, electronic device 500 includes a portable electronic device, such as a smart phone or a tablet. In some implementations, electronic device 500 includes a wearable computing device, such as an HMD or a watch. For example, in some implementations, many audience members 26 wear an HMD that presents content during a live performance.

[0090] In various implementations, the electronic device 500 includes an augmented content presentation system 600 (hereinafter referred to as "system 600" for brevity). In some implementations, the system 600 obtains a set of one or more environmental parameters 510 indicating a sensory condition at a location within the physical environment 10 (hereinafter referred to as "environmental parameters 510" for brevity). In some implementations, the electronic device 500 measures the environmental parameters 510 using an on-device sensor, and the environmental parameters 510 indicate the sensory condition at the location of the electronic device 500 within the physical environment 10. Alternatively, in a master-slave configuration, the electronic device 500 receives the environmental parameters 510 from an electronic device of a specific one of the audience members 26, and the environmental parameters 510 indicate the sensory condition at the location of the electronic device that the specific audience member 26 is using.

[0091] In some implementations, the environmental parameters 510 include acoustic parameters. In some implementations, the acoustic parameters include a loudness value indicating the loudness of an auditory signal received at a specific location within the physical environment 10. In some implementations, the acoustic parameters include a frequency response measured at a specific location within the physical environment 10. In some implementations, the acoustic parameters indicate the occurrence of echo or reverberation at a specific location within the physical environment 10. In some implementations, the acoustic parameters indicate a sound quality value characterizing the quality of the sound detected at a specific location within the physical environment 10.

[0092] In some implementations, the environmental parameters 510 include visual parameters. In some implementations, the visual parameters include an ambient light value indicating a brightness level at a specific location within the physical environment 10. In some implementations, the visual parameters include a color value indicating a color of light detected at a specific location within the physical environment 10. In some implementations, the visual parameters include a frequency of light detected at a specific location within the physical environment 10.

[0093] In some implementations, environmental parameters 510 include haptic parameters. In some implementations, the haptic parameters indicate a vibration level detected at a specific location within the physical environment 10. In some implementations, the haptic parameters indicate a type of vibration detected at a specific location. In some implementations, the haptic parameters indicate a frequency and / or intensity of a vibration detected at a specific location.

[0094] In some implementations, the system 600 triggers the presentation of the augmented content 530 based on the environmental parameter 510. In some implementations, the system 600 determines to present the augmented content 530 when the environmental parameter 510 is outside of an acceptable range. In various implementations, the augmented content 530 is presented to tend to enhance the sensory conditions at a particular location within the physical environment 10.

[0095] refer to Figure 5B In some implementations, the environmental parameters 510 include one or more loudness values ​​512 that indicate the loudness of sounds in an area of ​​the physical environment where the first subset 26a of the audience members 26 are located. The system 600 determines whether the loudness value 512 is within an acceptable loudness value range. Figure 5BIn the example of , the system 600 determines that the loudness value 512 is outside of an acceptable range of loudness values. For example, the system 600 determines that the loudness value 512 is below a loudness threshold (e.g., the sound is not loud enough in the area where the first subset 26a of the audience members 26 are seated). Therefore, the system 600 determines to present amplified acoustic content 532 in order to compensate for the low loudness in the area where the first subset 26a of the audience members 26 are seated. In some implementations, the amplified acoustic content 532 is an amplified version of a live performance occurring in the physical environment 10. The system 600 may send instructions to the electronic device of each audience member in the first subset 26a to present the amplified acoustic content 532. Presenting the amplified acoustic content 532 compensates for the relatively low loudness of the sound in the area where the first subset 26a is seated.

[0096] Although Figure 5B An example of amplifying the audio from the live performance is illustrated, but in some implementations, the audio from the live performance can be modified in another way. For example, in some implementations, the loudness value 512 can be greater than the loudness threshold. In such implementations, the enhanced content 530 can suppress the audio of the live performance instead of further amplifying the audio. In some implementations, the audience members in the first subset 26a can wear noise-canceling headphones that can output a suppressed version of the audio from the physical environment 10. For example, the noise-canceling headphones can reduce the amplitude of the audio from the physical environment 10.

[0097] refer to Figure 5C In some implementations, the environmental parameter 510 indicates that the area of ​​the physical environment 10 where the second subset 26b of the audience members 26 are seated is associated with the reverberation occurrence 514. The system 600 determines whether the reverberation occurrence 514 is greater than the reverberation threshold. Figure 5C In the example of , system 600 determines that reverberation occurrence 514 is greater than a reverberation threshold. In response to determining that reverberation occurrence 514 is greater than the reverberation threshold, system 600 presents compensatory acoustic content 534 that eliminates reverberation in an area where second subset 26b of audience members 26 are seated. More generally, in various implementations, augmented content 530 includes audio content that eliminates undesirable audio portions detected within a portion of physical environment 10 while allowing audience members 26 seated in the portion of physical environment 10 to hear desired audio portions.

[0098] refer to Figure 5DIn some implementations, the environmental parameters 510 indicate that the area of ​​the physical environment 10 where the third subset 26c of the audience members 26 are seated is associated with an ambient light value 516. The system 600 determines whether the ambient light value 516 is within an acceptable range of ambient light values. For example, the system 600 determines whether the ambient light value 516 is greater than or less than a brightness threshold. If the ambient light value 516 is less than the brightness threshold, the system 600 generates an augmented visual content 536 that darkens the area of ​​the physical environment 10 where the third subset 26c of the audience members 26 are located (e.g., the system 600 reduces the brightness level of the see-through display). If the ambient light value 516 is greater than the brightness threshold, the augmented visual content 536 brightens the area of ​​the physical environment 10 where the third subset 26c of the audience members 26 are located (e.g., the system 600 increases the brightness level of the see-through display).

[0099] Figure 6 6 illustrates an example implementation of system 600. In various implementations, system 600 includes data acquirer 610, environment evaluator 620, and augmented content presenter 630. In some implementations, data acquirer 610 acquires environmental parameters 612 (e.g., FIG. 5A to FIG. 5D In some implementations, environmental parameter 612 includes a perception value that indicates how a person perceives (e.g., hears, sees, feels, and / or smells) a portion of a physical environment.

[0100] In some implementations, the environmental parameters 612 include acoustic parameters 614 that indicate acoustic conditions of a particular area within the physical environment. In some implementations, the acoustic parameters 614 include a loudness value 614a (e.g., Figure 5B The loudness value 614a may include an amplitude value. In some implementations, the acoustic parameters 614 include a frequency response 614b indicating frequencies encountered within an area of ​​the physical environment. In some implementations, the acoustic parameters 614 indicate echo occurrence 614c or reverberation occurrence 614d within an area of ​​the physical environment (e.g., Figure 5C Reverberation occurs 514 as shown. In some implementations, the acoustic parameters 614 include a sound quality value 614e that indicates the quality of sound detected within an area of ​​the physical environment.

[0101] In some implementations, the environmental parameters 612 include visual parameters 616 indicating visual conditions (e.g., optical conditions or viewing conditions) of a particular area within the physical environment. In some implementations, the visual parameters 616 include an ambient light value 616a. The ambient light value 616a indicates how bright or dark the corresponding portion of the physical environment is. In some implementations, the visual parameters 616 indicate the intensity, color, and / or frequency of light in a particular area of ​​the physical environment.

[0102] In some implementations, environmental parameters 612 include haptic parameters 618 indicating haptic conditions of a particular area within the physical environment. In some implementations, haptic parameters 618 include vibration values ​​618a indicating the intensity of vibrations in a particular area of ​​the physical environment. In some implementations, haptic parameters 618 indicate the intensity and / or frequency of vibrations in a particular area of ​​the physical environment.

[0103] In various implementations, the environment evaluator 620 evaluates the sensory condition of a particular portion of the physical environment by comparing the environment parameter 612 to an acceptable range 622. In some implementations, the environment evaluator 620 determines whether the environment parameter 612 is within the acceptable range 622. If the environment parameter 612 is not within the acceptable range 622, the environment evaluator 620 generates a trigger 629 for the augmented content presenter 630 to present augmented content 632 to enhance the sensory condition of the portion of the physical environment.

[0104] In some implementations, the acceptable range 622 includes an acceptable acoustic range 624. The environment evaluator 620 determines whether the acoustic parameter 614 is within or outside the acceptable acoustic range 624. If the acoustic parameter 614 is outside the acceptable acoustic range 624, triggering 629 causes the enhanced content presenter 630 to present enhanced acoustic content 634 to improve the acoustic conditions of the portion of the physical environment. In some implementations, the environment evaluator 620 determines whether the loudness value 614a is within the acceptable loudness range 624a. If the loudness value 614a is below the lower end of the acceptable loudness range 624a, the enhanced content presenter 630 presents the amplified acoustic content 634a (e.g., Figure 5B Amplified acoustic content 532 is shown).

[0105] In some implementations, the environment evaluator 620 determines whether the frequency response 614b indicates a frequency within or outside the acceptable frequency range 624b. If the frequency indicated by the frequency response 614b is outside the acceptable frequency range 624b, the enhanced content presenter 630 may present the canceled acoustic content 634b that cancels the frequency outside the acceptable frequency range 624b.

[0106] In some implementations, the environment evaluator 620 determines whether the echo occurrence 614c indicates an occurrence of an echo within or outside an acceptable echo range 624c. For example, the environment evaluator 620 determines whether the number, duration, and / or intensity of the echo is within or outside the acceptable echo range 624c. If the echo occurrence 614c is outside the acceptable echo range 624c, the enhanced content presenter 630 presents the echo compensation content 634c to reduce the impact of the echo (e.g., to reduce the number, duration, and / or intensity of the echo).

[0107] In some implementations, the environment evaluator 620 determines whether the reverberation occurrence 614d indicates an occurrence of reverberation within or outside an acceptable reverberation range 624d. For example, the environment evaluator 620 determines whether the amount, duration, and / or intensity of the reverberation is within or outside the acceptable reverberation range 624d. If the reverberation occurrence 614d is outside the acceptable reverberation range 624d, the enhanced content presenter 630 presents the reverberation compensation content 634d to reduce the impact of the reverberation (e.g., to reduce the amount, duration, and / or intensity of the reverberation).

[0108] In some implementations, the environment evaluator 620 determines whether the sound quality value 614e is within or outside the acceptable sound quality range 624e. If the sound quality value 614e is outside the acceptable sound quality range 624e, the enhanced content presenter 630 presents the enhanced acoustic content 634 to change the sound quality value 614e to a modified sound quality value within the acceptable sound quality range 624e.

[0109] In some implementations, the acceptable range 622 includes an acceptable visual range 626. The environment evaluator 620 determines whether the visual parameter 616 is within or outside the acceptable visual range 626. If the visual parameter 616 is outside the acceptable visual range 626, a trigger 629 causes the augmented content presenter 630 to present augmented visual content 636 (e.g., Figure 5DThe enhanced visual content 536 shown in the figure is used to improve the optical conditions of the portion of the physical environment. In some specific implementations, the environment evaluator 620 determines whether the ambient light value 616a is within the acceptable ambient lighting range 626a. If the ambient light value 616a is outside the acceptable ambient lighting range 626a, the enhanced content presenter 630 presents the brightness adjustment content 636a. If the ambient light value 616a is below the lower end of the acceptable ambient lighting range 626a, the brightness adjustment content 636a may include a brighter version of the pass-through content, which has the effect of increasing the brightness of the environment that the user sees through the display. Conversely, if the ambient light value 616a is greater than the upper end of the acceptable ambient lighting range 626a, the brightness adjustment content 636a may include a darker version of the pass-through content, which has the effect of reducing the brightness of the environment that the user sees through the display.

[0110] In some implementations, the enhanced visual content 636 includes a visual effect 636b that tends to enhance the optical conditions in a portion of the environment. Figure 5D , the portion of the physical environment 10 where the third subset 26c of audience members 26 are seated may have a monotonous light, rather than the monotonous light of the first subset 26a (e.g. Figure 5B In this example, the augmented content presenter 630 may trigger the HMD worn by the third subset 26c to display a visual effect 636b with colored lighting, thereby providing the third subset 26c with the same visual experience as the first subset 26a. In some implementations, the visual effect 636b may include an optical illusion and / or animation that enhances the viewer's visual experience. For example, the visual effect 636b provided to the third subset 26c ( Figure 5D ) may include a larger representation of performer 24 so as to provide the illusion that performer 24 is closer to third subset 26c than performer 24 actually is (e.g., visual effect 636b may make performer 24 appear to be the same size as performer 24 appears to first subset 26a of audience members 26).

[0111] In some implementations, the acceptable range 622 includes an acceptable haptic range 628. The environment evaluator 620 determines whether the haptic parameter 618 is within or outside the acceptable haptic range 628. If the haptic parameter 618 is outside the acceptable haptic range 628, triggering 629 causes the augmented content presenter 630 to present augmented haptic content 638 to improve the haptic condition of the portion of the physical environment. In some implementations, the environment evaluator 620 determines whether the vibration value 618a is within the acceptable vibration range 628a. If the vibration value 618a is below the lower end of the acceptable vibration range 628a, the augmented haptic content 638 includes an additional haptic response 638a having the effect of increasing the vibration in the portion of the environment where the user is sitting (e.g., by vibrating the haptic seat, for example, by applying a vibration amplification haptic response via the haptic seat where the user is sitting). Conversely, if vibration value 618a is greater than an upper end of acceptable vibration range 628a, then enhancing haptic content 638 includes suppressing haptic responses 638b that have the effect of reducing vibrations in that portion of the physical environment (e.g., by applying vibration-cancelling haptic responses via a haptic chair).

[0112] Figure 7 is a flowchart representation of a method 700 for presenting augmented content in a portion of a physical environment. In various implementations, the method 700 comprises FIG. 5A to FIG. 5D The electronic device 500 and / or FIG. 5A to FIG. 6 The system 600 shown in is executed. In some specific implementations, the method 700 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the method 700 is performed by a processor executing a code stored in a non-transitory computer readable medium (e.g., a memory).

[0113] As represented by block 710, in various implementations, method 700 includes measuring an environmental parameter indicative of a sensory condition at a location of an electronic device within a physical environment. Figure 5A As shown, system 600 obtains environmental parameter 510. In some implementations, the electronic device obtains the environmental parameter from another device that measures the environmental parameter. In some implementations, the electronic device obtains a first environmental parameter indicating a first sensory condition at a first location within the physical environment, a second environmental parameter indicating a second sensory condition at a second location within the physical environment, ..., and an nth environmental parameter indicating an nth sensory condition at an nth location within the physical environment.

[0114] In some implementations, the environmental parameters include a perception value indicating how a person at the location of the electronic device perceives the physical environment. In some implementations, the environmental parameters include an acoustic parameter indicating an acoustic condition of a portion of the physical environment (e.g., Figure 6For example, the acoustic parameters indicate how loud the sound arriving at the location might sound to a person at the location (e.g., loudness value 614a), the frequency of the sound arriving at the location (e.g., frequency response 614b), whether there is an echo at the location (e.g., echo occurrence 614c), whether there is reverberation at the location (e.g., reverberation occurrence 614d), and / or the quality of the sound at the location (e.g., sound quality value 614e).

[0115] In some implementations, the environmental parameter includes a visual parameter (e.g., Figure 6 For example, the visual parameter indicates a brightness level of a portion of the physical environment where the electronic device is located (e.g., ambient light value 616a). In some implementations, the visual parameter indicates an amount of visual noise in the portion of the physical environment (e.g., variations in the color of light reaching the location of the electronic device).

[0116] In some implementations, the environmental parameter includes a tactile parameter (e.g., a vibration condition) indicating a tactile condition of a portion of the physical environment. Figure 6 For example, the haptic parameter indicates a vibration level (eg, vibration value 618a) at a portion of the physical environment where the electronic device is located.

[0117] As represented by block 720, in some implementations, method 700 includes determining whether the environmental parameter is within an acceptable range. Figure 6 As shown, the environment evaluator 620 determines whether the acoustic parameter 614 is within an acceptable range 622. In some implementations, the method 700 includes determining to present enhanced content when the environmental parameter is outside the acceptable range. Figure 6 As shown, when the environmental parameter 612 is outside the acceptable range 622 , the environmental evaluator 620 generates a trigger 629 for the enhanced content presenter 630 to present the enhanced content 632 .

[0118] As represented by block 720a, in some implementations, the environmental parameter includes an acoustic parameter and the acceptable range includes an acceptable acoustic range. Figure 6 As shown, the environmental parameters 612 include acoustic parameters 614 (e.g., loudness value 614a, frequency response 614b, echo generation 614c, reverberation generation 614d and / or sound quality value 614e), and the acceptable range 622 includes an acceptable acoustic range 624 (e.g., acceptable loudness range 624a, acceptable frequency range 624b, acceptable echo range 624c, acceptable reverberation range 624d and / or acceptable sound quality range 624e).

[0119] As represented by block 720b, in some implementations, the environmental parameters include visual parameters and the acceptable range includes acceptable lighting levels. Figure 6 As shown, environmental parameters 612 include visual parameters 616 (eg, ambient light value 616a), and acceptable range 622 includes acceptable visual range 626 (eg, acceptable ambient lighting range 626a).

[0120] As shown in block 720c, in some implementations, the environmental parameter includes a tactile parameter, and the acceptable range includes an acceptable tactile level at the location. Figure 6 As shown, environmental parameters 612 include haptic parameters 618 (eg, vibration value 618a), and acceptable range 622 includes acceptable haptic range 628 (eg, acceptable vibration range 628a).

[0121] As represented by block 730, in various implementations, method 700 includes, in response to determining that the environmental parameter is not within an acceptable range, triggering presentation of augmented content to enhance sensory conditions at the location of the electronic device. Figure 6 As shown, when one or more of the environmental parameters 612 is outside the acceptable range 622, the enhanced content presenter 630 triggers the presentation of the enhanced content 632. In various specific implementations, the presentation of the enhanced content enhances the sensory condition at the location by improving the acoustic condition, optical condition, and / or tactile condition of the location. In various specific implementations, before the presentation of the enhanced content, the sensory condition is below a threshold, and the presentation of the enhanced content improves the sensory condition so that the sensory condition becomes greater than the threshold.

[0122] As represented by block 730a, in some implementations, the augmented content augments the live performance in the physical environment. Figure 5A As shown, augmented content 530 augments the live performance delivered by performer 24. In some implementations, the augmented content improves the perception of the live performance by some audience members. In some implementations, the augmented content augments the live performance audibly, visually, haptically (e.g., by outputting a haptic response), and / or olfactorily (e.g., by diffusing a scent).

[0123] As represented by block 730b, in some implementations, the enhanced content includes acoustic content. Figure 6 As shown, enhanced content 632 includes enhanced acoustic content 634. In some implementations, triggering the presentation of the enhanced content includes outputting an auditory signal that cancels an undesired auditory signal received at the electronic device. Figure 6As shown, in some implementations, enhancing acoustic content 634 includes eliminating acoustic content 634b. In some implementations, triggering the presentation of enhanced content includes outputting an auditory signal that enhances a desired auditory signal received at the electronic device. For example, Figure 5B As shown, system 600 presents amplified acoustic content 532. In some implementations, the acoustic content compensates for reverberation at the location of the electronic device. Figure 6 As shown, enhanced acoustic content 634 may include reverberation compensated acoustic content. In some implementations, the acoustic content amplifies the auditory signal received at the electronic device. For example, Figure 6 As shown, enhanced acoustic content 634 may include amplified acoustic content 634a.

[0124] In some implementations, triggering the presentation of the enhanced content includes causing another electronic device to output an auditory signal. For example, the electronic device may operate as a master device that triggers various other devices operating as slave devices to play different enhanced content. FIG. 5B to FIG. 5D , the electronic device 500 may operate as a master device and instruct devices of a first subset 26a of the audience members 26 to play amplified acoustic content 532, instruct devices of a second subset 26b of the audience members 26 to play compensatory acoustic content 534, and instruct HMDs worn by a third subset 26c of the audience members to play augmented visual content 536.

[0125] As represented by block 730c, in some implementations, the enhanced content includes visual content. Figure 5D As shown, the enhanced content 530 provided to the devices of the third subset 26c of the audience members 26 includes enhanced visual content 536. In some implementations, triggering the presentation of the enhanced content includes increasing the brightness of the display to compensate for low ambient lighting at the location of the electronic device. In some implementations, triggering the presentation of the enhanced content includes decreasing the brightness of the display to compensate for high ambient lighting at the location of the electronic device. For example, Figure 6 As shown, enhanced visual content 636 includes brightness adjustment content 636a. In some implementations, triggering the presentation of enhanced content includes displaying a visual effect corresponding to an auditory signal received at the electronic device. Figure 6 As shown, enhanced visual content 636 may include visual effects 636b. In some implementations, the visual effects match the acoustic characteristics of the auditory signal received at the location of the electronic device. For example, if the auditory signal corresponds to a song with a relatively fast beat, the visual effects may include flashing lights that display a disco style.

[0126] In some implementations, triggering the presentation of the enhanced content includes causing another electronic device to display the visual content. Figure 5D As depicted, the electronic device 500 operates as a master device that instructs the HMDs worn by the third subset 26 c of the audience members 26 to display the augmented visual content 536 .

[0127] As represented by block 730d, in some implementations, the augmented content includes tactile content. Figure 6 As shown, enhanced content 632 may include enhanced tactile content 638. In some implementations, triggering presentation of the enhanced content includes outputting a tactile response via a tactile device based on auditory signal data received at the electronic device. For example, the tactile content includes vibrations exhibited by a device including a motor attached to an unbalanced load. In some implementations, triggering presentation of the enhanced content includes generating a tactile response to cancel a tactile response detected at a location of the electronic device. For example, Figure 6 As shown, enhanced haptic content 638 may include an additional haptic response 638a or a suppressed haptic response 638b.

[0128] Figure 8 is a block diagram of a device 800 according to some implementations. In some implementations, the device 800 implements FIG. 5A to FIG. 5D The electronic device 500 and / or FIG. 5A to FIG. 6 800 . Although some specific features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the device 800 includes one or more processing units (CPUs) 801, a network interface 802, a programming interface 803, a memory 804, one or more input / output (I / O) devices 808, and one or more communication buses 805 for interconnecting these and various other components.

[0129] In some implementations, a network interface 802 is provided to, among other purposes, establish and maintain a metadata tunnel between a cloud-hosted network management system and at least one private network including one or more compatible devices. In some implementations, one or more communication buses 805 include circuits for interconnecting and controlling communications between system components. Memory 804 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 804 optionally includes one or more storage devices located remotely from one or more CPUs 801. Memory 804 includes non-transitory computer-readable storage media.

[0130] In some implementations, the one or more I / O devices 808 include a speaker for outputting the enhanced acoustic content 634, a display for displaying the enhanced visual content 636, and / or a display for outputting the enhanced acoustic content 634. Figure 6 In some implementations, the display comprises an extended reality (XR) display. In some implementations, the display comprises an opaque display. Alternatively, in some implementations, the display comprises an optically transparent display. In some implementations, one or more I / O devices 808 include a display for capturing Figure 6 Environmental sensors for environmental parameters 612 are shown. For example, one or more I / O devices 808 include a microphone for capturing acoustic parameters 614, an image sensor (e.g., a visible light camera and / or an infrared light camera) for capturing visual parameters 616, and / or a tactile sensor for capturing tactile parameters 618.

[0131] In some implementations, the memory 804 or a non-transitory computer-readable storage medium of the memory 804 stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 806 , a data acquirer 610 , an environment evaluator 620 , and an enhanced content presenter 630 .

[0132] In various specific implementations, the data acquirer 610 includes a Figure 6 Instructions 610a and heuristics and metadata 610b for evaluating environmental parameters 612 shown. In some implementations, the environmental evaluator 620 includes instructions for evaluating environmental parameters 612 and heuristics and metadata 610b. Figure 6 Instructions 620a and heuristics and metadata 620b for environmental parameters 612 associated with acceptable ranges 622 are shown. In some implementations, the enhanced content presenter 630 includes a method for triggering Figure 6 Instructions 630a and heuristics and metadata 630b that enhance the presentation of content 632 are shown.

[0133] It should be understood that Figure 8 Serves as a functional description of various features that may be present in a particular implementation, as opposed to a structural schematic diagram of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 8 Some functional blocks shown separately in the figure may be implemented as a single block, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of blocks and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a specific specific implementation.

[0134] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.

Claims

1. A method, comprising: At an electronic device including a non-transitory memory, one or more processors, a display, and an image sensor: Displaying the virtual acoustic instrument on the display as an overlay onto a transmittal of the physical environment; performing an acoustic simulation based on respective characteristics of the virtual acoustic instrument to generate estimated acoustic parameters for respective locations within the physical environment; as well as An indication of the estimated acoustic parameter is displayed on the display. 2 . The method of claim 1 , wherein performing the acoustic simulation comprises obtaining an acoustic mesh for the physical environment and performing the acoustic simulation based on the acoustic mesh. The method of claim 2 , wherein the acoustic mesh indicates acoustic properties of materials in the physical environment.

4. A method according to any one of claims 1 to 3, further comprising displaying virtual audience members overlaid onto the pass-through of the physical environment.

5. The method of claim 4, wherein performing the acoustic simulation comprises simulating sound absorbed or reflected by the virtual audience member.

6. The method of any one of claims 4 and 5, wherein performing the acoustic simulation comprises simulating the virtual audience member emitting a sound.

7. The method according to any one of claims 1 to 6, further comprising: measuring actual acoustic parameters when a physical acoustic instrument is placed at a position corresponding to the virtual acoustic instrument; as well as The acoustic simulation is adjusted based on a difference between the actual acoustic parameter and the estimated acoustic parameter.

8. The method of any one of claims 1 to 7, wherein performing the acoustic simulation comprises playing pre-recorded sounds of a musical instrument.

9. The method of any one of claims 1 to 8, further comprising indicating areas of the physical environment where the estimated acoustic parameter is not within an acceptable range.

10. The method according to any one of claims 1 to 9, further comprising recommending a change in a configuration value of the virtual acoustic instrument. The method of claim 10 , further comprising performing another acoustic simulation after the changing of the configuration value has been performed.

12. The method according to any one of claims 10 and 11, wherein recommending the change comprises recommending moving some of the virtual acoustic instruments to different locations.

13. The method according to any one of claims 10 to 12, wherein recommending the change comprises recommending changing settings of some of the virtual acoustic instruments.

14. The method according to any one of claims 1 to 13, further comprising recommending a location within the physical environment for placing a physical acoustic instrument corresponding to the virtual acoustic instrument based on the estimated acoustic parameters.

15. The method of any one of claims 1 to 14, wherein displaying the indication comprises displaying a visualization of acoustic rays propagating through the physical environment.

16. The method of any one of claims 1 to 15, wherein displaying the indication comprises indicating areas of the physical environment where estimated reverberation is greater than an acceptable reverberation level.

17. The method of any one of claims 1 to 16, wherein displaying the virtual acoustic instrument comprises receiving a user input indicating a respective placement position of the virtual acoustic instrument.

18. The method of any one of claims 1 to 17, wherein displaying the virtual acoustic instrument comprises automatically placing the virtual acoustic instrument based on dimensions of the physical environment.

19. The method of any one of claims 1 to 18, wherein the virtual acoustic instrument represents a physical acoustic instrument.

20. A device, comprising: monitor; Environmental sensors; one or more processors; non-transitory memory; and One or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to perform any of the methods of claims 1 to 19.

21. A non-transitory memory storing one or more programs which, when executed by one or more processors of a device having an environmental sensor and a display, cause the device to perform any of the methods of claims 1 to 19.

22. A device, comprising: monitor; Environmental sensors; one or more processors; non-transitory memory; and Means for causing the device to perform any of the methods according to claims 1 to 19.

23. A method comprising: At an electronic device including a non-transitory memory, one or more processors, a display, and an image sensor: measuring an environmental parameter indicative of a sensory condition at a location of the electronic device within a physical environment; determining whether the environmental parameters are within an acceptable range; as well as In response to determining that the environmental parameter is not within the acceptable range, presentation of augmented content is triggered to enhance the sensory condition at the location of the electronic device.

24. The method of claim 23, wherein the augmented content augments a live performance in the physical environment.

25. The method of any one of claims 23 and 24, wherein the enhanced content includes acoustic content.

26. The method of any one of claims 23 to 25, wherein triggering presentation of the enhanced content comprises outputting an auditory signal that cancels an undesired auditory signal received at the electronic device.

27. The method of any one of claims 23 to 26, wherein triggering presentation of the enhanced content comprises outputting an auditory signal that enhances a desired auditory signal received at the electronic device.

28. The method of any one of claims 23 to 27, wherein the enhanced content compensates for reverberation at the location of the electronic device.

29. The method of any one of claims 23 to 28, wherein the enhancement content amplifies an auditory signal received at the electronic device.

30. The method of any one of claims 23 to 29, wherein triggering presentation of the enhanced content comprises causing another electronic device to output an auditory signal.

31. A method according to any one of claims 23 to 30, wherein the enhanced content comprises visual content.

32. The method of any one of claims 23 to 31, wherein triggering presentation of the enhanced content comprises increasing brightness of the display to compensate for low ambient lighting at the location of the electronic device.

33. The method of any one of claims 23 to 32, wherein triggering presentation of the augmented content comprises reducing brightness of the display to compensate for high ambient lighting at a location of the electronic device.

34. The method of any one of claims 23 to 33, wherein triggering presentation of the enhanced content comprises displaying a visual effect corresponding to an auditory signal received at the electronic device.

35. The method of any one of claims 23 to 34, wherein triggering presentation of the enhanced content comprises causing another electronic device to display visual content.

36. The method of any one of claims 23 to 35, wherein the augmented content comprises haptic content.

37. The method of any one of claims 23 to 36, wherein triggering presentation of the augmented content comprises outputting a tactile response via a haptic device based on auditory signal data received at the electronic device.

38. The method of any one of claims 23 to 37, wherein triggering presentation of the augmented content comprises generating a tactile response to eliminate a tactile response detected at the location of the electronic device.

39. The method of any one of claims 23 to 38, wherein the environmental parameter comprises an acoustic parameter and the acceptable range comprises an acceptable acoustic range.

40. A method according to any one of claims 23 to 38, wherein the environmental parameter comprises a visual parameter and the acceptable range comprises an acceptable lighting level.

41. A method according to any one of claims 23 to 40, wherein the environmental parameter comprises a tactile parameter and the acceptable range comprises an acceptable tactile level at the location.

42. A device, comprising: monitor; Environmental sensors; one or more processors; non-transitory memory; and One or more programs stored in the non-volatile memory, which, when executed by the one or more processors, cause the device to perform any of the methods of claims 23 to 41.

43. A non-transitory memory storing one or more programs which, when executed by one or more processors of a device having an environmental sensor and a display, cause the device to perform any of the methods of claims 23 to 41.

44. A device, comprising: monitor; Environmental sensors; one or more processors; non-transitory memory; and Means for causing the apparatus to perform any of the methods of claims 23 to 41.

Images