Visual indicator for microphone placement
By combining acoustic models and visual indicators, the microphone placement position is automatically determined and guided, the problem of sound differences at different positions is solved, the accuracy and efficiency of audio capture is improved, and the process of manual attempt by users is reduced.
Patent Information
- Application Number
- CN202410605644.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-17
- Filing Date
- 2024-05-16
- Publication Date
- 2024-11-22
AI Technical Summary
The captured sound varies greatly at different microphone placement positions, making it difficult for the device to properly capture sound in the environment, and the user needs to manually try multiple positions to determine the optimal placement position.
By generating an acoustic model of the environment, combined with the microphone's pickup mode, the microphone placement position is automatically determined and displayed, and the user is guided to place the microphone with visual indicators to ensure proper audio capture.
Reduces the need for users to manually determine the placement of microphones, improves the accuracy and efficiency of audio capture, reduces device power consumption, and ensures appropriate audio data quality.
Smart Images

Figure CN119002760A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to visual indicators for microphone placement. Background Art
[0002] Some devices include microphones that can detect sounds within an environment. Placing the microphone at different locations within the environment may result in different sounds being detected. Placing the microphone at different locations within the environment may also result in sounds being detected to different degrees. BRIEF DESCRIPTION OF THE DRAWINGS
[0003] So that the present disclosure may be understood by those of ordinary skill in the art, a more particular description may be had with reference to aspects of some illustrative embodiments, some of which are shown in the accompanying drawings.
[0004] Figures 1A to 1L is an illustration of an example operating environment according to some implementations.
[0005] Figure 2 is an illustration of a microphone placement guidance system according to some specific implementations.
[0006] Figure 3 is a flowchart representation of a method of indicating placement of a microphone according to some specific implementations.
[0007] Figure 4 is a block diagram of an apparatus for determining placement of a microphone according to some specific implementations.
[0008] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, for clarity, the sizes of the various features may be arbitrarily expanded or reduced. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention
[0009] Various embodiments disclosed herein include devices, systems, and methods for indicating the placement of a microphone. In some embodiments, a method is performed by a device comprising a display, an environmental sensor, a non-transitory memory, and one or more processors coupled to the display, the environmental sensor, and the non-transitory memory. In various embodiments, a method includes obtaining an acoustic model of an environment. In some embodiments, the acoustic model indicates a set of one or more acoustic characteristics of the environment. In various embodiments, the method includes determining the placement of a microphone within the environment based on the set of one or more acoustic characteristics of the environment and the pickup pattern of the microphone. In various embodiments, the method includes displaying a representation of the environment on a display and a visual indicator overlaid on the representation of the environment to indicate the placement of the microphone.
[0010] According to some specific implementations, the device includes one or more processors, multiple sensors, non-transitory memory, and one or more programs. In some specific implementations, one or more programs are stored in non-transitory memory and executed by one or more processors. In some specific implementations, one or more programs include instructions for executing or causing the execution of any method described herein. According to some specific implementations, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of a device, cause the device to execute or cause the execution of any of the methods described herein. According to some specific implementations, a device includes one or more processors, non-transitory memory, and a device for executing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION
[0011] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.
[0012] Some devices include microphones that can detect sounds generated in an environment. However, placing the microphone at different locations within the environment may result in different sounds being detected. Furthermore, placing the microphone at different locations within the environment may result in sounds being detected to different degrees. In order to capture the sounds being generated in an appropriate manner, it may be necessary to place the microphone at a specific location within the environment.
[0013] The present disclosure provides a method, system and / or device for determining the placement of a microphone and displaying a visual indication of the placement. The device determines the placement of the microphone based on an acoustic model of the environment and a pickup pattern of the microphone. The device displays a visual indicator that serves as a guide to place the microphone at the placement position determined by the device. Placing the microphone at the placement position determined by the device allows the microphone to properly capture sounds originating from the environment. The device can automatically determine the placement of the microphone, thereby reducing the need for the user of the device to manually determine the placement of the microphone.
[0014] The acoustic model of the environment may include an acoustic grid of the environment. The acoustic model indicates how sound propagates within the environment. The device may generate the acoustic model based on the size of the environment and the placement of objects within the environment. For example, the device may generate the acoustic model of the environment based on a visual grid of the environment, which indicates the size of the environment and the placement of objects within the environment.
[0015] The device may capture an image of the environment in order to identify objects that may be generating sound. For example, the device may scan a room to identify musical instruments located within the room. The acoustic model of the environment may indicate the location of musical instruments in the environment. For example, the acoustic model may indicate the location of a piano in a room, the location of a drum set in a room, etc. When determining the placement of the microphone, the device may select a placement that allows the microphone to adequately capture the sound being generated by the sound-generating object (e.g., a musical instrument).
[0016] The device may generate an acoustic model based on material properties of the environment. The material properties may indicate the sound absorption quality of the environment. For example, the material properties may indicate the degree to which floors and / or furniture in the environment absorb sound. The material properties may indicate the sound reflectivity of the environment. For example, the material properties may indicate the degree to which walls and / or wall hangings in the environment reflect sound. As an example, a carpeted floor may absorb sound to a greater extent than a floor with tile. As another example, a room with plastic chairs may reflect more sound than a room with an overstuffed sofa.
[0017] The pickup pattern of the microphone may indicate the directional pattern of the microphone. For example, the pickup pattern may indicate whether the microphone is a unidirectional microphone, a bidirectional microphone, an omnidirectional microphone, a cardioid microphone, a sub-cardioid microphone, etc. The device may determine the model and / or serial number of the microphone and use the model and / or serial number to look up the pickup pattern of the microphone from a database storing pickup patterns of various microphones.
[0018] The acoustic model may indicate the placement of other microphones in the environment. The device may determine the placement of a particular microphone based on the corresponding placement of the other microphones. For example, if another microphone is placed adjacent to a set of drums, the current microphone may be placed away from the set of drums because the current microphone may not need to capture the sound emanating from the set of drums.
[0019] The device may perform a simulation that illustrates how sound captured by the microphone may sound if the microphone is placed at different locations. For example, before placing the microphone at a first location in an environment, the user may play a first simulation that shows how well the microphone will capture sound from the first location. If the simulation indicates that the microphone will not properly capture sound from the first location, the user may place the microphone at an alternative second location indicated by the device.
[0020] Figure 1A 1 is a diagram illustrating an example physical environment 10 according to some implementations. Although relevant features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the physical environment 10 includes a front wall 20, a side wall 24, a piano 30, a piano player 34, a bench 40, an audience member 44, a microphone 50, an electronic device 100, and a user 102 of the electronic device 100. In some implementations, the physical environment 10 represents an enclosed space, such as a room (e.g., a living room in a home), a banquet hall, a concert venue, a stadium, etc. In some implementations, the physical environment 10 represents an open space, such as a park, an amphitheater, a backyard of a home, etc.
[0021] In various implementations, the electronic device 100 includes a microphone placement guidance system 200 (hereinafter referred to as "system 200" for brevity) that provides guidance to the user 102 in placing the microphone 50 at an appropriate location within the physical environment 10. In some implementations, the system 200 automatically determines an appropriate location for placing the microphone 50 within the physical environment 10, and the electronic device 100 displays a visual indicator indicating the location to the user 102. In various implementations, automatically determining the location for placing the microphone 50 reduces the need for the user 102 to manually determine the location for placing the microphone 50. In some implementations, automatically determining the location for placing the microphone 50 reduces the need for trial and error, in which the user 102 physically places the microphone 50 at various locations within the physical environment 10 in order to select a location that appears appropriate for the microphone 50. Automatically determining the placement location of the microphone 50 reduces inappropriate audio capture by the microphone 50 and tends to reduce power consumption of the electronic device 100 associated with playing inappropriate audio capture.
[0022] See also Figure 1B , the system 200 obtains an acoustic model 60 of the physical environment 10. In various implementations, the acoustic model 60 indicates how sound propagates within the physical environment 10. In some implementations, the acoustic model 60 indicates the respective positions of various objects within the physical environment 10. In some implementations, the acoustic model 60 indicates the respective sound absorptivity of objects in the physical environment 10. For example, the acoustic model 60 may include sound absorptivity values that characterize the extent to which the respective objects absorb sound. In some implementations, the acoustic model 60 indicates the respective sound reflectivity of objects in the physical environment 10. For example, the acoustic model 60 may include sound reflectivity values that characterize the extent to which the respective objects reflect sound. Figure 1B In the example of , the acoustic model 60 includes the sound absorption / reflection value 22 of the front wall 20 , the sound absorption / reflection value 26 of the side wall 24 , and the sound absorption / reflection value 42 of the bench 40 .
[0023] In some implementations, the system 200 determines the sound absorbance / reflectance value of the object based on an image analysis of the object. For example, the system 200 identifies the object type of the object and retrieves the sound absorbance / reflectance value of the object type from a data repository storing sound absorbance / reflectance values of various object types. In some implementations, the system 200 identifies the material that the object is made of and the system 200 retrieves the sound absorbance / reflectance value of the material from a data repository storing sound absorbance / reflectance values of various materials.
[0024] In some implementations, the acoustic model 60 indicates corresponding vocalization likelihoods of objects in the physical environment 10. For example, the acoustic model 60 may include vocalization likelihood values that indicate corresponding probabilities that the corresponding objects generate sounds. In some implementations, the system 200 determines the vocalization likelihood of an object based on the object type of the object. For example, the vocalization likelihood of a musical instrument is greater than the vocalization likelihood of a piece of furniture. As another example, the vocalization likelihood of a first musical instrument may be greater than the vocalization likelihood of a second musical instrument, for example, because the first instrument may be a core instrument (e.g., a lead instrument) and the second instrument may be a supporting instrument. In some implementations, the system 200 determines the vocalization likelihood of a person based on the person's body posture. For example, the vocalization likelihood of a person who has begun to move (e.g., shifting in his / her seat, starting a gesture, etc.) may be greater than the vocalization likelihood of a person who is still. Figure 1B In the example of , the acoustic model 60 includes the utterance likelihood value 32 of the piano 30 , the utterance likelihood value 36 of the piano player 34 , and the utterance likelihood value 46 of the audience member 44 .
[0025] In some implementations, the system 200 generates the utterance likelihood values 32, 36, and / or 46 based on historical environmental data about the physical environment 10. In some implementations, the historical environmental data includes an image of the environment and / or audio data corresponding to sounds emitted from the physical environment 10. In some implementations, the system 200 uses the historical environmental data to identify objects, people, and / or locations within the physical environment 10 that have historically generated sounds. For example, the historical environmental data may indicate that the person sitting on the bench 40 has not historically spoken or sung a song while the piano 30 is playing. Therefore, the system 200 may set the utterance likelihood value 46 of the audience member 44 to a relatively low number. For example, the system 200 may set the utterance likelihood value 46 to less than a utterance likelihood threshold, such as 0.05, indicating that the likelihood of the audience member 44 uttering is less than five percent. As another example, the system 200 may set the utterance likelihood value 36 to a medium value, such as 0.30, based on historical environmental data indicating that the pianist 34 sings about thirty percent of the time when playing the piano 30. As yet another example, the system 200 may set the utterance likelihood value 32 to a relatively high value, such as 0.85, based on historical environmental data indicating that the piano 30 is played when the player 34 sits in front of the piano 30 approximately eighty-five percent of the time.
[0026] In various implementations, the system 200 determines a pickup pattern 52 associated with the microphone 50. The pickup pattern 52 varies with the directivity of the microphone 50. The microphone 50 may be a unidirectional microphone, a bidirectional microphone, or an omnidirectional microphone. Figure 1BIn the example of , microphone 50 is a unidirectional microphone, and pickup pattern 52 is a cardioid pickup pattern. Thus, microphone 50 is suitable for capturing sound emanating from a single source located in front of microphone 50. Pickup pattern 52 makes microphone 50 less suitable for capturing sound originating from the sides of microphone 50, and less suitable for capturing sound originating from behind microphone 50. Other exemplary pickup patterns include hypercardioid, figure-of-eight, semi-cardioid, omnidirectional, and supercardioid.
[0027] See also Figure 1C In various implementations, the system 200 determines a location for placing the microphone 50 within the physical environment 10 based on the acoustic model 60 of the physical environment 10 and the pickup pattern 52 of the microphone 50. In some implementations, the electronic device 100 displays a display indicating Figure 1A and Figure 1B An environmental representation 110 of the physical environment 10 shown in FIG. Figure 1C In the example of FIG. 1 , the environment representation 110 is a top view of the physical environment 10. Figure 1C , environment representation 110 includes representations of objects in physical environment 10. To this end, environment representation 110 includes front wall representation 120 representing front wall 20, side wall representation 124 representing side wall 24, piano representation 130 representing piano 30, piano player representation 134 representing piano player 34, bench representation 140 representing bench 40, and audience member representation 144 representing audience member 44. In some implementations, environment representation 110 includes a see-through of physical environment 10, and representations 120, 124, 130, 134, 140, and 144 include see-through representations of front wall 20, side wall 24, piano 30, piano player 34, bench 40, and audience member 44, respectively.
[0028] like Figure 1C As shown, the electronic device 20 displays a visual indicator 150 that indicates the location that the system 200 has automatically determined for placing the microphone 50. In some implementations, the visual indicator 150 includes an augmented reality (AR) element of the electronic device 100 overlaid on a pass-through of the physical environment 10. In some implementations, the AR element includes an AR microphone. Figure 1CIn the example of , the system 200 determines that the microphone 50 is to be placed adjacent to and to the left of the piano 30. In some implementations, the system 200 automatically selects the position adjacent to the piano 30 so that the microphone 50 can appropriately capture the sound generated by the piano 30. In some implementations, the system 200 selects the position adjacent to the piano 30 because the utterance likelihood value 32 of the piano 30 is greater than the utterance likelihood value 36 of the pianist 34 and the utterance likelihood value 46 of the audience member 44. More generally, in various implementations, the system 200 recommends a placement position adjacent to the object having the greatest likelihood of generating sound.
[0029] exist Figure 1C In the example of FIG. 2 , the system 200 recommends placing the microphone 50 on the left side of the piano 30 instead of placing the microphone 50 on the right side of the piano 30. In some implementations, the system 200 recommends placing the microphone 50 on the left side of the piano 30 based on the sound absorptivity / reflectivity value 22 of the front wall 20 and the sound absorptivity / reflectivity value 26 of the side wall 24. For example, the system 200 determines that the left side of the piano 30 may produce better audio capture than the right side of the piano 30 due to fewer reflections from the side wall 24. More generally, in various implementations, the system 200 determines placement positions that may result in a direct-to-reverberation sound energy ratio (DRR) that is less than a threshold DRR. For example, the system 200 determines candidate placement positions and corresponding DRRs for the candidate placement positions. In this example, the system 200 selects a particular one of the candidate placement positions that results in a minimum DRR value.
[0030] In some implementations, visual indicator 150 indicates a direction for pointing microphone 50. Pickup pattern 52 of microphone 50 indicates the directivity of microphone 50, and system 200 automatically determines a direction for pointing microphone 50 based on the directivity indicated by pickup pattern 52. Figure 1C In the example of , the pickup pattern 52 of the microphone 50 indicates that the microphone 50 is a unidirectional microphone. Therefore, the visual indicator 150 suggests that the microphone 50 be pointed downward toward the piano 30 so that the microphone 50 can properly capture the sound generated by the piano 30. In some implementations, the visual indicator 150 has a shape that corresponds to (e.g., matches or is similar to) the shape of the pickup pattern 52. In some implementations, the visual indicator 150 has a conical shape. For example, the visual indicator 150 can be an AR cone with a mouth that opens toward the piano 30.
[0031] See also Figure 1D, the electronic device 100 detects a user input 160 involving the visual indicator 150. In some implementations, the electronic device 100 displays the visual indicator 150 on a touch screen display, and the user input 160 includes a tap at a display location corresponding to the visual indicator 150. In some implementations, the electronic device 100 displays the visual indicator 150 on a transparent display, and the user input 160 includes a three-dimensional gesture pointing to a display location corresponding to the visual indicator 150.
[0032] See also Figure 1E In some implementations, the electronic device 100 displays a menu 162 in response to detecting a user input 160 involving the visual indicator 150. In some implementations, the menu 162 includes various options related to the placement location indicated by the visual indicator 150. Figure 1E In the example of , the menu 162 includes a first option 162a for accepting the recommended position and updating the acoustic model 60, a second option 162b for recommending a new position, a third option 162c for playing a simulation of sound capture from the recommended position, and a fourth option 162d for recommending a position for another microphone.
[0033] exist Figure 1E In the example of , the electronic device 100 detects a user input 164 related to the first option 162a. The user input 164 corresponds to a request to accept the placement location indicated by the visual indicator 150 and update the acoustic model 60. Figure 1F , in response to detecting user input 164 (in Figure 1E ), the electronic device 100 accepts the recommended position and updates the acoustic model 60 to generate an updated acoustic model 60' indicating the microphone position 62 indicated by the visual indicator 150. Figure 1F As can be seen in FIG. 1 , the electronic device 100 changes the visual appearance of the visual indicator 150 from using a dashed line to using a solid line in order to indicate that the user 102 has accepted the recommended location for placing the microphone 50 .
[0034] See also Figure 1D , the electronic device 100 detects a user input 166 related to the second option 162b. The user input 166 corresponds to a request to determine an alternative location for placing the microphone 50. Figure 1H , in response to detecting user input 166 (in Figure 1G ), the electronic device 100 automatically determines another location for placing the microphone 50 and displays a visual indicator 150' at the other location. Figure 1H In the example of FIG. 1 , another location indicated by visual indicator 150′ is on the right side of piano 30. User 102 can tap visual indicator 150′ to trigger electronic device 100 to display the same Figure 1G Menu 162 is shown as a similar menu and obtains a recommendation for yet another location.
[0035] See also Fig. 1I , the electronic device 100 detects a user input 168 relating to the third option 162c. The user input 168 corresponds to a request to play a simulation of what the captured audio would sound like if the microphone 50 was placed at the position indicated by the visual indicator 150. Figure 1J , in response to detecting user input 168 (in Fig. 1I ), the electronic device 100 plays an analog audio output 172, which indicates if the microphone 50 is placed with Fig. 1I 1. At the simulated microphone position 170 corresponding to the position indicated by the visual indicator 150 shown in FIG. 1, what the audio captured by the microphone 50 sounds like.
[0036] See also Figure 1K In some implementations, the system 200 suggests a plurality of potential locations for placing the microphone 50, and the user 102 selects one of the potential locations suggested by the system 200. Figure 1K In the example of FIG. 1 , the electronic device 100 displays respective visual indicators for a first suggested microphone position 150a and a second suggested microphone position 150b. The electronic device 100 may display a menu 174 that allows the user 102 to further explore the suggested microphone positions 150a and 150b. The menu 174 may include a first option 174a for playing a simulated sound capture from the first suggested microphone position 150a, a second option 174b for accepting the first suggested microphone position 150a, a third option 174c for playing a simulated sound capture from the second suggested microphone position 150b, a fourth option 174d for accepting the second suggested microphone position 150b, and a fifth option 174e for suggesting additional positions.
[0037] Selecting the first option 174a triggers the electronic device 100 to play an audio clip indicating how audio captured from the first suggested microphone position 150a will sound. Selecting the second option 174b triggers the system 200 to update the acoustic model 60 to indicate that the microphone 50 has been placed at the first suggested microphone position 150a. Selecting the third option 174c triggers the electronic device 100 to play an audio clip indicating how audio captured from the second suggested microphone position 150b will sound. Selecting the fourth option 174d triggers the system 200 to update the acoustic model 60 to indicate that the microphone 50 has been placed at the second suggested microphone position 150b. Selecting the fifth option 174e triggers the system 200 to determine suggested microphone positions other than the first suggested microphone position 150a and the second suggested microphone position 150b.
[0038] Figure 1L Another environment representation 110' is shown including representations of additional physical objects and people. Figure 1L In the example of , other environment representations 110' include a drum set representation 180 representing a physical drum set, a drummer representation 182 representing a drummer playing the physical drum set, a singer representation 184 representing a person singing, an existing microphone representation 186 representing a physical microphone placed in front of the person singing, a guitar representation 188 representing a physical guitar, and a guitarist representation 190 representing a guitarist playing the guitar.
[0039] exist Figure 1L In the example of , the system 200 determines the location in front of the guitar as the most appropriate location for placing the microphone 50. The system 200 may determine the location in front of the guitar as the appropriate location for placing the microphone 50 based on the expected amplitude of the sounds originating from the guitar, the drum set, and the piano. For example, the system 200 determines that the sounds originating from the guitar are expected to have lower amplitudes than the sounds originating from the drum set and the piano. Therefore, in this example, the system 200 determines that the sounds originating from the guitar require the greatest amplification. Figure 1L As shown, the electronic device 100 displays a visual indicator 192 to indicate that the microphone 50 will be placed in front of the guitar. More generally, in various implementations, the system 200 determines the placement of the new microphone based on the placement of the existing microphones. In some implementations, the acoustic model 60 indicates the placement of the existing microphones. In addition, in some implementations, the acoustic model 60 indicates the expected amplitude of the sound from various objects in the physical environment 10.
[0040] Figure 2 is a block diagram of a system 200 according to some implementations. In some implementations, the system 200 resides in Figures 1A to 1L The electronic device 100 is shown. In various implementations, the system 200 includes a data acquirer 210 , an acoustic model determiner 220 , a microphone placement position determiner 230 , and a content presenter 240 .
[0041] In various implementations, the data acquirer 210 acquires data related to the physical environment (e.g., Figures 1A to 1LIn some implementations, the environmental data 212 includes image data 212a captured by an image sensor (e.g., a collection of one or more images captured by a visible light camera and / or an infrared (IR) camera). In some implementations, the environmental data 212 includes depth data 212b captured by a depth sensor (e.g., a depth camera). In some implementations, the environmental data 212 includes a visual grid 212c of the physical environment. The visual grid indicates the size and / or location of physical objects in the physical environment. In some implementations, the visual grid 212c indicates objects such as musical instruments and furniture (e.g., Figure 1A 20, side walls 24, piano 30, and bench 40). In some implementations, visual grid 212c indicates the size and / or location of inanimate objects such as musicians playing instruments and audience members listening to the music being played (e.g., Figure 1A The size and / or position of persons such as the pianist 34 and audience member 44 shown in the figure. In some implementations, the data acquirer 210 generates the visual grid 212c based on the image data 212a and / or the depth data 212b.
[0042] In some implementations, the data acquirer 210 obtains information about microphones (e.g., Figures 1A to 1L 50 shown). In some implementations, the microphone information 214 includes a microphone identifier 214a that identifies the microphone. The microphone identifier 214a may include a serial number, a model number, a barcode number, etc. that can be used to find the pickup pattern (e.g., directional pattern) of the microphone. In some implementations, the microphone information 214 indicates a microphone type 214b. The microphone type 214b may indicate the pickup pattern of the microphone. For example, the microphone type 214b may indicate whether the microphone is a unidirectional microphone, a bidirectional microphone, or an omnidirectional microphone. In some implementations, the microphone information 214 indicates a pickup pattern 214c of the microphone. For example, the microphone information 214 may indicate whether the microphone has a cardioid pickup pattern (e.g., a cardioid pickup pattern), a high cardioid pickup pattern, a figure-of-eight pickup pattern, a semi-cardioid pickup pattern, an omnidirectional pickup pattern, a supercardioid pickup pattern, or a pickup pattern with another shape. In some implementations, the data acquirer 210 uses the microphone identifier 214a and / or the microphone type 214b to look up the pickup pattern 214c from a data repository storing pickup patterns of various microphones.
[0043] In various implementations, the acoustic model determiner 220 determines an acoustic model 222 of the physical environment based on the environmental data 212 about the physical environment. The acoustic model 222 indicates a set of one or more acoustic characteristics 224 about the physical environment. The acoustic characteristics 224 indicate how sound propagates within the physical environment. In some implementations, the acoustic characteristics 224 include a sound absorptivity value 224a, which indicates whether the corresponding object absorbs sound and the degree to which the corresponding object absorbs sound. In some implementations, the acoustic characteristics 224 include a sound reflectivity value 224b, which indicates whether the corresponding object reflects sound and the degree to which the corresponding object reflects sound. For example, the sound absorptivity value 224a and the sound reflectivity value 224b include Figure 1B Sound absorption / reflectivity values shown are 22, 26 and 42.
[0044] In some implementations, the acoustic model determiner 220 determines the sound absorption value 224a and / or the sound reflectivity value 224b based on the environment data 212. For example, the acoustic model determiner 220 performs image analysis on the image data 212a to identify the material of the object, and looks up the sound absorption coefficient and / or the sound reflectivity coefficient of the material in a data repository storing sound absorption coefficients and / or sound reflectivity coefficients of various materials.
[0045] In some implementations, the acoustic characteristics 224 include a voicing likelihood value 224c, which indicates the corresponding likelihood of the corresponding object generating the sound. Figure 1B The utterance likelihood values shown are 32, 36, and 46. In some implementations, the acoustic model determiner 220 determines the utterance likelihood value 224c based on historical audio data indicating the number of times the corresponding object generated a sound within a given time period.
[0046] In various implementations, a microphone placement position determiner 230 (hereinafter referred to as "placement determiner 230") determines a suggested microphone placement position 232 (hereinafter referred to as "placement position 232") based on an acoustic model 222 of the physical environment and a pickup pattern 214c of the microphone. In some implementations, the acoustic model 222 indicates an object that is most likely to generate a sound, and the placement determiner 230 selects the placement position 232 as a position that is closest to the object. For example, the placement determiner 230 selects the placement position 232 that is adjacent to the object having the largest utterance likelihood value 224c.
[0047] In some implementations, the acoustic model 222 indicates surfaces that reflect sound in excess of a threshold amount, and the placement determiner 230 selects the placement location 232 to be close to a possible sound source and away from overly reflective surfaces in order to avoid capturing echoes. For example, the placement determiner 230 selects the placement location 232 to be close to an object having a maximum utterance likelihood value 224c and away from an object having a maximum sound reflectivity value 224b.
[0048] In some implementations, the acoustic model 222 indicates surfaces that absorb sound above a threshold amount, and the placement determiner 230 selects the placement location 232 to be close to a possible sound source and a surface that absorbs sound so as to reduce captured echoes. For example, the placement determiner 230 selects the placement location 232 to be close to an object having a maximum utterance likelihood value 224c and relatively close to an object having a maximum sound absorptivity value 224a.
[0049] In some implementations, the placement determiner 230 generates a set of candidate placement locations and selects a placement location 232 from the set of candidate placement locations. In some implementations, the placement determiner 230 generates corresponding quality scores for the candidate placement locations and selects one of the candidate placement locations based on the quality scores. In some implementations, the quality scores of the candidate placement locations indicate an expected quality of sound captured from the candidate placement locations. In such implementations, the placement determiner 230 selects the candidate placement location with the largest quality score as the placement location 232 of the microphone. In some implementations, the quality score includes an expected DRR value. In such implementations, the placement determiner 230 selects the candidate placement location with the smallest DRR value.
[0050] In some implementations, the placement determiner 230 utilizes a machine learning model to determine the placement location 232. The machine learning model may include a collection of one or more neural networks that accepts the acoustic model 222 and the pickup pattern 214c as inputs and outputs the placement location 232 of the microphone.
[0051] In some implementations, the content presenter 240 presents an environment representation 242 that represents the physical environment in which the microphone will be placed. Figure 1C The environment representation 110 is shown. In some implementations, the environment representation 242 includes a map of the physical environment. In some implementations, the environment representation 242 includes an overhead view of the physical environment. In some implementations, the environment representation 242 includes a pass-through (e.g., an optical pass-through or a video pass-through) of the physical environment.
[0052] In some implementations, the content presenter 240 overlays a visual indicator 244 on top of the environment representation 242. The visual indicator 244 indicates the placement location 232 for placing the microphone. In some implementations, the environment representation 242 is a pass-through of the physical environment, and the visual indicator 244 is an AR element overlaid onto the pass-through of the physical environment. In some implementations, the visual indicator 244 indicates a direction for pointing the microphone. The placement determiner 230 can determine the direction for pointing the microphone based on the pickup pattern 214c of the microphone.
[0053] Figure 3 is a flowchart representation of a method 300 for indicating a microphone placement location. In various implementations, the method 300 comprises Figures 1A to 1L The electronic device 100 and / or Figures 1A to 1L The system 200 shown is executed. In some specific implementations, the method 300 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the method 300 is performed by a processor executing a code stored in a non-transitory computer readable medium (e.g., a memory).
[0054] As represented by block 310, in various implementations, method 300 includes obtaining an acoustic model of an environment. In some implementations, the environment includes a physical environment. In some implementations, the environment includes an enclosed space, such as a room, a banquet hall, a stage, etc. In some implementations, the environment includes an open space, such as a park, a backyard of a home, an open amphitheater, etc. In some implementations, the device receives the acoustic model. Alternatively, in some implementations, the device generates the acoustic model, for example, based on environmental data associated with the physical environment. For example, Figure 2 As shown, the acoustic model determiner 220 generates an acoustic model 222 based on the environmental data 212 .
[0055] As represented by block 310a, in some implementations, obtaining the acoustic model includes obtaining a visual grid indicating the size of the environment and the placement of objects in the environment, and generating the acoustic model based on the visual grid of the environment. Figure 2 As shown, in some implementations, the data acquirer 210 obtains the visual grid 212 c , and the acoustic model determiner 220 generates the acoustic model 222 using the visual grid 212 c .
[0056] In some implementations, the acoustic model indicates a set of one or more acoustic properties of the environment. As represented by block 310b, in some implementations, the set of one or more acoustic properties indicates how sound propagates within the environment. In some implementations, the acoustic properties indicate which objects absorb sound and the degree to which the objects absorb sound. For example, Figure 2As shown, acoustic characteristic 224 includes sound absorption rate value 224a. In some embodiments, the acoustic characteristic indicates which objects reflect sound and the degree to which the objects reflect sound. Figure 2 As shown, acoustic characteristics 224 include sound reflectivity values 224b. In some implementations, the acoustic characteristics indicate which objects are likely to generate sound and the expected amplitude of the sound. Figure 2 As shown, acoustic characteristics 224 include utterance likelihood value 224c.
[0057] In some implementations, the acoustic model indicates the corresponding positions of objects that can generate sound. For example, in some implementations, the acoustic model indicates the corresponding positions of musical instruments in the environment. Figure 1B As shown, the acoustic model 60 indicates the position of the piano 30 and the sound likelihood value 32 of the piano 30. In some implementations, the acoustic model indicates the corresponding position of objects that can absorb or reflect sound. For example, in some implementations, the acoustic model indicates the corresponding absorption value and / or the corresponding reflectivity value of the object. Figure 1B As shown, the acoustic model 60 includes sound absorption / reflectivity values 22, 26, and 42 for the front wall 20, the side wall 24, and the bench 40, respectively. In some implementations, the acoustic properties vary with the material properties of the environment. Figure 1B As discussed, the sound absorption / reflectivity values 22, 26, and 42 may vary depending on the materials that the front wall 20, the side walls 24, and the bench 40 are respectively composed of. In some implementations, the acoustic model indicates the corresponding positions of existing microphones in the physical environment. For example, Figure 1L As shown, the environment representation 110 ′ includes an existing microphone representation 186 , and the acoustic model 60 may indicate the locations of the existing microphones.
[0058] As represented by block 320, in various implementations, method 300 includes determining a placement of a microphone within an environment based on a set of one or more acoustic characteristics of the environment and a pickup pattern of the microphone. Figure 1C As shown, the system 200 determines the placement of the microphone 50 based on the acoustic model 60 and the pickup pattern 52 of the microphone 50. As another example, Figure 2 As shown, the placement determiner 230 determines the placement position 232 based on the acoustic characteristics 224 indicated by the acoustic model 222 and the pickup pattern 214c.
[0059] As represented by block 320a, in some implementations, the set of one or more acoustic characteristics indicates a corresponding position of an object capable of generating sound. In such implementations, determining a placement position for a microphone includes selecting a placement position based on the corresponding position of the object capable of generating sound so as to capture the sound that the object can generate. In some implementations, the acoustic model indicates the position of a musical instrument. For example, as described with respect to Figure 2 As discussed, the acoustic model 60 may indicate the location of the piano 30 within the physical environment 10 and the sound likelihood value 32 of the piano 30. In some implementations, the device selects a placement location that is adjacent to an object that has the greatest likelihood of generating a sound. Figure 1C As shown, system 200 may recommend placing microphone 50 adjacent to piano 30 because piano 30's sound likelihood value 32 may be greater than piano player 34 and audience member 44's sound likelihood values 36 and 46, respectively. In some implementations, user 102 may not know which object in the physical environment may generate the sound that needs to be captured. Therefore, automatically selecting a placement location based on the location of an object that can generate sound reduces the need for user 102 to guess which object may generate the sound that needs to be captured.
[0060] As represented by block 320b, in some implementations, the set of one or more acoustic characteristics indicates a location of a musical instrument within the environment. In such implementations, determining a placement position for a microphone includes selecting a placement position based on the location of the musical instrument within the environment so as to capture the sound generated by the musical instrument. In some implementations, the device recommends a placement position adjacent to the musical instrument so as to capture the sound being generated by the musical instrument. For example, Figure 1C As shown, system 200 recommends placing microphone 50 adjacent to piano 30. In some implementations, the physical environment includes multiple musical instruments, and the device recommends placing the microphone adjacent to the instrument that generates the sound with the lowest amplitude. Figure 1L As shown, system 200 recommends placing microphone 50 in front of a guitar rather than a drum kit or piano. In some implementations, the physical environment includes multiple musical instruments, and the device prioritizes the centrally located instrument. For example, see Figure 1L , system 200 may recommend placing microphone 50 adjacent to a guitar rather than a piano or drum set because a guitar is closer to the center of the physical environment than a piano or drum set.
[0061] As represented by block 320c, in some implementations, the set of one or more acoustic characteristics indicates a position of a second microphone having a second pickup pattern. In such implementations, determining the placement of the microphone includes selecting the placement based on the position of the second microphone so as to reduce overlap between the pickup pattern of the microphone and the second pickup pattern of the second microphone. For example, Figure 1LAs shown, system 200 recommends a placement position for microphone 50 based on the positions of existing microphones. Automatically taking into account the positions of existing microphones allows the device to suggest a placement position that reduces overlap between two microphones.
[0062] As represented by block 320d, in some implementations, the set of one or more acoustic characteristics varies with material properties of the environment. In some implementations, the material properties indicate materials present in the environment. For example, the material properties may indicate the materials of walls, floors, and / or furniture in the physical environment. In such implementations, determining the placement of the microphone includes determining the placement of the microphone based on the material properties of the environment. Figure 1B As discussed, the system 200 may determine the sound absorption / reflectivity values 22, 26, and 42 based on the materials that the front wall 20, the side walls 24, and the bench 40 are respectively composed of. Figure 1C As described, system 200 may recommend placing microphone 50 on the left side of piano 30, for example, because sidewall 24 may reflect too much sound, and placing the microphone on the right side of piano 30 may result in excessive echo being captured by microphone 50. Automatically determining microphone placement based on material properties of the environment helps ensure that the quality of the audio data captured by the microphone meets a threshold level of quality.
[0063] As represented by block 320e, in some implementations, the placement position allows the microphone to detect sounds from a threshold portion of the environment. In some implementations, the device recommends a placement position that allows the microphone to appropriately capture sounds being generated from a threshold portion of the environment. For example, the device may recommend placing the microphone at a position that allows the microphone to capture sounds originating from 80% of the physical environment.
[0064] In some implementations, the placement position results in a DRR that is less than a threshold direct-to-reverberation sound energy ratio (DRR). Reducing the DRR value results in audio capture with less echo. In some implementations, the device generates a set of candidate placement positions and determines corresponding expected DRR values for the candidate placement positions. In such implementations, the device selects the candidate placement position with the smallest expected DRR value. More generally, in various implementations, the placement position results in a detected sound quality that meets a threshold sound quality. In some implementations, the device generates a set of candidate placement positions and determines corresponding expected quality values for audio data captured from the candidate placement positions. In such implementations, the device selects the candidate placement position with the largest expected quality value.
[0065] As represented by block 320f, in some implementations, the pickup pattern indicates a directional pattern of the microphone. For example, the pickup pattern may indicate whether the microphone is unidirectional, bidirectional, omnidirectional, cardioid, sub-cardioid, etc. In some implementations, the method 300 includes determining an identifier (e.g., a model or serial number) that identifies the microphone, and retrieving the pickup pattern associated with the identifier from a data repository storing pickup patterns for a plurality of microphones.
[0066] As represented by block 320g, in some implementations, determining the placement position includes automatically selecting a placement position from a plurality of candidate positions to place the microphone based on corresponding scores associated with the plurality of candidate positions. In some implementations, the corresponding scores indicate an expected quality of audio capture from the candidate positions, and the device selects the candidate position associated with the highest expected quality. In some implementations, the corresponding scores indicate an expected DRR value for the candidate positions, and the device selects the candidate position associated with the lowest DRR value. Automatically selecting the placement position reduces the need for a user of the device (e.g., an audio engineer) to perform an acoustic analysis of the physical environment in order to manually select a placement position for the microphone.
[0067] As represented by block 320h, in some implementations, determining the placement location includes determining a plurality of candidate locations for placing the microphone, displaying corresponding visual indicators of the plurality of candidate locations for placing the microphone on a display, and detecting a user selection of one of the corresponding visual indicators of the plurality of candidate locations displayed on the display. Figure 1K As shown, electronic device 100 displays visual indicators of two suggested locations from which user 102 can select.
[0068] As represented by block 330, in various implementations, method 300 includes displaying a representation of the environment on a display and a visual indicator overlaid onto the representation of the environment to indicate the placement of the microphone. Figure 1C As shown, the electronic device 100 displays a visual indicator 150 for indicating the position where the microphone 50 can be placed. Displaying the visual indicator is used as a guide for placing the microphone so as to properly capture the sound originating from the physical environment. Displaying the visual indicator reduces the need for the user (e.g., audio engineer) of the device to determine the placement of the microphone based on trial and error. The incorrect placement of the microphone in the physical environment can lead to inappropriate audio capture (e.g., the sound is not captured, the audio capture has too many echoes, etc.). The playback of inappropriate audio capture tends to detract from the user experience of the device and unnecessarily drain the battery of the electronic device. Therefore, automatically determining the placement of the microphone and displaying the visual indicator of the placement position tends to enhance the user experience of the device and save power by allowing the user to playback the appropriate audio capture.
[0069] As represented by block 330a, in some implementations, the visual indicator includes a balloon that matches the shape of the microphone's pickup pattern. Overlaying the balloon on the representation of the environment provides an indication of the directionality of the microphone and further allows the user to point the microphone in the correct direction.
[0070] As represented by block 330b, in some implementations, method 300 includes generating a simulation of sound capture by the simulated microphone at the placement location, and providing an option to play a sound recording indicating the results of the simulation. In some implementations, method 300 includes playing back the sound recording that simulates the sound capture by the microphone from the placement location. For example, Fig. 1I and Figure 1J As shown, menu 162 includes a third option 162c for playing a simulation of a sound capture from the location indicated by visual indicator 150. Fig. 1I and Figure 1J In the example of , user selection of the third option 162c triggers the electronic device 100 to playback the simulated audio output 172. Enabling the user to play a simulation of sound capture before the user places the microphone in the recommended position allows the user to evaluate whether the recommended position appropriately captures sounds originating from the physical environment.
[0071] As represented by block 330c, in some implementations, the representation of the environment includes a transparent representation of the environment, and the visual indicator is overlaid onto the transparent representation of the environment. Figure 1C , the environment representation 110 may include Figure 1A and Figure 1B A see-through representation of the physical environment 10 is shown, and the visual indicator 150 may include an AR element overlaid onto the see-through representation of the physical environment 10 .
[0072] Figure 4 is a block diagram of a device 400 according to some implementations. In some implementations, the device 400 implements Figures 1A to 1L The electronic device 100 and / or Figures 1A to 1L 2. Although some specific features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some implementations, the device 400 includes one or more processing units (CPUs) 401, a network interface 402, a programming interface 403, a memory 404, one or more input / output (I / O) devices 408, and one or more communication buses 405 for interconnecting these and various other components.
[0073] In some implementations, a network interface 402 is provided to, among other purposes, establish and maintain a metadata tunnel between a cloud-hosted network management system and at least one private network including one or more compatible devices. In some implementations, one or more communication buses 405 include circuits for interconnecting and controlling communications between system components. Memory 404 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 404 optionally includes one or more storage devices located remotely from one or more CPUs 401. Memory 404 includes non-transitory computer-readable storage media.
[0074] In some implementations, the one or more I / O devices 408 include a display Figure 1C The environment shown represents a display 110. In some implementations, the display comprises an extended reality (XR) display. In some implementations, the display comprises an opaque display. Alternatively, in some implementations, the display comprises an optically transparent display. In some implementations, one or more I / O devices 408 include a display for capturing Figure 2 The environmental sensors of the environmental data 212 are shown. For example, one or more I / O devices 408 include an image sensor (e.g., a visible light camera and / or an infrared camera) for capturing image data 212a and / or an image sensor (e.g., a visible light camera and / or an infrared camera) for capturing Figure 2 Depth data 212b is shown as being from a depth sensor (eg, a depth camera).
[0075] In some embodiments, the memory 404 or a non-transitory computer-readable storage medium of the memory 404 stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 406, a data acquirer 210, an acoustic model determiner 220, a placement determiner 230, and a content presenter 240.
[0076] In various specific implementations, the data acquirer 210 includes a Figure 2 Instructions 210a and heuristics and metadata 210b for determining the environment data 212 and / or microphone information 214 are shown. In some implementations, the acoustic model determiner 220 includes an acoustic model for determining the environment (e.g., Figure 1B The acoustic model 60 shown and / or Figure 2 In some implementations, the placement determiner 230 includes instructions 220a for determining the placement of the microphone based on the acoustic model and the pickup pattern of the microphone (e.g., Figure 2 The placement position 232 shown and / or Figure 1C In some implementations, the content presenter 240 includes instructions 230a for displaying a visual indicator indicating the placement location of the microphone (e.g., Figure 1C The visual indicator 150 shown, Figure 1H The visual indicator 150 ′ shown, Figure 1K Visual indicators of suggested microphone locations 150a and 150b are shown, Figure 1L The visual indicator 192 shown and / or Figure 2 Instructions 240a of the system 240 and heuristics and metadata 240b are shown as visual indicators 244.
[0077] It should be understood that Figure 4 Serves as a functional description of various features that may be present in a particular implementation, as opposed to a structural schematic diagram of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 4 Some functional blocks shown separately in the figure may be implemented as a single block, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of blocks and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a specific specific implementation.
[0078] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.
Claims
1. A method comprising: At a device including a display, an environmental sensor, a non-transitory memory, and one or more processors coupled to the display, the environmental sensor, and the non-transitory memory: obtaining an acoustic model of an environment, wherein the acoustic model indicates a set of one or more acoustic characteristics of the environment; determining a placement of the microphone within the environment based on the set of one or more acoustic characteristics of the environment and a pickup pattern of the microphone; as well as A representation of the environment is displayed on the display along with a visual indicator overlaid onto the representation of the environment to indicate the placement location of the microphone.
2. The method according to claim 1, wherein obtaining the acoustic model comprises: obtaining a visual grid indicating dimensions of the environment and placement of objects within the environment; as well as The acoustic model is generated based on the visual mesh of the environment.
3. The method according to any one of claims 1 and 2, wherein the set of one or more acoustic characteristics indicates how sound propagates within the environment; and Wherein determining the placement location of the microphone includes selecting the placement location based on how sound propagates within the environment.
4. The method according to any one of claims 1 to 3, wherein the set of one or more acoustic characteristics indicates a respective position of an object capable of generating sound; and Determining the placement position of the microphone includes selecting the placement position based on the corresponding position of the object capable of generating sound so as to capture the sound that the object can generate.
5. A method according to any one of claims 1 to 4, wherein the set of one or more acoustic characteristics indicates a location of a musical instrument within the environment; and Wherein determining the placement position of the microphone includes selecting the placement position based on the position of the musical instrument within the environment so as to capture sound generated by the musical instrument.
6. The method of any one of claims 1 to 5, wherein the set of one or more acoustic characteristics indicates a position of a second microphone having a second pickup pattern; and Wherein determining the placement position of the microphone includes selecting the placement position based on the position of the second microphone so as to reduce overlap between the pickup pattern of the microphone and the second pickup pattern of the second microphone.
7. The method of any one of claims 1 to 6, wherein the set of one or more acoustic properties is a function of material properties of the environment; and Wherein determining the placement position of the microphone comprises determining the placement position of the microphone based on the material properties of the environment.
8. The method of any one of claims 1 to 7, wherein the placement position allows the microphone to detect sound from a threshold portion of the environment.
9. The method of any one of claims 1 to 8, wherein the placement position results in a direct-to-reverberation acoustic energy ratio (DRR) that is less than a threshold DRR.
10. The method of any one of claims 1 to 9, wherein the placement position results in a detected sound quality that meets a threshold sound quality.
11. The method according to any one of claims 1 to 10, wherein the pickup pattern is indicative of a directivity pattern of the microphone.
12. The method according to any one of claims 1 to 11, further comprising: determining an identifier identifying the microphone; as well as The pickup pattern associated with the identifier is retrieved from a data repository storing pickup patterns of a plurality of microphones.
13. The method according to any one of claims 1 to 12, wherein determining the placement position comprises: The placement location is automatically selected from a plurality of candidate locations for placing the microphone based on respective scores associated with the plurality of candidate locations.
14. The method according to any one of claims 1 to 13, wherein determining the placement position comprises: determining a plurality of candidate locations for placing the microphone; displaying on the display respective visual indicators of the plurality of candidate locations for placing the microphone; as well as A user selection of one of the corresponding visual indicators of the plurality of candidate positions displayed on the display is detected.
15. The method according to any one of claims 1 to 14, further comprising: A simulation is generated that simulates sound capture by the microphone at the placement location, and an option is provided to play a sound recording indicative of a result of the simulation.
16. The method according to any one of claims 1 to 14, further comprising: A sound recording is played back, the sound recording simulating sound capture by the microphone from the placement location.
17. The method of any one of claims 1 to 16, wherein the representation of the environment comprises a transparent representation of the environment, and the visual indicator is overlaid onto the transparent representation of the environment.
18. A device comprising: monitor; Environmental sensors; one or more processors; non-transitory memory; and One or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to perform any of the methods according to claims 1 to 17.
19. A non-transitory memory storing one or more programs which, when executed by one or more processors of a device having an environmental sensor and a display, cause the device to perform any of the methods of claims 1 to 17.
20. An apparatus comprising: monitor; Environmental sensors; one or more processors; non-transitory memory; and Means for causing the device to perform any of the methods according to claims 1 to 17.
