Determination of acoustic filters in conjunction with local effects of room patterns

Abstract

The invention relates to determination of acoustic filters in conjunction with local effects of room patterns. Determination of an acoustic filter for incorporating a local effect of a room pattern within a target area is presented herein. A model portion of the target region is determined based on a three-dimensional virtual representation of the target region. In some embodiments, a model is selected from a set of candidate models. A room pattern of the target area is determined based on the shape and / or size of the model. Room pattern parameters are determined based on at least one of the room patterns and a location of the user within the target area. The room mode parameters describe an acoustic filter that, when applied to the audio content, simulates acoustic distortions at a user location and at frequencies associated with the at least one room mode. An acoustic filter is generated at the headset based on the room mode parameters and is used to present the audio content.

Description

[0001] This application is a divisional application of an application with an application date of April 16, 2020, application number 202080035034.8, and invention name “Determination of an acoustic filter incorporating local effects of room modes”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Application No. 16 / 418,426, filed May 21, 2019, the contents of which are incorporated herein by reference in their entirety for all purposes.

[0004] background

[0005] The present disclosure relates generally to the rendering of audio, and in particular to the determination of acoustic filters for incorporating local effects of room modes.

[0006] A physical area (e.g., a room) can have one or more room modes. Room modes are caused by sound reflected from different room surfaces. Room modes result in antinodes (peaks) and nodes (valleys) in the room's frequency response. The nodes and antinodes of these standing waves cause the loudness of the resonant frequency to be different at different locations in the room. In addition, the effects of room modes are particularly prominent in small rooms such as bathrooms, offices, and small conference rooms. Traditional virtual reality systems do not take into account the room modes associated with a particular virtual reality environment. They typically rely on geometric acoustic simulations that are unreliable at low frequencies or artistic renderings that are not related to the physical modeling of the environment. As a result, the audio presented by traditional virtual reality systems may lack the realism associated with a virtual reality environment (e.g., a small room).

[0007] Overview

[0008] Embodiments of the present disclosure support methods, computer-readable media, and devices for determining an acoustic filter to incorporate local effects of room modes. In some embodiments, a model of a target area (e.g., a virtual area, a user's physical environment, etc.) is determined in part based on a three-dimensional (3D) virtual representation of the target area. The room mode of the target area is determined using the model. One or more room mode parameters are determined based on at least one of the room modes and the user's position within the target area. The one or more room mode parameters describe an acoustic filter. The acoustic filter can be generated based on the one or more room mode parameters. The acoustic filter simulates acoustic distortion at a frequency associated with at least one room mode. Audio content is presented based in part on the acoustic filter. The audio content is presented so that it sounds like it originates from an object (e.g., a virtual object) in the target area.

[0009] According to the present invention, a device is provided, which includes: a matching module, which is configured to determine a model of a target area based in part on a three-dimensional virtual representation of the target area; a room mode module, which is configured to use the model to determine the room mode of the target area; and an acoustic filter module, which is configured to determine one or more room mode parameters based on at least one room model of the room mode and a position of a user in the target area, wherein the one or more room mode parameters describe an acoustic filter used by a head-mounted device to present audio content to the user, and when applied to the audio content, the acoustic filter simulates acoustic distortion at the user position and at frequencies associated with at least one room mode.

[0010] Optionally, the matching module is configured to determine a model of the target area based in part on the three-dimensional reconstruction of the target area by: comparing the three-dimensional virtual representation with multiple candidate models; and identifying a candidate model among the multiple candidate models that matches the three-dimensional virtual representation as the model of the target area.

[0011] Optionally, the room mode module is configured to determine the room mode of the target area by determining the room mode based on a shape of the model using the model.

[0012] Alternatively, the acoustic distortion describes the amplification as a function of frequency.

[0013] Optionally, the acoustic filter module is configured to transmit parameters describing the acoustic filter to the head-mounted device for rendering the audio content at the head-mounted device.

[0014] According to the present invention, a method is also provided, which includes determining a model of a target area based in part on a three-dimensional virtual representation of the target area; determining a room mode of the target area using the model; and determining one or more room mode parameters based on at least one of the room modes and a position of a user within the target area, wherein the one or more room mode parameters describe an acoustic filter used by a head-mounted device to present audio content to the user, and when applied to the audio content, the acoustic filter simulates acoustic distortion at the user position and at a frequency associated with at least one room mode.

[0015] Optionally, the method further comprises receiving depth information describing at least a portion of the target area from the head mounted device; and generating at least a portion of the three-dimensional reconstruction using the depth information.

[0016] Optionally, determining the model of the target region based in part on the three-dimensional reconstruction of the target region includes comparing the three-dimensional virtual representation with a plurality of candidate models; and identifying a candidate model of the plurality of candidate models that matches the three-dimensional virtual representation as the model of the target region.

[0017] Optionally, the method further includes receiving color image data of at least a portion of the target area; determining material composition of surfaces in the portion of the target area using the color image data; determining attenuation parameters for each surface based on the material composition of the surface; and updating the model using the attenuation parameters for each surface.

[0018] Optionally, determining the room mode of the target area using the model further comprises determining the room mode based on a shape of the model.

[0019] Optionally, the method further comprises transmitting parameters describing the acoustic filter to a head mounted device for rendering audio content at the head mounted device.

[0020] Optionally, the target area is a virtual area. Optionally, the virtual area is different from the user's physical environment. Optionally, the target area is the user's physical environment.

[0021] According to the present invention, a method is also provided, which includes generating an acoustic filter based on one or more room mode parameters, the acoustic filter simulating the acoustic distortion at a user's position within a target area and at a frequency associated with at least one room mode of the target area; and presenting audio content to the user by using the acoustic filter, the audio content sounds originating from an object in the target area and is received at the user's position within the target area.

[0022] Optionally, the acoustic filter comprises a plurality of infinite impulse response filters having a Q value or gain at a modal frequency of at least one room mode. Optionally, the acoustic filter further comprises a plurality of all-pass filters having a Q value or gain at a modal frequency of at least one room mode.

[0023] Optionally, the method further includes sending a room mode query to an audio server, the room mode query including virtual information of the target area and positioning information of the user; and receiving one or more room mode parameters from the audio server.

[0024] Optionally, the method further comprises dynamically adjusting the acoustic filter based on changes in at least one room mode and a user position. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Shown are localized effects of room modes in a room in accordance with one or more embodiments.

[0027] Figure 2 An axial mode, a tangential mode, and an oblique mode of a cubic room are shown in accordance with one or more embodiments.

[0028] Figure 3 is a block diagram of an audio system according to one or more embodiments.

[0029] Figure 4 is a block diagram of an audio server according to one or more embodiments.

[0030] Figure 5 is a flow chart illustrating a process for determining room mode parameters describing an acoustic filter in accordance with one or more embodiments.

[0031] Figure 6 is a block diagram of audio components according to one or more embodiments.

[0032] Figure 7 is a flow chart illustrating a process for rendering audio content based in part on an acoustic filter in accordance with one or more embodiments.

[0033] Figure 8 is a block diagram of a system environment including a head mounted device and an audio server according to one or more embodiments.

[0034] Fig. 9 is a perspective view of a head mounted device including an audio component according to one or more embodiments.

[0035] The accompanying drawings depict embodiments of the present disclosure for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles or benefits of the present disclosure described herein.

[0036] Detailed Description

[0037] Embodiments of the present disclosure may include an artificial reality system or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or content generated in combination with captured (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of them may be presented in a single channel or multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or to be used in other ways in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including head-mounted devices, head-mounted displays (HMDs) connected to a host computer system, stand-alone HMDs, near-eye displays (NEDs), mobile devices or computing systems, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0038] An audio system for determining an acoustic filter to incorporate the local effects of room modes is presented herein. Audio content presented by an audio component is filtered using an acoustic filter so that acoustic distortion (e.g., amplification as a function of frequency and position) caused by room modes associated with a target area of ​​a user can be part of the audio content presented. Note that amplification as used herein can be used to describe an increase or decrease in signal strength. The target area can be a local area occupied by a user or a virtual area. The virtual area can be based on the local area, some other virtual area, or some combination thereof. For example, the local area can be a living room occupied by a user of the audio system, and the virtual area can be a virtual concert venue or a virtual conference room.

[0039] The audio system includes an audio component communicatively coupled to an audio server. The audio component may be implemented on a head mounted device worn by a user. The audio component may request one or more room mode parameters from the audio server (e.g., via a network). The request may include, for example, visual information (depth information, color information, etc.) of at least a portion of a target area, positioning information of a user, positioning information of a virtual sound source, visual information of a local area occupied by a user, or some combination thereof.

[0040] The audio server determines one or more room mode parameters. The audio server uses the information in the request to identify and / or generate a model of the target area. In some embodiments, the audio server develops a 3D virtual representation of at least a portion of the target area based on the visual information of the target area in the request. The audio server selects a model from a plurality of candidate models using the 3D virtual representation. The audio server determines the room mode of the target area by using the model. For example, the audio server determines the room mode based on the shape or size of the model. The room mode may include one or more types of room modes. The types of room modes may include, for example, axial modes, tangential modes, and tilt modes. For each type, the room mode may include a first-order mode, a higher-order mode, or some combination thereof. The audio server determines one or more room mode parameters (e.g., Q factor, gain, amplitude, modal frequency, etc.) based on at least one of the room modes and the user position. The audio server may also determine the room mode parameters using the positioning information of the virtual sound source. For example, the audio server uses the positioning information of the virtual sound source to determine whether the room mode is excited. The audio server may determine that the room mode is not excited based on the virtual sound source being located at the antinode position.

[0041] The room mode parameters describe an acoustic filter that, when applied to the audio content, simulates acoustic distortion at the user's location within the target area. The acoustic distortion may represent an amplification at frequencies associated with at least one room mode. The audio server transmits the one or more room mode parameters to the headset.

[0042] The audio component generates an acoustic filter using one or more room mode parameters from the audio server. The audio component renders the audio content using the generated acoustic filter. In some embodiments, the audio component dynamically detects changes in the user's position and / or changes in the relative position between the user and the virtual object, and updates the acoustic filter based on these changes.

[0043] In some embodiments, the audio content is spatialized audio content. Spatialized audio content is audio content presented in such a way that it sounds like it originates from one or more points in the user's surroundings (e.g., from a virtual object in a target area).

[0044] In some embodiments, the target area may be a local area of ​​the user. For example, the target area is an office where the user sits. When the target area is an actual office, the audio component generates an acoustic filter that causes the rendered audio content to be spatialized in a manner consistent with how a real sound source would sound from a specific location in the office.

[0045] In some other embodiments, the target area is a virtual area presented to the user (e.g., via a head-mounted device). For example, the target area may be a virtual conference room. When the target area is a virtual conference room, the audio component generates an acoustic filter that causes the presented audio content to be spatialized in a manner consistent with how real sound sources sound from specific positioning in the virtual conference room. For example, a user may be presented with virtual content that makes him / her appear to be sitting next to a virtual audience watching a virtual speaker's speech. And the presented audio content modified by the acoustic filter will make it sound to the user as if the speaker is speaking in a conference room, even though the user is actually in an office (an office will have significantly different acoustic properties than a large conference room).

[0046] Figure 1 The local effect of room modes in room 100 is shown according to one or more embodiments. Sound source 105 is located in room 100 and emits sound waves into room 100. The sound waves cause fundamental resonance of room 100, and room modes appear in room 100. Figure 1 The first mode 110 of the room at the first modal frequency and the second mode 120 at the second modal frequency twice the first modal frequency are shown. Figure 1 As shown in FIG. 1 , higher order room modes may also exist in the room 100. Both the first order mode 110 and the second order mode 120 may be axial modes.

[0047] Room modes depend on the shape, size and / or acoustic properties of the room 100. Room modes cause different amounts of acoustic distortion at different locations within the room 100. The acoustic distortion can be a positive amplification (i.e., an increase in amplitude) or a negative amplification (i.e., an attenuation) of the audio signal at the modal frequencies (and multiples of the modal frequencies).

[0048] The first order mode 110 and the second order mode 120 have peaks and valleys at different locations in the room 100 , which results in different amplification levels of the sound waves as a function of frequency and location within the room 100 . Figure 1Three different locations 130, 140, and 150 within the room 100 are shown. At location 130, the first order mode 110 and the second order mode 120 each have a peak. Moving to location 140, both the first order mode 110 and the second order mode 120 are reduced, and the second order mode 120 has a valley. Moving further to location 150, there is a null at the first order mode 110 and a peak at the second order mode 120. Combining the effects of the first order mode 110 and the second order mode 120, the amplification of the audio signal is highest at location 130 and lowest at location 150. Therefore, the sound perceived by the user can vary significantly based on what room they are in and where they are in the room. As described below, a system is described that simulates room modes for a target area occupied by a user, presents audio content to the user taking into account the room modes to provide the user with increased realism.

[0049] Figure 2 Axial modes 210, tangential modes 220, and tilt modes 230 of a cubic room are shown in accordance with one or more embodiments. Room modes are caused by sound reflecting off different room surfaces. Figure 2 The room in the example has a cubic shape and includes six surfaces: four walls, a ceiling, and a floor. There are three types of modes in the room: axial modes 210, tangential modes 220, and oblique modes 230, which are Figure 2 denoted by dashed lines in the figure. Axial modes 210 include resonances between two parallel surfaces of the room. Three axial modes 210 occur in the room: one involving the ceiling and floor, and two each involving a pair of parallel walls. For rooms of other shapes, a different number of axial modes 210 may occur. Tangential modes 220 involve two sets of parallel surfaces, all four walls, or two walls with the ceiling and floor. Oblique modes 230 involve all six surfaces of the room.

[0050] The axial mode 210 is the strongest of the three modes. The tangential mode 220 may be half as strong as the axial mode 210, and the tilted mode 230 may be one-quarter as strong as the axial mode 210. In some embodiments, an acoustic filter that simulates acoustic distortion in a room when applied to audio content is determined based on the axial mode 210. In some other embodiments, the tangential mode 220 and / or the tilted mode 230 are also used to determine the acoustic filter. Each of the axial mode 210, the tangential mode 220, and the tilted mode 230 may occur at a range of modal frequencies. The modal frequencies of the three room modes may be different.

[0051] Figure 3 3 is a block diagram of an audio system 300 according to one or more embodiments. The audio system 300 includes a headset 310 connected to an audio server 320 via a network 330. A user 340 may wear the headset 310 in a room 350.

[0052] The network 330 connects the headset 310 to the audio server 320. The network 330 may include any combination of target area and / or wide area networks using wireless and / or wired communication systems. For example, the network 330 may include the Internet and a mobile phone network. In one embodiment, the network 330 uses standard communication technologies and / or protocols. Therefore, the network 330 may include links using technologies such as Ethernet, 802.11, Worldwide Interoperability for Microwave Access (WiMAX), 2G / 3G / 4G mobile communication protocols, digital subscriber lines (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express Advanced Switching, etc. Similarly, the network protocols used on the network 330 may include Multi-Protocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), etc. The data exchanged through the network 330 may be represented using technologies and / or formats including image data in binary form (e.g., Portable Network Graphics (PNG)), Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc. In addition, all or part of the link can be encrypted using conventional encryption techniques, such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. The network 330 can also connect multiple head-mounted devices located in the same or different rooms to the same audio server 320.

[0053] The head mounted device 310 presents media content to the user. In one embodiment, the head mounted device 310 may be, for example, an NED or an HMD. Typically, the head mounted device 310 may be worn on the user's face such that the media content is presented using one or both lenses of the head mounted device 310. However, the head mounted device 310 may also be used such that the media content is presented to the user in a different manner. Examples of media content presented by the head mounted device 310 include one or more images, video content, audio content, or some combination thereof. The head mounted device 310 includes an audio component, and may also include at least one depth camera assembly (DCA) and / or at least one passive camera assembly (PCA). As described below with reference to Figure 8As described in detail, the DCA generates depth image data describing the 3D geometry of some or all of the target area (e.g., the room 350), and the PCA generates color image data of some or all of the target area. In some embodiments, the DCA and PCA of the head mounted device 310 are part of a simultaneous localization and mapping (SLAM) sensor mounted on the head mounted device 310 for determining visual information of the room 350. Therefore, the depth image data captured by at least one DCA and / or the color image data captured by at least one PCA can be referred to as visual information determined by the SLAM sensor of the head mounted device 310. In addition, the head mounted device 310 can include a position sensor or an inertial measurement unit (IMU) that tracks the position (e.g., position and posture) of the head mounted device 310 within the target area. The head mounted device 310 can also include a global positioning system (GPS) receiver to further track the position of the head mounted device 310 within the target area. The position (including orientation) of the head mounted device 310 within the target area is referred to as the positioning information of the head mounted device 310. The positioning information of the head mounted device can indicate the position of the user 340 of the head mounted device 310.

[0054] The audio component presents audio content to the user 340. The audio content can be presented in such a way that it sounds like it originates from an object (real or object) in a target area, also referred to as spatialized audio content. The target area can be the user's physical environment, such as room 350, or a virtual area. For example, the audio content presented by the audio component may sound like it originates from a virtual speaker in a virtual conference room (being presented to the user 340 through the head-mounted device 310). In some embodiments, the local effects of the room mode associated with the position of the user 340 in the target area are incorporated into the audio content. The local effects of the room mode are represented by the acoustic distortion (of a specific frequency) that occurs at the position of the user 340 in the target area. The acoustic distortion can change as the position of the user in the target area changes. In some embodiments, the target area is room 350. In some other embodiments, the target area is a virtual area. The virtual area can be based on a real room different from room 350. For example, room 350 is an office. The target area is a virtual area based on a conference room. The audio content presented by the audio component can be a speech from a speaker located in a conference room. The position in the conference room corresponds to the position of the user in the target area. The audio content is rendered so that it sounds like it originates from speakers in the conference room and is received at locations within the conference room.

[0055] The audio component uses an acoustic filter to incorporate the local effects of room modes. The audio component requests an acoustic filter by sending a room mode query to the audio server 320. A room mode query is a request for one or more room mode parameters, based on which the audio component can generate an acoustic filter that simulates the acoustic distortion caused by the room mode when applied to the audio content (e.g., amplification as a function of frequency and position). The room mode query may include visual information describing some or all target areas (e.g., room 350 or virtual area), user positioning information, information about the audio content, or some combination thereof. The visual information describes the 3D geometry of some or all target areas, and may also include color image data of some or all target areas. In some embodiments, the visual information of the target area may be captured by the head mounted device 310 (e.g., in an embodiment where the target area is the room 350) and / or by a different device. The user's positioning information indicates the position of the user 340 within the target area, and may include positioning information of the head mounted device 310 or information describing the position of the user 340. The information about the audio content includes, for example, information describing the positioning of a virtual sound source of the audio content. The virtual sound source of the audio content may be a real object and / or a virtual object in the target area. The head mounted device 310 may transmit the room mode query to the audio server 320 via the network 330 .

[0056] In some embodiments, the head mounted device 310 obtains one or more room mode parameters describing an acoustic filter from the audio server 320. Room mode parameters are parameters describing an acoustic filter that, when applied to audio content, simulates the acoustic distortion caused by one or more room modes in a target area. The room mode parameters include a Q factor, a gain, an amplitude, a modal frequency of the room mode, some other characteristic describing the acoustic filter, or some combination thereof. The head mounted device 310 uses the room mode parameters to generate a filter for rendering the audio content. For example, the head mounted device 310 generates an infinite impulse response filter and / or an all-pass filter. The infinite impulse response filter and / or the all-pass filter include a Q value and a gain corresponding to each modal frequency. Figure 4 , Figure 8 and Fig. 9 Additional details regarding the operation and components of the head mount 310 are discussed.

[0057] The audio server 320 determines one or more room mode parameters based on the room mode query received from the head mounted device 310. The audio server 320 determines a model of the target area. In some embodiments, the audio server 320 determines the model based on visual information of the target area. For example, the audio server 320 obtains a 3D virtual representation of at least a portion of the target area based on the visual information. The audio server 320 compares the 3D virtual representation with a set of candidate models and identifies the candidate model that matches the 3D virtual representation as the model. In some embodiments, the candidate model is a model of the room, which includes the shape of the room, one or more dimensions of the room, or material acoustic parameters (e.g., attenuation parameters) of the surface within the room. The candidate model group may include room models with different shapes, different sizes, and different surfaces. The 3D virtual representation of the target area includes a 3D mesh of the target area, which defines the shape and / or size of the target area. The 3D virtual representation may use one or more material acoustic parameters (e.g., attenuation parameters) to describe the acoustic properties of the surface within the target area. The audio server 320 determines that the candidate model matches the 3D virtual representation based on a determination that the difference between the candidate model and the 3D virtual representation is below a threshold. The differences may include differences in shape, size, surface acoustic properties, etc. In some embodiments, the audio server 320 uses a fit metric to determine the difference between the candidate model and the 3D virtual representation. The fit metric may be based on one or more geometric features, such as the squared error of the Hausdorff distance, openness (e.g., indoor vs. outdoor), volume, etc. The threshold may be based on a perceived just noticeable difference (JND) in room mode changes. For example, if a user can detect a 10% change in modal frequency, then geometric deviations that would result in up to a 10% change in modal frequency are tolerable. The threshold may be the geometric deviation that results in a 10% change in modal frequency.

[0058] The audio server 320 uses the model to determine the room mode of the target area. For example, the audio server 320 uses conventional techniques, such as numerical simulation techniques (e.g., finite element method, boundary element method, finite difference time domain method, etc.) to determine the room mode. In some embodiments, the audio server 320 determines the room mode based on the shape, size and / or material acoustic parameters of the model to determine the room mode. The room mode may include one or more of an axial mode, a tangential mode, and an inclined mode. In some embodiments, the audio server 320 determines the room mode based on the user's position. For example, the audio server 320 identifies the target area based on the user's position and retrieves the room mode of the target area based on the identification.

[0059] The audio server 320 determines one or more room mode parameters based on at least one of the room modes and the user's position within the target area. The room mode parameters describe an acoustic filter that, when applied to the audio content, simulates the acoustic distortion that occurs at the user's position within the target area for frequencies associated with the at least one room mode. The audio server 320 transmits the room mode parameters to the head mounted device 310 for rendering the audio content. In some embodiments, the audio server 320 may generate an acoustic filter based on the room mode parameters and transmit the acoustic filter to the head mounted device 310.

[0060] Figure 4 4 is a block diagram of an audio server 400 according to one or more embodiments. An embodiment of the audio server 400 is the audio server 320. The audio server 400 determines one or more room mode parameters of a target area in response to a room mode query from an audio component. The audio server 400 includes a database 410, a mapping module 420, a matching module 430, a room mode module 440, and an acoustic filter module 450. In other embodiments, the audio server 400 may have any combination of the listed modules and any additional modules. One or more processors (not shown) of the audio server 400 may run some or all of the modules within the audio server 400.

[0061] Database 410 stores data for audio server 400. The stored data may include virtual models, candidate models, room modes, room mode parameters, acoustic filters, audio data, visual information (depth information, color information, etc.), room mode queries, other information that may be used by audio server 400, or some combination thereof.

[0062] The virtual model describes one or more areas and the acoustic properties of these areas (e.g., room modes). Each positioning in the virtual model is associated with the acoustic properties of the corresponding area (e.g., room modes). The areas whose acoustic properties are described in the virtual model include virtual areas, physical areas, or some combination thereof. In contrast to virtual areas, physical areas are real areas (e.g., actual physical rooms). Examples of physical areas include conference rooms, bathrooms, corridors, offices, bedrooms, dining rooms, outdoor spaces (e.g., courtyards, gardens, parks, etc.), living rooms, auditoriums, some other real areas, or some combination thereof. Virtual area descriptions may be completely fictional and / or based on spaces of real physical areas (e.g., rendering a physical room as a virtual area). For example, a virtual area may be a fictional dungeon, a rendering of a virtual conference room, etc. Note that virtual areas may be based on real places. For example, a virtual conference room may be based on a real conference center. A specific positioning in the virtual model may correspond to the current physical positioning of the head mounted device 310 within the room 350. Based on the positioning within the virtual model obtained from the mapping module 420, the acoustic properties of the room 350 may be retrieved from the virtual model.

[0063] A room mode query is a request for room mode parameters describing an acoustic filter that is used to combine the effect of a room mode of a target area for a user position within the target area. The room mode query includes target area information, user information, audio content information, some other information that the audio server 320 can use to determine the acoustic filter, or some combination thereof. The target area information is information describing the target area (e.g., its geometry, objects, materials, colors, etc. therein). It may include depth image data of the target area, color image data of the target area, or some combination thereof. User information is information describing the user. It may include information describing the user's position within the target area, information about the physical area where the user is physically located, or some combination thereof. Audio content information is information describing the audio content. It may include localization information of virtual sound sources of the audio content, localization information of physical sound sources of the audio content, or some combination thereof.

[0064] The candidate models may be models of rooms having different shapes and / or sizes. The audio server 400 uses the candidate models to determine a model of the target area.

[0065] The mapping module 420 maps the information in the room mode query to a location within the virtual model. The mapping module 420 determines a location within the virtual model that corresponds to a target area. In some embodiments, the mapping module 420 searches the virtual model to identify a mapping between (i) information about the target area and / or information about the user's location and (ii) a corresponding configuration of the area within the virtual model. The area within the virtual model may describe a physical area and / or a virtual area. In one embodiment, the mapping is performed by matching the geometry of the visual information of the target area with the geometry associated with the location within the virtual model. In another embodiment, the mapping is performed by matching the information of the user's location with the location within the virtual model. For example, in an embodiment where the target area is a virtual area, the mapping module 420 identifies a location associated with a virtual area in the virtual model based on information indicating the user's location. The match indicates that the location within the virtual model is a representation of the target area.

[0066] If a match is found, the mapping module 420 retrieves the room mode associated with the positioning within the virtual model and sends the room mode to the acoustic filter module 450 for determining room mode parameters. In some embodiments, the virtual model does not include a room mode associated with the positioning of the matching target area within the virtual model, but instead includes a candidate model associated with the positioning. The mapping module 420 can retrieve the candidate model and send it to the room mode module 440 to determine the room mode of the target area. In some embodiments, the virtual model does not include a room mode or candidate model associated with the positioning of the matching target area within the virtual model. The mapping module 420 can retrieve a 3D representation of the positioning and send it to the matching module 430 to determine the model of the target area.

[0067] If no match is found, this indicates that the virtual model does not yet describe the configuration of the target area. In this case, the mapping module 420 can develop a 3D virtual representation of the target area based on the visual information in the room mode query, and update the virtual model with the 3D virtual representation. The 3D virtual representation of the target area may include a 3D grid of the target area. The 3D grid includes points and / or lines representing the boundaries of the target area. The 3D virtual representation may also include a virtual representation of surfaces within the target area, such as walls, ceilings, floors, furniture surfaces, electrical appliance surfaces, surfaces of other types of objects, and the like. In some embodiments, the virtual model uses one or more material acoustic parameters (e.g., attenuation parameters) to describe the acoustic properties of surfaces within the virtual area. In some embodiments, the mapping module 420 can develop a new model including a 3D virtual representation, and use one or more material acoustic parameters to describe the acoustic properties of surfaces within the virtual area. The new model can be stored in the database 410.

[0068] The mapping module 420 may also notify at least one of the matching module 430 and the room mode module 440 that no match is found, so that the matching module 430 may determine a model of the target area and the room mode module 440 may determine the room mode of the target area by using the model.

[0069] In some embodiments, mapping module 420 may also determine a location within the virtual model that corresponds to a local area (eg, room 350) where the user is physically located.

[0070] The target area may be different from the local area. For example, the local area is the office where the user sits, but the target area is a virtual area (e.g., a virtual conference room).

[0071] If a match is found, mapping module 420 retrieves the room mode associated with the location corresponding to the target area within the virtual model and sends the room mode to acoustic filter module 450 for determining room mode parameters. If a match is not found, mapping module 420 can develop a 3D virtual representation of the target area based on the visual information in the room mode query and update the virtual model with the 3D virtual representation of the target area. Mapping module 420 can also notify at least one of matching module 430 and room mode module 440 that a match was not found, so that matching module 430 can determine a model of the target area, and room mode module 440 can determine the room mode of the target area by using the model.

[0072] The matching module 430 determines the model of the target area based on the 3D virtual representation of the target area. Taking the target area as an example, in some embodiments, the matching module 430 selects a model from multiple candidate models. The candidate model can be a model of a room, including information about the shape, size or surface in the room. The candidate model group can include models of rooms with different shapes (e.g., square, circle, triangle, etc.), different sizes (such as shoe boxes, large conference rooms, etc.) and different surfaces. The matching module 430 compares the 3D virtual representation of the target area with each candidate model and determines whether the candidate model matches the 3D virtual representation. The matching module 430 determines that the candidate model matches the 3D virtual representation based on the determination that the difference between the candidate model and the 3D virtual representation is lower than a threshold. The difference may include differences in shape, size, surface acoustic properties, etc. In some embodiments, the matching module 430 can determine that the 3D virtual representation matches multiple candidate models. The matching module 430 selects the candidate model with the best match, that is, the candidate model with the smallest difference from the 3D virtual representation.

[0073] In some embodiments, the matching module 430 compares the shape of the candidate model to the shape of the 3D mesh contained in the 3D virtual representation. For example, the matching module 430 traces rays from the center of the 3D mesh target area in multiple directions and determines the points where the rays intersect the 3D mesh calculations. The matching module 430 identifies the candidate model that matches these points. The matching module 430 can shrink or expand the candidate model to exclude any size differences between the candidate model and the target area from the comparison.

[0074] The room mode module 440 uses the model of the target area to determine the room mode of the target area. The room mode may include at least one of three room modes: an axial mode, a tangential mode, and an oblique mode. In some embodiments, for each type of room mode, the room mode module 440 determines a first-order mode, and may also determine higher-order modes. The room mode module 440 determines the room mode based on the shape and / or size of the model. For example, in an embodiment where the model has a rectangular uniform shape, the room mode module 440 determines the axial, tangential, and oblique modes of the model. In some embodiments, the room mode module 440 uses the size of the model to calculate room modes that fall within a range from a lower frequency (e.g., 63 Hz) in the audible or reproducible frequency range to the Schroeder frequency of the target area. The Schroeder frequency of the target area may be a frequency at which the room mode overlaps too densely in frequency to be distinguished individually. The room mode module 440 may determine the Schroeder frequency based on the volume of the target area and the reverberation time (e.g., RT60) of the target area. The room mode module 440 may determine the room modes using, for example, numerical simulation techniques (eg, finite element method, boundary element method, finite difference time domain method, etc.).

[0075] In some embodiments, the room mode module 440 uses material acoustic parameters (e.g., attenuation parameters) of surfaces within the 3D virtual representation of the target area to determine the room mode. For example, the room mode module 440 uses color image data of the target area to determine the material composition of the surfaces. The room mode module 440 determines the attenuation parameters for each surface based on the material composition of the surface and updates the model with the material composition and the attenuation parameters.

[0076] In one embodiment, the room mode module 440 uses machine learning techniques to determine the material composition of the surface. The initialization module can input the image data of the target area (or a portion of the image data related to the surface) and / or the audio data into the machine learning model, and the machine learning model outputs the material composition of each surface. The machine learning model can be trained using different machine learning techniques, such as linear support vector machines (linear SVM), boosting for other algorithms (e.g., AdaBoost), neural networks, logistic regression, naive Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, or boosted stumps. As part of the training of the machine learning model, a training set is formed. The training set includes a set of image data and / or audio data of surfaces and the material composition of the surfaces in the set.

[0077] For each room mode or combination of multiple room modes, the room mode module 440 determines amplification as a function of frequency and location. Amplification includes an increase or decrease in signal strength caused by the corresponding room mode.

[0078] The acoustic filter module 450 determines one or more room mode parameters for the target area based on at least one of the room modes and the position of the user within the target area. In some embodiments, the acoustic filter module 450 determines the room mode parameters based on amplification as a function of frequency and position (e.g., the position of the user) within the target area. The room mode parameters describe the acoustic distortion caused by at least one room mode at the user's position. In some embodiments, the acoustic filter module 450 also uses the position of the sound source of the audio content to determine the acoustic distortion.

[0079] In some embodiments, the audio content is rendered by one or more speakers outside the head-mounted device. The acoustic filter module 450 determines one or more room mode parameters of the local area of ​​the user. In some embodiments, the target area is different from the local area. For example, the local area of ​​the user is an office where the user sits, and the target area is a virtual conference room including a virtual sound source (e.g., a speaker). The room mode parameters of the local area describe an acoustic filter of the local area, which can be used to render audio content from speakers outside the head-mounted device (e.g., on a console or coupled to a console). The acoustic filter of the local area mitigates the room mode of the local area at the user's position in the local area. In some embodiments, the acoustic filter module 450 determines the room mode parameters of the local area based on one or more room modes of the local area determined by the room mode module 440. The room mode of the local area can be determined based on a model of the local area determined by the mapping module 420 or the matching module 430.

[0080] Figure 5 is a flow chart illustrating a process 500 for determining room mode parameters describing an acoustic filter in accordance with one or more embodiments. Figure 5 The process 500 may be performed by components of the apparatus, such as Figure 4 The audio server 400 of FIG. 400. In other embodiments, other entities (e.g., parts of the headset and / or console) may perform some or all of the steps of the process. Likewise, embodiments may include different and / or additional steps, or perform the steps in a different order.

[0081] The audio server 400 determines 510 a model of the target area based in part on the 3D virtual representation of the target area. The target area may be a local area or a virtual area. The virtual area may be based on a real room. In some embodiments, the audio server 510 determines the model by retrieving the model from a database based on the position of the user in the target area. For example, the database stores virtual models describing one or more areas and includes models of these areas. Each area corresponds to a location in the virtual model. These areas include virtual areas, physical areas, or some combination thereof. The audio server 400 may, for example, identify a location associated with the target area in the virtual model based on the position of the user in the target area. The audio server 400 retrieves a model associated with the identified location. In some other embodiments, the audio server 400 receives depth information describing at least a portion of the target area, for example from a head-mounted device. In some embodiments, the audio server 400 generates at least a portion of the 3D virtual representation using the depth information. The audio server 400 compares the 3D virtual representation with a plurality of candidate models. The audio server 400 identifies a candidate model of the matching three-dimensional virtual representation among the plurality of candidate models as a model of the target area. In some embodiments, the audio server 400 determines that the candidate model matches the three-dimensional virtual representation based on a determination that the difference between the shape of the candidate model and the 3D virtual representation is below a threshold. The audio server 400 can shrink or expand the candidate model during the comparison to eliminate any difference in size between the candidate model and the 3D virtual representation. In some embodiments, the audio server 400 determines an attenuation parameter for each surface in the 3D virtual representation and updates the model with the attenuation parameter.

[0082] The audio server 400 uses the model to determine 520 the room mode of the target area. In some embodiments, the audio server 320 determines the room mode based on the shape of the model. The room mode can be calculated using conventional techniques. The audio server 400 can also use the attenuation parameters of the surface in the model size and / or 3D virtual representation to determine the room mode. The room mode may include an axial mode, a tangential mode, or an inclined mode. In some embodiments, the room mode falls within the range from the lower frequencies of the audible frequency range (e.g., 63 Hz) to the Schroeder frequency of the target area. The room mode describes the sound amplification of a specific frequency as a function of the position within the target area. The audio server 400 can determine the amplification corresponding to a combination of multiple room modes.

[0083] The audio server 400 determines 530 one or more room mode parameters (e.g., Q factor, etc.) based on at least one room mode and the user's position within the target area. The room mode is represented by the amplification of the signal strength as a function of frequency and position. In some embodiments, the audio server 400 combines the amplification associated with more than one room mode to more fully describe the amplification as a function of frequency and position. The audio server 400 determines the amplification as a function of the frequency at the user's position. Based on the amplification at the user's position and the function of the frequency, the audio server 400 determines the room mode parameters. The room mode parameters describe an acoustic filter that, when applied to the audio content, simulates the acoustic distortion at the user's position at a frequency associated with at least one room mode. In some embodiments, at least one room mode is a first-order axial mode. In some embodiments, the audio server 320 determines one or more room mode parameters based on the amplification corresponding to at least one room mode at the user's position within the target area. The head mounted device may present the audio content to the user using the acoustic filter.

[0084] Figure 6 6 is a block diagram of an audio component 600 according to one or more embodiments. Some or all of the audio component 600 may be part of a head mounted device (e.g., head mounted device 310). The audio component 600 includes a speaker component 610, a microphone component 620, and an audio controller 630. In one embodiment, the audio component 600 also includes an input interface (not shown). Figure 6 ), for example, to control the operation of different components of the audio component 600. In other embodiments, the audio component 600 can have any combination of the listed components and any additional components. In some embodiments, one or more functions of the audio server 400 can be performed by the audio component 600.

[0085] The speaker assembly 610 generates sound for the user's ears, for example, based on audio instructions from the audio controller 630. In some embodiments, the speaker assembly 610 is implemented as a pair of air conduction transducers (e.g., one for each ear) that generate sound by, for example, generating air-propagated sound pressure waves in the user's ears according to audio instructions from the audio controller 630. Each air conduction transducer of the speaker assembly 610 may include one or more transducers to cover different parts of the frequency range. For example, a piezoelectric transducer may be used to cover a first part of the frequency range, while a dynamic coil transducer may be used to cover a second part of the frequency range. In some other embodiments, each transducer of the speaker assembly 610 is implemented as a bone conduction transducer that generates sound by vibrating a corresponding bone in the user's head. Each transducer implemented as a bone conduction transducer may be placed behind an auricle coupled to a portion of the user's bone to vibrate a portion of the user's bone that generates tissue-propagated sound pressure waves that propagate toward the user's cochlea, thereby bypassing the eardrum. In some other embodiments, each transducer of the speaker assembly 610 is implemented as a cartilage conduction transducer that produces sound by vibrating one or more portions of ear cartilage around the outer ear (e.g., the auricle, the tragus, some other portion of the ear cartilage, or some combination thereof). The cartilage conduction transducer produces air-borne sound pressure waves by vibrating one or more portions of the ear cartilage.

[0086] The microphone assembly 620 detects sounds from the target area. The microphone assembly 620 may include multiple microphones. The multiple microphones may include, for example, at least one microphone configured to measure sounds at the entrance of the ear canal of each ear, one or more microphones positioned to capture sounds from the target area, one or more microphones positioned to capture sounds from the user (e.g., user voice), or some combination thereof.

[0087] The audio controller 630 generates a room mode query to request room mode parameters. The audio controller 630 may generate the room mode query based at least in part on the visual information of the target area and the positioning information of the user. The audio controller 630 may obtain the visual information of the target area, for example, from one or more cameras of the head mounted device 310. The visual information describes the 3D geometry of the target area. The visual information may include depth image data, color image data, or a combination thereof. The depth image data may include geometric information about the shape of the target area defined by the surface of the target area (e.g., the surface of the wall, floor, and ceiling of the target area). The color image data may include information about the acoustic material associated with the surface of the target area. The audio controller 630 may obtain the positioning information of the user from the head mounted device 310. In one embodiment, the positioning information of the user includes the positioning information of the head mounted device. In another embodiment, the local information of the user specifies the position of the user in a real room or a virtual room.

[0088] The audio controller 630 generates an acoustic filter based on the room mode parameters received from the audio server 400, and provides an audio instruction to the speaker assembly 610 to present the audio content using the acoustic filter. For example, the audio controller 630 generates a bell-shaped parameter infinite impulse response filter based on the room mode parameters. The bell-shaped parameter infinite impulse response filter includes a Q value and a gain corresponding to each modal frequency. In some embodiments, the audio controller 630 applies these filters to render the audio signal, for example, by increasing the amplitude of the audio signal at the modal frequency. In some embodiments, the audio controller 630 places these filters in the feedback loop of an artificial reverberator (e.g., Schroeder, FDN, or nested all-pass reverberator), or modifies the reverberation time at the modal frequency. The audio controller 630 applies the acoustic filter to the audio content so that the acoustic distortion (e.g., amplification as a function of frequency and position) caused by the room mode associated with the user's target area can be part of the audio content presented.

[0089] As another example, the audio controller 630 generates an all-pass filter based on the room mode parameters. The Q value of the all-pass filter is centered on the modal frequency. The audio controller 630 uses the all-pass filter to delay the audio signal at the modal frequency and produce a ringing feeling at the modal frequency. In some embodiments, the audio controller 630 uses a bell-shaped parameter infinite impulse response filter and an all-pass filter to render the audio signal. In some embodiments, the audio controller 630 dynamically updates the filter based on changes in the user's position.

[0090] Figure 7 is a flow chart illustrating a process 700 of rendering audio content using an acoustic filter in accordance with one or more embodiments. Figure 7The process 700 may be performed by components of the apparatus, such as Figure 6 In other embodiments, other entities (e.g., Fig. 9 Components of the head mounted device 900 and / or Figure 8 The components shown in the figure) can perform some or all steps of the process. Likewise, embodiments can include different and / or additional steps, or perform the steps in a different order.

[0091] The audio component 600 generates 710 an acoustic filter based on one or more room mode parameters. When applied to the content, the acoustic filter simulates acoustic distortion at a user position within a target area and at frequencies associated with at least one room mode of the target area. When sound is emitted in the target area, the acoustic distortion is represented by amplification at the user position within the target area. The target area can be a local area of ​​the user or a virtual area. In some embodiments, the acoustic filter includes an infinite impulse response filter with a Q value and a gain at a modal frequency of the room mode and / or an all-pass filter with a Q value centered at the modal frequency.

[0092] In some embodiments, the audio component 600 receives one or more room mode parameters from an audio server (e.g., the audio server 400). The audio component sends a room mode query to the audio server, and the audio server determines one or more room mode parameters based on the information in the room mode query. In some other embodiments, the audio component 600 determines one or more room mode parameters based on at least one room mode of the target area. The at least one room mode of the target area can be determined by the audio server and sent to the audio component 600.

[0093] The audio component 600 presents 720 audio content to the user by using an acoustic filter. For example, the audio component 600 applies an acoustic filter to the audio content so that acoustic distortion (e.g., an increase or decrease in signal strength) caused by room modes associated with a target area for the user can be part of the audio content presented. The audio content sounds like it originates from objects in the target area and is received at the user's location within the target area, even though the user may not be physically in the target area. For example, a user is sitting in an office and audio content (e.g., music) can be presented to sound like it originates from speakers in a virtual conference room and is received at the user's location in the virtual conference room.

[0094] System environment

[0095] Figure 88 is a block diagram of a system environment 800 including a head mounted device 810 and an audio server 400 according to one or more embodiments. The system 800 may operate in an artificial reality environment (e.g., a virtual reality environment, an augmented reality environment, a mixed reality environment, or some combination thereof). Figure 8 The illustrated system 800 includes a headset 810, an audio server 400, and an input / output (I / O) interface 840 coupled to a console 860. The headset 810, the audio server 400, and the console 860 communicate via a network 880. Figure 8 An example system 800 is shown that includes one head mounted device 810 and one I / O interface 850, but in other embodiments, any number of these components may be included in the system 800. For example, there may be multiple head mounted devices 810, each head mounted device 810 having an associated I / O interface 850, and each head mounted device 810 and I / O interface 850 communicating with a console 860. In alternative configurations, different and / or additional components may be included in the system 800. Furthermore, in some embodiments, in combination with Figure 8 The functionality described by one or more of the components shown may be combined with Figure 8 The described methods are distributed among the components in different ways. For example, some or all of the functions of the console 860 can be provided by the head mounted device 810.

[0096] The head mounted device 810 includes a display component 815, an optical block 820, one or more position sensors 835, a DCA 830, an inertial measurement unit (IMU) 825, a PCA 840, and an audio component 600. Some embodiments of the head mounted device 810 have a configuration that is combined with Figure 8 The components described are different components. In addition, by combining Figure 8 The functionality provided by the various components described may be distributed differently among the components of the head mount 810 in other embodiments, or may be captured in a separate component remote from the head mount 810. Figure 3 The head mounted device 310 or Fig. 9 The head mounted device 900 is shown in FIG.

[0097] The display component 815 may include an electronic display that displays a 2D or 3D image to the user based on data received from the console 860. The image may include an image of a local area of ​​the user, an image of a virtual object combined with light from the local area, an image of a virtual area, or some combination thereof. The virtual area may be mapped to a real room away from the user. In various embodiments, the display component 815 includes a single electronic display or multiple electronic displays (e.g., a display for each eye of the user). Examples of electronic displays include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active matrix organic light emitting diode display (AMOLED), a waveguide display, some other display, or some combination thereof.

[0098] The optical block 820 amplifies the image light received from the electronic display, corrects the optical errors associated with the image light, and presents the corrected image light to the user of the head mounted device 810. In various embodiments, the optical block 820 includes one or more optical elements. Example optical elements included in the optical block 820 include: an aperture, a Fresnel lens, a convex lens, a concave lens, a filter, a reflective surface, or any other suitable optical element that affects the image light. In addition, the optical block 820 may include a combination of different optical elements. In some embodiments, one or more optical elements in the optical block 820 may have one or more coatings, such as a partially reflective coating or an anti-reflective coating.

[0099] The magnification and focusing of the image light by the optical block 820 allows the electronic display to be physically smaller, lighter, and consume less power than a larger display. In addition, the magnification can increase the field of view of the content presented by the electronic display. For example, the field of view of the displayed content is such that the displayed content uses almost all (e.g., about 110 degrees diagonally), and in some cases, all of the user's field of view to be presented. In addition, in some embodiments, the amount of magnification can be adjusted by adding or removing optical elements.

[0100] In some embodiments, the optical block 820 can be designed to correct one or more types of optical errors. Examples of optical errors include barrel or pincushion distortion, longitudinal chromatic aberration, or lateral chromatic aberration. Other types of optical errors may also include spherical aberration, chromatic aberrations, or errors due to lens field curvature, astigmatism, or any other type of optical error. In some embodiments, the content provided to the electronic display for display is pre-distorted, and after the optical block 820 receives image light generated based on the content from the electronic display, the optical block corrects the distortion.

[0101] The IMU 825 is an electronic device that generates data indicating the position of the head mounted device 810 based on measurement signals received from one or more position sensors 835. The position sensor 835 generates one or more measurement signals in response to the movement of the head mounted device 810. Examples of the position sensor 835 include one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects movement, a type of sensor used for error correction of the IMU 825, or some combination thereof. The position sensor 835 can be located outside the IMU 825, inside the IMU 825, or some combination of these locations.

[0102] The DCA 830 generates depth image data of a target area, such as a room. The depth image data includes pixel values ​​that define the distance from the imaging device and thus provides a (eg, 3D) map of the location captured in the depth image data. Figure 8 The DCA 830 in FIG. 8 includes a light projector 833, one or more imaging devices 836, and a controller 830. In some other embodiments, the DCA 830 includes a set of cameras for stereoscopic imaging.

[0103] The light projector 833 can project a structured light pattern or other light (e.g., a time-of-flight infrared flash) that is reflected by objects in the target area and captured by the imaging device 836 to generate depth image data. For example, the light projector 833 can project multiple structured light (SL) elements (e.g., lines, grids, or dots) of different types onto a portion of the target area around the head-mounted device 810. In various embodiments, the light projector 833 includes an emitter and a diffractive optical element. The emitter is configured to illuminate the diffractive optical element with light (e.g., infrared light). The illuminated diffractive optical element projects an SL pattern including multiple SL elements into the target area. For example, each SL element projected by the illuminated diffractive optical element is a point associated with a specific location on the diffractive optical element.

[0104] The SL pattern projected into the target area by the DCA 830 deforms when encountering various surfaces and objects in the target area. One or more imaging devices 836 are each configured to capture one or more images of the target area. Each of the captured one or more images may include multiple SL elements (e.g., points) projected by the light projector 833 and reflected by objects in the target area. Each of the one or more imaging devices 836 may be a detector array, a camera, or a video camera.

[0105] In some embodiments, the light projector 833 projects light pulses that are reflected by objects in the local area and captured by the imaging device 836 to generate depth image data using time-of-flight technology. For example, the light projector 833 projects infrared flashes for time-of-flight. The imaging device 836 captures infrared flashes reflected by the object. The controller 837 can use the image data from the imaging device 836 to determine the distance to the object. The controller 837 can provide instructions to the imaging device 836 so that the imaging device 836 captures the reflected light pulses synchronously with the light projector 833 projecting the light pulses.

[0106] The controller 837 generates depth image data based on the light captured by the imaging device 836. The controller 837 may further provide the depth image data to the console 860, an audio controller, or some other component.

[0107] The PCA 840 includes one or more passive cameras that generate color (e.g., RGB) image data. Unlike the DCA 830 that uses active light emission and reflection, the PCA 840 captures light from the environment of the target area to generate image data. The pixel values ​​of the image data can define the visible color of the object captured in the imaging data, rather than the pixel values ​​that define the depth or distance from the imaging device. In some embodiments, the PCA 840 includes a controller that generates color image data based on the light captured by the passive imaging device. In some embodiments, the DCA 830 and the PCA 840 share a common controller. For example, the common controller can map each of the one or more images captured in the visible spectrum (e.g., image data) and the infrared spectrum (e.g., depth image data) to each other. In one or more embodiments, the common controller is configured to provide one or more images of the target area to the audio controller or console 860 in addition or alternatively.

[0108] The audio component 600 uses an acoustic filter to present audio content to a user of the head mounted device 810 to incorporate the local effects of the room mode into the audio content. In some embodiments, the audio component 600 sends a room mode query to the audio server 400 to request room mode parameters describing the acoustic filter. The room mode query includes virtual information of the target area, positioning information of the user, information of the audio content, or some combination thereof. The audio component 600 receives the room mode parameters from the audio server 400 via the network 880. The audio component 600 uses the room mode parameters to generate a series of filters (e.g., infinite impulse response filters, all-pass filters, etc.) to render the audio content. The filter has a Q value and gain at the modal frequency and simulates the acoustic distortion of the user's position in the target area. The audio content is spatialized, and when presented, it sounds like it originates from an object (e.g., a virtual object or a real object) in the target area and is received at the user's position in the target area.

[0109] In one embodiment, the target area is at least a portion of the user's local area, and the spatialized audio content can sound as if it originates from a virtual object in the local area. In another embodiment, the target area is a virtual area. For example, the user is in a small office, but the target area is a large virtual conference room where a virtual speaker is giving a speech. The virtual conference room has different acoustic properties, such as room modes, than the small office. The audio component 600 presents speech to the user as if it originates from the virtual speaker in the virtual conference room (i.e., using the room modes of the conference room as if it were real positioning, rather than using the room modes of the small office).

[0110] The audio server 400 determines one or more room mode parameters of the target area based on the information in the room mode query from the audio component 600. In some embodiments, the audio server 400 determines the model of the target area based on the 3D representation of the target area. The 3D representation of the target area can be determined based on the information in the room mode query, such as visual information of the target area and / or positioning information of the user indicating the position of the user in the target area. The audio server 400 compares the 3D representation with the candidate model and selects the candidate model that matches the 3D representation as the model of the target area. The audio server 400 uses the model to determine the room mode of the target area, for example, based on the shape and / or size of the model. The room mode can be represented by amplification as a function of frequency and position. Based on at least one of the room modes and the position of the user in the target area, the audio server 400 determines one or more room mode parameters.

[0111] In some embodiments, the audio component 600 has some or all of the functionality of the audio server 400. The audio component 600 of the headset 810 and the audio server 400 may communicate via a wired or wireless communication link (eg, network 880).

[0112] The I / O interface 850 is a device that allows a user to send an action request and receive a response from the console 860. An action request is a request to perform a specific action. For example, an action request may be an instruction to start or end capturing image or video data, or an instruction to perform a specific action within an application. The I / O interface 850 may include one or more input devices. Example input devices include a keyboard, a mouse, a game controller, or any other suitable device for receiving an action request and transmitting the action request to the console 860. The action request received by the I / O interface 850 is transmitted to the console 860, and the console 860 performs the action corresponding to the action request. In some embodiments, as further described above, the I / O interface 850 includes an IMU825, which captures calibration data indicating the estimated position of the I / O interface 850 relative to the initial position of the I / O interface 850. In some embodiments, the I / O interface 850 can provide tactile feedback to the user according to the instruction received from the console 860. For example, tactile feedback is provided after an action request is received, or after console 860 transmits an instruction to I / O interface 850 that causes I / O interface 850 to generate tactile feedback after console 860 performs an action.

[0113] The console 860 provides content to the head mounted device 810 for processing based on information received from one or more of the DCA 830, the PCA 840, the head mounted device 810, and the I / O interface 850. Figure 8 In the example shown, the console 860 includes an application store 863, a tracking module 865, and an engine 867. Some embodiments of the console 860 have a Figure 8 Similarly, the functions described further below may be implemented in different ways in combination with the modules or components described above. Figure 8 The described approach is distributed among the components of the console 860. In some embodiments, the functions discussed herein with reference to the console 860 can be implemented in the head mounted device 810 or in a remote system.

[0114] The application storage 863 stores one or more applications for execution by the console 860. An application is a set of instructions that, when executed by a processor, generates content for presentation to a user. The content generated by the application may be in response to input received from the user via the movement of the head mounted device 810 or the I / O interface 850. Examples of applications include: gaming applications, conferencing applications, video playback applications, or other suitable applications.

[0115] The tracking module 865 uses one or more calibration parameters to calibrate a local area of ​​the system 800, and can adjust one or more calibration parameters to reduce errors in the determination of the position of the head mounted device 810 or the I / O interface 850. For example, the tracking module 865 transmits the calibration parameters to the DCA 830 to adjust the focus of the DCA 830 to more accurately determine the position of the SL elements captured by the DCA 830. The calibration performed by the tracking module 865 can also take into account information received from the IMU 825 in the head mounted device 810 and / or the IMU 825 included in the I / O interface 850. In addition, if tracking of the head mounted device 810 is lost (e.g., the DCA 830 loses line of sight to at least a threshold number of projected SL elements), the tracking module 865 can recalibrate part or all of the system 800.

[0116] The tracking module 865 tracks the movement of the head mounted device 810 or the I / O interface 850 using information from the DCA 830, the PCA 840, the one or more position sensors 835, the IMU 825, or some combination thereof. For example, the tracking module 865 determines the location of a reference point of the head mounted device 810 in a map of the local area based on the information from the head mounted device 810. The tracking module 865 can also determine the location of an object (real object or virtual object) in the local area or virtual area. In addition, in some embodiments, the tracking module 865 can use the portion of the data from the IMU 825 indicating the location of the head mounted device 810 and the representation of the local area from the DCA 830 to predict the future positioning of the head mounted device 810. The tracking module 865 provides the estimated or predicted future location of the head mounted device 810 or the I / O interface 850 to the engine 867.

[0117] The engine 867 executes the application and receives the position information, acceleration information, velocity information, predicted future position, or some combination thereof of the head mounted device 810 from the tracking module 865. Based on the received information, the engine 867 determines the content to be provided to the head mounted device 810 for presentation to the user. For example, if the received information indicates that the user is at the location of the target area, the engine 867 generates virtual content (e.g., images and audio) associated with the target area. The target area may be a virtual area, such as a virtual conference room. The engine 867 may generate an image of the virtual conference room and a speech given in the virtual conference room for the head mounted device 801 to display to the user. The target area may be a local area of ​​the user. The engine 867 may generate an image of a virtual object combined with a real object from a local area and audio content associated with the virtual object or the real object. As another example, if the received information indicates that the user has looked to the left, the engine 867 generates content for the head mounted device 810 that reflects (mirror) the user's movement in the virtual target area or in the target area that enhances the target area with additional content. In addition, the engine 867 executes an action within an application executed on the console 860 in response to an action request received from the I / O interface 850, and provides feedback to the user that the action is executed. The feedback provided may be visual or auditory feedback via the head mounted device 810, or tactile feedback via the I / O interface 850.

[0118] Fig. 9 is a perspective view of a head mounted device 900 including an audio component according to one or more embodiments. The head mounted device 900 may be Figure 3 The head mounted device 310 or Figure 8 In some embodiments (such as Fig. 9 In an alternative embodiment (not shown), the head mounted device 900 is implemented as a NED. Fig. 9 ), the head mounted device 900 is implemented as an HMD. Typically, the head mounted device 900 can be worn on the face of a user so that content (e.g., media content) is presented using one or two lenses 910 of the head mounted device 900. However, the head mounted device 900 can also be used so that media content is presented to the user in a different manner. Examples of media content presented by the head mounted device 900 include one or more images, videos, audio, or some combination thereof. The head mounted device 900 may include components such as a frame 905, lenses 910, a DCA 925, a PCA 930, a position sensor 940, and an audio component. The DCA 925 and the PCA 930 may be part of a SLAM sensor mounted on the head mounted device 900 to capture visual information of a target area around some or all of the head mounted device 900. Although Fig. 9The components of the head mount 900 are shown in example locations on the head mount 900, but the components may be located elsewhere on the head mount 900, on a peripheral device that is paired with the head mount 900, or some combination thereof.

[0119] The head mounted device 900 may correct or enhance the user's vision, protect the user's eyes, or provide images to the user. The head mounted device 900 may be glasses that correct the user's vision defects. The head mounted device 900 may be sunglasses that protect the user's eyes from the sun. The head mounted device 900 may be safety glasses that protect the user's eyes from collisions. The head mounted device 900 may be a night vision device or infrared goggles to enhance the user's night vision. The head mounted device 900 may be a near-eye display that generates artificial reality content for the user. Alternatively, the head mounted device 900 may not include the lens 910 and may be a frame 905 with an audio component that provides audio content (e.g., music, radio, podcast) to the user.

[0120] The frame 905 holds the other components of the head mounted device 900. The frame 905 includes a front portion that holds the lens 910 and an end piece that attaches to the user's head. The front portion of the frame 905 bridges the top of the user's nose. The end piece (e.g., temples) is the portion of the frame 905 to which the user's temples are attached. The length of the end piece can be adjustable (e.g., adjustable temple length) to fit different users. The end piece can also include a portion that curls behind the user's ear (e.g., temple tip, ear piece).

[0121] The lens 910 provides or transmits light to the user wearing the head mounted device 900. The lens 910 may include a prescription lens (e.g., a single vision lens, a bifocal and trifocal or a progressive lens) to help correct the user's vision defects. The prescription lens transmits ambient light to the user wearing the head mounted device 900. The transmitted ambient light can be changed by the prescription lens to correct the user's vision defects. The lens 910 may include a polarized lens or a tinted lens to protect the user's eyes from sunlight. The lens 910 may include one or more waveguides as part of a waveguide display, wherein the image light is coupled to the user's eyes through the end or edge of the waveguide. The lens 910 may include an electronic display for providing image light, and may also include an optical block for amplifying the image light from the electronic display. The lens 910 may be an embodiment of a combination of a display assembly 815 and an optical block 820.

[0122] DCA 925 captures depth image data describing depth information of a local area (e.g., a room) surrounding head mounted device 310. DCA 925 may be an embodiment of DCA 830. In some embodiments, DCA 925 may include a light projector (e.g., structured light and / or flash lighting for time-of-flight), an imaging device, and a controller (not shown). Fig. 9 ). The captured data may be an image of light projected onto the local area by the light projector captured by the imaging device. In one embodiment, the DCA 925 may include a controller and two or more cameras oriented to capture portions of the local area in a stereoscopic manner. The captured data may be an image captured in a stereoscopic manner by two or more cameras of the local area. The controller of the DCA 925 uses the captured data and depth determination technology (e.g., structured light, time of flight, stereo imaging, etc.) to calculate depth information of the local area. Based on the depth information, the controller of the DCA 925 determines the absolute position information of the head mounted device 310 within the local area. The DCA 925 may be integrated with the head mounted device 310, or may be located within the local area outside the head mounted device 310. In some embodiments, the controller of the DCA 925 may transmit the depth image data to the audio controller 920 of the head mounted device 310, for example for further processing and transmission to the audio server 400.

[0123] PCA 930 includes one or more passive cameras that generate color (e.g., RGB) image data. PCA 930 can be an embodiment of PCA 840. Unlike DCA 925, which uses active light emission and reflection, PCA 930 captures light from the environment of a local area to generate color image data. The pixel values ​​of the color image data can define the visible color of the object captured in the image data, rather than the pixel values ​​defining the depth or distance from the imaging device. In some embodiments, PCA 930 includes a controller that generates color image data based on the light captured by the passive imaging device. PCA 930 can provide the color image data to the audio controller 920, for example, for further processing and transmission to the audio server 400.

[0124] In some embodiments, DCA 925 and PCA 930 are the same camera assembly, such as a color camera system that uses stereo imaging to generate depth information.

[0125] The position sensor 940 generates positioning information of the head mounted device 900 based on one or more measurement signals in response to the movement of the head mounted device 9010. The position sensor 940 may be an embodiment of one of the position sensors 835. The position sensor 940 may be located on a portion of the frame 905 of the head mounted device 900. The position sensor 940 may include a position sensor, an IMU, or both. Some embodiments of the head mounted device 900 may include or exclude the position sensor 940, or may include more than one position sensor 940. In an embodiment where the position sensor 940 includes an IMU, the IMU generates IMU data based on the measurement signals from the position sensor 940. Examples of the position sensor 940 include: one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects movement, a type of sensor used for error correction of the IMU, or some combination thereof. The position sensor 940 may be located outside the IMU, inside the IMU, or some combination thereof.

[0126] Based on one or more measurement signals, the position sensor 940 estimates the current position of the head mounted device 900 relative to the initial position of the head mounted device 900. The estimated position may include the positioning of the head mounted device 900 and / or the orientation of the head mounted device 900 or the head of the user wearing the head mounted device 900, or some combination thereof. The orientation may correspond to the position of each ear relative to a reference point. In some embodiments, the position sensor 940 uses the depth information and / or the absolute position information from the DCA 925 to estimate the current position of the head mounted device 900. The position sensor 940 may include multiple accelerometers that measure translational motion (forward / backward, up / down, left / right) and multiple gyroscopes that measure rotational motion (e.g., pitch, yaw, roll). In some embodiments, the IMU samples the measurement signals quickly and calculates the estimated position of the head mounted device 900 based on the sampled data. For example, the IMU integrates the measurement signal received from the accelerometer over time to estimate a velocity vector, and integrates the velocity vector over time to determine an estimated position of a reference point on the head mounted device 900. A reference point is a point that can be used to describe the position of the head mounted device 900. Although a reference point may generally be defined as a point in an area; however, in practice, a reference point is defined as a point within the head mounted device 900.

[0127] The audio component renders the audio content to incorporate the local effects of the room mode. The audio component of the head mounted device 900 is the above combined Figure 61 and 1 . An embodiment of an audio component 600 is described. In some embodiments, the audio component sends a query for an acoustic filter to an audio server (e.g., audio server 400). The audio component receives room mode parameters from the audio server and generates an acoustic filter to render the audio content. The acoustic filter may include an infinite impulse response filter and / or an all-pass filter having a Q value and a gain at a modal frequency of the room mode. In some embodiments, the audio component includes speakers 915a and 915b, an acoustic sensor array 935, and an audio controller 920.

[0128] Speakers 915a and 915b produce sound for the user's ears. Speakers 915a, 915b are Figure 6 6. Speakers 915a and 915b receive audio instructions from audio controller 920 to produce sound. Speaker 915a can obtain the left audio channel from audio controller 920, and speaker 915b can obtain the right audio channel from audio controller 920. Fig. 9 As shown, each speaker 915a, 915b is coupled to an end piece of the frame 905 and is placed in front of the entrance of the user's corresponding ear. Although the speakers 915a and 915b are shown outside the frame 905, the speakers 915a and 915b can be enclosed in the frame 905. In some embodiments, instead of a separate speaker 915a and 915b for each ear, the headgear 310 includes a speaker array integrated into, for example, the end piece of the frame 905 (not shown). Fig. 9 ), to improve the directionality of the presented audio content.

[0129] The acoustic sensor array 935 monitors and records sounds in a local area surrounding some or all of the head mounted device 310. The acoustic sensor array 935 is Figure 6 An embodiment of the microphone assembly 620. Fig. 9 As shown, the acoustic sensor array 935 includes a plurality of acoustic sensors having a plurality of acoustic detection locations located on the head mounted device 310 .

[0130] The audio controller 920 requests one or more room mode parameters from an audio server (e.g., audio server 400) by sending a room mode query to the audio server. The room mode query includes target area information, user information, audio content information, some other information that the audio server 320 can use to determine the acoustic filter, or some combination thereof. In some embodiments, the audio controller 920 generates the room mode query based on information from a console (e.g., console 860) connected to the head mounted device 900. The audio controller 920 can generate visual information describing at least a portion of the target area based on an image of the target area. In some embodiments, the audio controller 920 generates the room mode query based on information from other components of the head mounted device 900. For example, the visual information describing at least a portion of the target area can include depth image data captured by the DCA 925 and / or color image data captured by the PCA 930. The user's positioning information can be determined by the position sensor 940.

[0131] The audio controller 920 generates an acoustic filter based on the room mode parameters received from the audio server. The audio controller 920 provides audio instructions to the speakers 915a, 915b for generating sound by using the acoustic filter so that the local effect of the room mode of the target area is incorporated into the sound. The audio controller 920 may be Figure 6 An embodiment of the audio controller 630.

[0132] In one embodiment, the communication module (eg, transceiver) may be integrated into the audio controller 920. In another embodiment, the communication module may be external to the audio controller 920 and integrated into the framework 905 as a separate module coupled to the audio controller 920.

[0133] Additional configuration information

[0134] The foregoing description of the embodiments of the present disclosure has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Those skilled in the relevant art may recognize that many modifications and variations are possible in light of the above disclosure.

[0135] Some parts of this description describe embodiments of the present disclosure from the aspect of algorithms and symbolic representations of operations on information. Those skilled in the art of data processing typically use these algorithmic descriptions and representations to effectively convey the essence of their work to other technicians in the field. Although these operations are described functionally, computationally, or logically, it should be understood that they will be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, it is sometimes proven to be convenient to refer to these arrangements of operations as modules without loss of generality. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0136] Any steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules alone or in combination with other devices. In one embodiment, the software modules are implemented using a computer program product including a computer-readable medium containing computer program code, which may be executed by a computer processor to perform any or all of the steps, operations or processes described.

[0137] Embodiments of the present disclosure may also relate to an apparatus for performing the operations described herein. The apparatus may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory, tangible computer-readable storage medium, or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. In addition, any computing system mentioned in the specification may include a single processor, or may be an architecture that employs a multi-processor design to increase computing power.

[0138] Embodiments of the present disclosure may also relate to products produced by the computing processes described herein. Such products may include information obtained from the computing processes, wherein the information is stored on a non-transitory, tangible computer-readable storage medium, and may include any embodiment of a computer program product or other data combination described herein.

[0139] Finally, the language used in the specification is selected primarily for readability and instructional purposes, and it may not be selected to describe or limit the inventive subject matter. Therefore, it is intended that the scope of the disclosure be limited not by this detailed description, but by any claims issued on an application based thereon. Therefore, the disclosure of the embodiments is intended to be illustrative, rather than restrictive, of the scope of the disclosure, which is set forth in the appended claims.

Claims

1. A method comprising: determining one or more room mode parameters associated with the target area based on a position of a user within the target area and a model of the target area, The one or more room mode parameters describe an acoustic filter that is used to present audio content to a user, and the acoustic filter simulates acoustic distortion at the user's location when applied to the audio content. 2 . The method according to claim 1 , wherein the model of the target area is determined based on a three-dimensional virtual representation of the target area. 3 . The method of claim 2 , wherein the three-dimensional virtual representation of the target area is generated by using depth information of at least a portion of the target area.

4. The method according to claim 3, further comprising: The depth information is received from a head mounted device, the head mounted device being configured to present the audio content to a user.

5. The method according to claim 1, further comprising: Determining a model of the target area based in part on the three-dimensional virtual representation of the target area by: comparing the three-dimensional virtual representation to a plurality of candidate models; and A candidate model among the plurality of candidate models that matches the three-dimensional virtual representation is identified as a model of the target area.

6. The method according to claim 1, further comprising: receiving image data of at least a portion of the target area; determining a material composition of a surface in the portion of the target area using the image data; determining an attenuation parameter for each surface based on a material composition of the surface; and The model is updated with the attenuation parameters for each surface.

7. The method of claim 1 , wherein determining the one or more room mode parameters associated with the target area comprises: The room mode is determined based on a shape of the model of the target area.

8. The method of claim 1, wherein the acoustic distortion describes amplification as a function of frequency. The method of claim 8 , wherein the target area is different from the user's physical environment.

10. The method according to claim 1, further comprising: Parameters describing the acoustic filter are transmitted to a head mounted device for rendering the audio content at the head mounted device.

11. A system comprising: Computer processors; as well as A non-transitory computer-readable storage medium storing executable computer program instructions, the computer program instructions comprising instructions that when executed cause the computer processor to perform steps comprising: determining one or more room mode parameters associated with the target area based on a position of a user within the target area and a model of the target area, The one or more room mode parameters describe an acoustic filter that is used to present audio content to a user, and the acoustic filter simulates acoustic distortion at the user's location when applied to the audio content.

12. The system of claim 11, wherein the model of the target area is determined based on a three-dimensional virtual representation of the target area. 13 . The system of claim 12 , wherein the three-dimensional virtual representation of the target area is generated by using depth information of at least a portion of the target area.

14. The system of claim 13, wherein the steps further comprise: The depth information is received from a head mounted device, the head mounted device being configured to present the audio content to a user.

15. The system of claim 11, wherein the steps further comprise: Determining a model of the target area based in part on the three-dimensional virtual representation of the target area by: comparing the three-dimensional virtual representation to a plurality of candidate models; and A candidate model among the plurality of candidate models that matches the three-dimensional virtual representation is identified as a model of the target area.

16. The system of claim 11, wherein the steps further comprise: receiving image data of at least a portion of the target area; determining a material composition of a surface in the portion of the target area using the image data; determining an attenuation parameter for each surface based on a material composition of the surface; and The model is updated with the attenuation parameters for each surface.

17. The system of claim 11, wherein determining the one or more room mode parameters associated with the target area comprises: The room mode is determined based on a shape of the model of the target area.

18. The system of claim 11, wherein the acoustic distortion describes amplification as a function of frequency.

19. The system of claim 11, wherein the steps further comprise: Parameters describing the acoustic filter are transmitted to a head mounted device for rendering the audio content at the head mounted device.

20. The system of claim 11, wherein the target area is distinct from a user's physical environment.

21. A non-transitory computer-readable storage medium configured to store program code instructions that, when executed by a processor, cause the processor to perform steps comprising: determining one or more room mode parameters associated with the target area based on a position of a user within the target area and a model of the target area, The one or more room mode parameters describe an acoustic filter that is used to present audio content to a user, and the acoustic filter simulates acoustic distortion at the user's location when applied to the audio content.

Images