User interface for controlling audio zones

Abstract

The present invention discloses a technique and a device, which includes a memory configured to store audio data in a first audio region or a second audio region in a layered sound field. The memory is coupled to one or more processors, and the memory is configured to store the audio data in the first audio region and the second audio region in the layered sound field. The one or more processors are configured to receive an interactive command to control the audio data in the first audio region and the second audio region in the layered sound field, and to generate one or more indicators of receiving an interactive command to control the audio data in the first audio region or the second audio region in the layered sound field.

Description

[0001] Claiming priority under 35 USC § 119

[0002] This patent application claims priority to non-provisional application No. 16 / 058,785, filed on August 8, 2018, entitled “USER INTERFACE FOR CONTROLLING AUDIO ZONES,” which is assigned to the present assignee and is hereby expressly incorporated by reference into the present invention. Technical Field

[0003] The present invention relates to user interfaces for media data, such as audio data. Background Art

[0004] In recent years, there has been a growing interest in augmented reality (AR), virtual reality (VR), and mixed reality (MR) technologies. The development of image processing and computer vision technologies into the wireless domain has led to better rendering and allocation of computing resources to improve the visual quality and immersive visual experience of these technologies.

[0005] In VR technology, a head-mounted display can be used to present virtual information to a user, so that the user can visually experience an artificial world on a screen in front of their eyes. In AR technology, the real world is enhanced by superimposed visual objects, or superimposed on physical objects in the real world. The enhancement can insert new visual objects or disguise the real world environment with visual objects. In MR technology, the boundary between what is real or synthetic / virtual and what the user visually experiences is becoming difficult to discern. Summary of the invention

[0006] The present invention generally relates to auditory aspects of user experience of computer-mediated reality systems, including virtual reality (VR), mixed reality (MR), augmented reality (AR), computer vision and graphics systems. The technology may include a memory configured to store audio data in a first audio region or a second audio region of a layered sound field. The memory may be coupled to one or more processors configured to receive interactive commands to control audio data in a first audio region or a second audio region in the layered sound field. The one or more processors may also be configured to generate one or more indicators of receiving interactive commands to control audio data in a first audio region or a second audio region in the layered sound field.

[0007] The technology may also include a method that includes receiving an interactive command to control audio data in a first audio region or a second audio region in a layered sound field. The method may also include generating one or more indicators that an interactive command is received to control audio data in a first audio region or a second audio region in a layered sound field.

[0008] In addition, the technology can be implemented in a device that includes a component for receiving an interactive command to control audio data in a first audio area or a second audio area in a layered sound field. The device can also include a component for generating one or more indicators that an interactive command is received to control audio data in the first audio area or the second audio area in the layered sound field.

[0009] The details of one or more examples of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of various aspects of the technology will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Fig.-1a shows a representation of a display device, which may be an AR, MR or VR device.

[0011] Fig.-1b shows a representation of a display device, which may be an AR, MR or VR device.

[0012] FIG. 1c shows a representation of a context associated with one or more audio regions that may be used in performing various aspects of the techniques in this disclosure.

[0013] Fig. 2a shows a representation of two views. The first view on the left portion of the figure is a representation of one or more audio regions associated with the initial user position. The second view on the right portion of the figure is a representation of one or more audio regions associated with the user position after translation and / or directional movement of the user.

[0014] Fig. 2b shows a visualization of a layered sound field and two views representing the positions of the first and second devices.

[0015] Fig-2c shows a representation of an audio blocking application.

[0016] Fig. 2d and Fig. 2e illustrate amplification and / or attenuation of audio data in different audio regions in a layered sound field.

[0017] Figure 2f Tracking of an audio region is shown, the audio region moving in proportion to the movement of the device from a first device position to a second device position, but a second audio region remaining static relative to the movement of the device from the first device position to the second device position.

[0018] Figure 3a and Figure 3b An external object that interacts with the display device is shown.

[0019] FIG. 4 is a block diagram illustrating blocks that may be used to implement techniques according to various aspects of the present invention.

[0020] FIG. 5a is a block diagram illustrating example inputs and outputs of a regional control unit that may be used to perform various aspects of the present invention.

[0021] FIG. 5 b is another block diagram illustrating example inputs and outputs of a regional control unit that may be used to perform various aspects of the present invention.

[0022] Figure-5c is another block diagram showing more granularity around the inputs and outputs of a regional control unit that can be used to perform various aspects of the present invention.

[0023] FIG. 6 is a block diagram illustrating one embodiment of a regional control unit that may be used to implement various aspects of the present invention.

[0024] FIG. 7a is a block diagram illustrating a second embodiment of a regional control unit that may be used to implement various aspects of the present invention.

[0025] FIG. 7 b is a block diagram illustrating a third embodiment of a regional control unit that may be used to implement various aspects of the present invention.

[0026] FIG. 8 is a flow chart illustrating an example process that may be performed in accordance with various aspects of the present invention. DETAILED DESCRIPTION

[0027] The specific embodiments of the present invention are described below with reference to the accompanying drawings. In the specification, common features are designated by common reference numerals throughout the accompanying drawings. As used in the present invention, various terms are only used for the purpose of describing specific embodiments and are not intended to be limited. For example, unless the context clearly indicates otherwise, the singular forms "one", "one" and "the" are also intended to include plural forms. It can be further understood that the term "comprising" can be used interchangeably with "including". In addition, it will be understood that the term "wherein" can be used interchangeably with "wherein". As used in the present invention, "exemplary" can indicate examples, embodiments and / or an aspect, and should not be interpreted as limiting or indicating preferences or preferred embodiments. As used in the present invention, ordinal terms (e.g., "first", "second", "third", etc.) for modifying elements such as structures, components, operations, etc. do not themselves indicate any priority or order of the element relative to another element, but only distinguish the element from another element with the same name (but for the use of ordinal terms). As used in the present invention, the term "set" refers to the grouping of one or more elements, and the term "multiple" refers to multiple elements.

[0028] As used in the present invention, "coupling" may include "communication coupling", "electrical coupling" or "physical coupling", and may also (or alternatively) include any combination thereof. Two devices (or components) may be directly or indirectly coupled (e.g., communication coupling, electrical coupling or physical coupling) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof), etc. By way of example and not limitation, two electrically coupled devices (or components) may be included in the same device or different devices, and may be connected via electronic devices, one or more connectors, or inductive coupling. In some embodiments, two devices (or components) that are communicatively coupled (such as in an electrically communicative manner) may directly or indirectly (such as via one or more wires, buses, networks, etc.) send and receive electrical signals (digital signals or analog signals). As used in the present invention, "direct coupling" may include two devices that are coupled (e.g., communication coupling, electrical coupling or physical coupling) without an intermediate component.

[0029] As used in this invention, "integrated" can include "manufactured or sold with." A device may be integrated if a user purchases a product package that bundles the device or includes the device as part of the package. In some descriptions, two devices may be coupled but not necessarily integrated (for example, different peripheral devices may not be integrated into a command device, but may still be "coupled"). Another example may be any transceiver or antenna described in this invention that may be "coupled" to a processor, but is not necessarily part of a product package that includes an AR, VR, or MR device. When the term "integrated" is used, other examples may be inferred from the context disclosed herein (including this paragraph).

[0030] As used in the present invention, a "wireless" connection between devices can be based on various wireless technologies, such as Bluetooth, Wireless Fidelity (Wi-Fi), or a variant of Wi-Fi (e.g., Wi-Fi Direct). Devices can be "wirelessly connected" based on different cellular communication systems, such as a Long Term Evolution (LTE) system, a Code Division Multiple Access (CDMA) system, a Global System for Mobile Communications (GSM) system, a Wireless Local Area Network (WLAN) system, or some other wireless system. A CDMA system can implement Wideband CDMA (WCDMA), CDMA 1X, Evolution Data Optimized (EVDO), Time Division Synchronous CDMA (TD-SCDMA), or some other version of CDMA. In addition, when the two devices are in line of sight, the "wireless connection" can also be based on other wireless technologies, such as ultrasound, infrared, pulsed radio frequency electromagnetic energy, structured light, or direction of arrival techniques used in signal processing (e.g., audio signal processing or radio frequency processing).

[0031] As used in the present invention, A "and / or" B may mean "A and B", or "A or B", or both "A and B" and "A or B" are applicable or acceptable.

[0032] The term "computing device" is generally used in this invention to refer to any or all of a server, a personal computer, a laptop computer, a tablet computer, a mobile device, a cellular telephone, a smartbook, an ultrabook, a palmtop, a personal data assistant (PDA), a wireless email receiver, an Internet-enabled multimedia cellular telephone, a Global Positioning System (GPS) receiver, a wireless game controller, and similar electronic devices that include a programmable processor and circuitry for wirelessly sending and / or receiving information.

[0033] As used in the present invention, independent control can include manipulation, adjustment, or rotation of audio data in a first audio zone independently of a second audio zone. For example, in one audio zone, audio data can be compensated for loudness (an example of adjustment), while in another audio zone, audio data can be compressed for dynamic range. In another example, audio zones, audio data in one audio zone can be rotated, while audio zones in another audio zone can be equalized. Typically, audio data exists in the entire sound field and is manipulated or adjusted all at once without the ability to split the three-dimensional sound field into different audio zones as described in the present invention. If audio data that is part of the sound field is manipulated or adjusted, it will not be completed in two audio zones with the ability to control two different types of operations (e.g., loudness compensation, equalization, rotation, dynamic range compression, etc.).

[0034] As used in the present invention, "mapping points" may include sampling points or samples of the sound field of an audible scene. The audible scene is three-dimensional, and the sampling points may be evenly distributed or unevenly distributed within the three-dimensional sound field. The mapping points may be visualized, for example, as part of a user interface, but do not necessarily have to be visualized. AR, VR, and / or MR devices may include a display that projects a visual scene. In the case where the audible scene and the visual scene coexist to represent a three-dimensional space, the mapping points may represent pixels of a video object, because the sound in the sound field may be associated with or near a video object. There may be different types of audio formats, where the audio format is a scene-based audio format, and the mapping points may be anchor points, but do not necessarily have to be anchor points. Mapping points (o) may also be points of interest (x). An audio object or source or ambient sound source may contain several mapping points. Mapping points may be clustered together by association with an audio object or source or ambient sound source to form one or more points of interest to represent an audio object or source or ambient sound source. In one example, an audio source or object may be a "pulse", in which case the mapping points and points of interest are the same.

[0035] As used in the present invention, a point of interest can be a cluster of several mapped points. In some examples, the location of a sound source or ambient sound can be an audio point of interest. In other examples, a video object or pixel cluster can be used to generate a visual point of interest. Visual points of interest and audio points of interest can be included in a cluster of mapped points.

[0036] As used in the present invention, a unit may include, for example, dedicated hard-wired circuitry, software and / or firmware in conjunction with programmable circuitry, or a combination thereof.

[0037] References in this specification to "an embodiment," "one embodiment," etc., mean that the particular feature, structure, or characteristic being described is included in at least one embodiment of the present invention. The appearance of such phrases in this specification does not necessarily refer to the same embodiment. On the other hand, different embodiments may not be mutually exclusive.

[0038] An audio region is a collection of points of interest. The size of the audio region is based on including one or more audio sound sources within a spatial region of the sound field. When including visual objects that produce sound, the audio region may include the visual objects from which the sound is emitted and contain audio data including one audio source or multiple audio sources and / or ambient sound sources.

[0039] The terms “AR device,” “MR device,” and / or “VR device” may be integrated with any electronic device that includes circuitry for wirelessly transmitting and / or receiving information, including cellular telephones, personal or mobile multimedia players, watches, wrist displays, medical devices, headsets, headphones and / or standalone AR / MR / VR devices, speakers, microphones, computing devices, and other similar electronic products.

[0040] While full three-dimensional audible rendering still presents challenges, the techniques in this invention bring one step closer to that goal. The audio aspects of AR, MR, and / or VR can be divided into three separate immersion categories. The first category provides the lowest level of immersion and is referred to as three degrees of freedom (3DOF). 3DOF refers to audio rendering that takes into account movement of the head in three degrees of freedom (yaw, pitch, and roll), allowing the user to look around freely in any direction. However, 3DOF does not address the issue of translational (and directional) head movement (in which the center of the head is not at the optical and acoustic center of the sound field).

[0041] The second category, called 3DOF plus (3DOF+), provides three degrees of freedom (yaw, pitch, and roll) in addition to the limited spatial translational (and directional) movement caused by moving the head away from the optical and acoustic centers in the sound field. 3DOF+ can provide support for perceptual effects such as motion parallax, which can enhance immersion.

[0042] The third category, referred to as six degrees of freedom (6DOF), renders audio data in a way that takes into account three degrees of freedom in terms of head movement (yaw, pitch, and roll) but also takes into account the translation of the person in space (x, y, and z translation). Spatial translation can be caused by sensors that track the person's position in the physical world or by input controllers.

[0043] The audio aspects of VR may be less immersive than the video aspects, potentially reducing the overall immersion experienced by the user. However, with advances in processors and wireless connectivity, 6DOF rendering may be possible with wearable AR, MR, and / or VR devices. In addition, the movement of vehicles that have the functionality of AR, MR, and / or VR devices and provide an immersive audio experience may be considered in the future. In addition, those of ordinary skill in the art will recognize that mobile devices such as mobile phones may also implement VR, AR, and / or MR technologies.

[0044] According to the techniques described in the present invention, various ways of adjusting audio data (whether in an audio channel format, an audio object format, and / or an audio scene-based format) can allow for 6DOF audio rendering. 6DOF rendering provides a more immersive listening experience by rendering audio data in a manner that simultaneously accounts for both three degrees of freedom in terms of head movement (yaw, pitch, and roll) and translational movement (in a spatial three-dimensional coordinate system - x, y, z). In implementations, adjustments can be made to provide 6DOF rendering in situations where head movement may not be centered around the optical and acoustic center, and is not necessarily limited to a spatial two-dimensional coordinate system. As disclosed in the present invention, the following figures and description allow for 6DOF audio rendering.

[0045] As disclosed in the present invention, the following figures and descriptions allow 6DOF audio rendering and user interface techniques that allow users to select to enable / disable 3DOF, 3DOF+ and / or 6DOF head tracking. In addition, user interface techniques are described for attenuating or amplifying audio data within different audio zones and in a layered sound field.

[0046] Figure 1aA representation of a display device 102 is shown, which may be an AR, MR, and / or VR device. A person may wear the device 102. The device 102 may be an AR, MR, and / or VR device. The device 102 may include one or more sensors, which may include an orientation / translation unit, a vision unit, and / or an audio capture device. Although the device 102 is drawn as a wearable device, the device 102 may have other form factors. For example, the device 102 may be included in a vehicle (not explicitly drawn), or a robot (not explicitly drawn), or other electronic devices with AR MR or VR capabilities (not explicitly drawn). The device 102 may be coupled to a speaker associated with the device 102. The speaker may also be integrated with the device 102, or may be external to the device 102.

[0047] The AR, VR, and / or MR device 102 may also include other types of wearable devices in addition to those shown. For example, a watch (such as a so-called "smart watch"), glasses (such as so-called "smart glasses"), headphones (including wireless headphones coupled via a wireless connection, or smart headphones coupled via a wired or wireless connection), and any other type of wearable device may be an AR, VR, and / or MR device 102. In this way, the various techniques disclosed in the present invention may be performed by several wearable devices, and when worn by a user, the user may interact with the wearable device through these techniques.

[0048] Figure 1b is a representation of a display device that may be an AR, MR, and / or VR device. A person may wear device 140. Device 140 may be an AR, MR, or VR device. Device 140 may include an orientation / translation unit, a visual sensor, and / or an audio capture device. The orientation / translation unit may provide one or more measurements representing the movement of device 140. Although device 140 is drawn as a wearable device, device 140 may have other form factors. For example, device 140 may be included in a vehicle (not explicitly drawn), or a robot (not explicitly drawn), or other electronic device with AR, MR, or VR capabilities (not explicitly drawn).

[0049] Similar to Figure 1a In addition to the device 102 shown in FIG. 1 , the AR, VR, and / or MR device 140 may also include other types of wearable devices in addition to those shown. For example, a watch (such as a so-called "smart watch"), glasses (such as so-called "smart glasses"), headphones (including wireless headphones coupled via a wireless connection, or smart headphones coupled via a wired or wireless connection), and any other type of wearable device may be an AR, VR, and / or MR device 140.

[0050] A device that covers a person's eyes and does not allow light to enter from the outside can behave as Figure 1b AR, MR, and / or VR devices may be associated with a VR device. However, AR, MR, and / or VR devices may have different form factors. For example, an AR, MR, and / or VR device may have a display screen that allows viewing of external video objects or video scenes, or alternatively, multiple display screens (e.g., a semi-transparent screen or a projection screen) may be included in a device that performs AR, MR, and / or VR. Thus, devices such as Figure 1a or Figure 1b The form factor can be used in AR, MR and / or VR devices. Figure 1b The device 140 is also coupled to a speaker 144.

[0051] Figure 1c A representation of a scene associated with one or more audio regions that may be used when performing various aspects of the techniques in this disclosure is shown. Figure 1c The representation may be displayed in two dimensions or three dimensions. The representation may include visual objects, audible objects, and / or visual objects and audible objects. A portion of the display screen 90 on the display device 100 (see Figure 4 ) can be used to represent visual objects in the scene. For example, in Figure 1c In the embodiment of the present invention, there is a portion of a dog 7A at the bottom left of the display screen 90, a woman 7B near the top of the display screen 90, a man 7C talking to the woman 7B near the top of the display screen 90, and another man 7C at the bottom of the display screen 90. Mapping points (o) associated with AR, MR and / or VR devices are superimposed on the display screen 90. The mapping points can be evenly spaced or non-evenly spaced. The mapping points can be evenly spaced or non-evenly spaced. The mapping points can be visualized or not. Neither the mapping points nor the points of interest in the present invention need to be anchor points. The audio regions and the tracking of the audio regions can allow for more independent flexibility in processing and controlling the audio data. The audio data being manipulated is not limited to the ambient stereo domain (as in scene-based audio formats but used with other formats (e.g., channel and / or object formats)). In addition, video objects can be associated with audio regions, allowing, in some embodiments, the use of facial detection technology that is already readily available in different mobile devices.

[0052] Figure 2aA representation of two views is shown. The first view on the left portion of the figure is a representation of one or more audio regions associated with an initial user position. The second view on the right portion of the figure is a representation of one or more audio regions associated with the user position after translation and / or directional movement. Person 40 may wear an AR, MR, and / or VR device (102, 140). The AR, MR, and / or VR device (102, 140) may be coupled to a display device 100. The display device 100 may be configured to represent an audio region, which may include audio data distributed spatially. For example, for illustrative purposes, an audio region is represented as a square with a dashed line around the border of the audio region. An audio region may be associated with a visual object, an audio object, and / or both a visual object and an audio object. In one embodiment, Figure 1c The mapping point (o) shown in FIG. 1 can be mapped to an interest point (x). When the scene consists of a visual scene and an audible scene, the mapping point can be a pixel. In an alternative embodiment, when the scene is an audible scene, the mapping point can be a sampling point for sampling the sound field.

[0053] The AR, MR, and / or VR device (102, 140) may include one or more processors including Figure 4 The device 400 may be coupled to the memory 120. The memory 120 may be configured to store the audio data in two or more audio regions associated with one or more points of interest, wherein one or more points of interest are associated with each of the two or more audio regions. Figure 2a , four audio regions (20A, 20B, 20C, 20D) are displayed in each of the left view (initial user position) and the right view (after panning / directional movement). Points of interest are represented by "X". In one embodiment, the points of interest may be the same as the sampling points. In another embodiment, the points of interest may be the same as the mapping points. In yet another embodiment, the points of interest are located near where there is sound or where there may be sound. For example, in the audio region 20A associated with the dog 7A, there is a sound 60A in the audio region 20A. The sound 60A may be represented by audio data in the audio region 20A. The audio data may be associated with one point of interest in the audio region 20A. Similarly, the sound 60B in the audio region 20B associated with the woman 7B may be represented by audio data associated with two points of interest. The sound 60C in the audio region 20C associated with the man 7C speaking to the woman 7B may be represented by audio data associated with three points of interest. In general, audio data may be represented by audio data associated with one or more points of interest. The number of points of interest may be a function of the loudness of the sound and the spatial distribution of the sound.

[0054] The AR, MR and / or VR device (102, 140) may be wearable ( Figure 2a (not explicitly shown in the figure). There may be reference points or points on the wearable device for one or more audio zones. For example, the point between the person's eyes may be considered a reference point, or if there is a headband or strip that goes over the person's head, the midpoint of the strip or headband may also be considered a reference point. Other points located as part of the person or part of the wearable device may be considered reference points. Figure 2a , the left view represents a reference point with a star 66. The right view represents a reference point with a different star 67. One or more processors in the AR, MR and / or VR device can obtain a first distance 62 between the audio area 20C and the reference point 66. The reference point 66 is related to the initial position of the person 40. Therefore, if the person is wearing an AR, MR and / or VR device, the device position can be used as a reference point. For example, one or more processors can obtain the first distance 62 between the audio area 20C and the first device position. After the person makes a translational movement (e.g., the person moves either forward, backward, downward, upward, or sideways), the person moves from the first device position to the second device position. After the person makes a rotational movement (i.e., the person turns his / her head clockwise, counterclockwise, or up and down), the device (102, 140) moves from the first device position to the second device position. This movement (whether translational or rotational) is designated as the updated movement 17 when the first device position moves to the second device position. Thus, the one or more processors may obtain an updated first distance 62 after the movement of the device (102, 140) has changed from the first device location to the second device location. When there is at least a second audio area, there may be a distance (not explicitly shown) between the first device location (or a reference point on a person at the initial user location). The one or more processors may obtain a distance (second distance) between the first device location and the second audio area. After the movement of the device (102, 140) has changed from the first device location to the second device location, the one or more processors may obtain an updated second distance between the second audio area and the first device location.

[0055] The orientation / translation sensor unit 404 in the device (102, 140) may represent a microelectromechanical system (MEMS) for sensing or a combination of any other type of sensors capable of providing information to support head and / or body tracking. In one example, the orientation / translation sensor unit 404 may represent a MEMS for sensing translational movement, similar to those used in cellular telephones such as so-called "smart phones". More information about MEMS for sensing used to perform head tracking can be found in the paper entitled "Head Tracking for the Oculus Rift" by LaValle et al., accessible on August 17, 2017 at the URL address msl.cs.illinois.edu / ~lavalle / papers / LavYerKatAnt14.pdf. Additionally, more information on tracking the head and body using cameras and various other sensors, such as vision sensor 130 (which can also be used to track head and / or body movements), can be found in the paper entitled “Interior Design and Navigation in Virtual Reality” by Jesper Tingvall et al., dated November 1, 2015.

[0056] In an alternative embodiment, the person 40 may be in a vehicle into which the display device 100 is integrated. The head movement and / or eye movement of the person 40 may be tracked with an internal camera mounted in the vehicle cabin to determine the head movement or eye movement. As part of an electronic device coupled or integrated with the vehicle, there may be one or more processors that may obtain a first distance from the person's head movement and / or eye movement at a first position to an audio area, and then obtain an updated first distance after the head moves and / or the eye moves to a second position. In a similar manner, a second distance to a second audio area may be obtained from the person's head movement and / or eye movement at a first position, and after the head moves and / or the eye moves to the second position, one or more processors in the vehicle may obtain an updated second position. An initial and final position of an alternative device (102, 140) on a wearable device, the first and final positions being based on the person's initial and final head movement and / or eye movement in the vehicle.

[0057] Thus, the AR, MR and / or VR devices (102, 140) may be coupled or integrated into the vehicle. Note that for convenience, the terms (102, 140) are used to refer to Figure 1aand the two devices shown in FIG-1b. The device (102, 140) may further include one or more sensors configured to estimate a distance representing the movement of the device. The estimated distance may be a first distance to a first audio area, or a second distance to a second audio area, or some other distance that enables one or more processors to determine the first distance to the first audio area or the second distance to the second audio area, and takes into account the movement of the device from a first device position to a second device position. The one or more sensors may alternatively be included in a vehicle and help determine a person's head movement or eye movement. In an alternative embodiment, a vehicle may be considered to move from one device position to a second device position, wherein the device position is a vehicle position.

[0058] In addition, the device (102, 140) or the vehicle may include a transmitter configured to transmit the first device location, the second device location, and the difference between the first device location and the second device location, or the first device location, or the second device location, or the difference between the first device location and the second device location.

[0059] In one embodiment, the device (102, 140) or vehicle may include a camera configured to capture one or more images and a region of interest associated with the first audio zone and the second audio zone. If the camera is included in the vehicle, they may be in-cab cameras to capture different audio zones within the vehicle. In an alternative embodiment, the camera may be outside the vehicle and may be an out-cab camera to capture different audio zones outside the vehicle.

[0060] The AR, VR and / or MR device (102, 140) or vehicle may also include one or more speakers configured to receive a speaker feed based on control of the first audio zone and the second audio zone, render the speaker feed and play audio data in the first audio zone or the second audio zone, or play audio data in the first audio zone and the second audio zone.

[0061] The AR, VR and / or MR device (102, 140) or vehicle may include a display device configured to visualize the audio area. Figure 2aIn the figure, four audio areas are shown: audio areas 20A, 20B, 20C and 20D. The display device 100 can be configured to visualize one or more video objects included in at least one audio area of ​​two or more audio areas. For example, a small part of the nose tip of the dog 7A, a woman 7B, a man 7C who speaks to the woman, and another man 7D can all be considered as video objects and can all be visible, i.e., displayed on the display device 100 of the AR, VR and / or MR device (102, 140). In addition, the display device 100 can be configured to visualize two or more audio areas (e.g., any one of the audio areas 60A, 60B, 60C and 60D) in a layered sound field. In addition, the display device 100 can be configured to visualize two or more points of interest in the layered sound field. For example, the visualized point of interest "X" can help the person 40 know where there is audio energy or audio data that can be manipulated.

[0062] Figure 2b A representation of two views is shown. The first view on the left portion of the figure is a representation of one or more audio regions in a layered sound field relative to an initial user position. The second view on the right portion of the figure is a representation of one or more audio regions in the layered sound field relative to the user position after translation and / or directional movement. In both the first view and the second view, the first audio region and the second audio region are located in the layered sound field. The layered sound field can represent the presence of higher resolution audio data between layers. The layered sound field can also represent the relative radial distances between a person or user of a device and different audio regions. As an example, Figure 2b Two sound fields are drawn in , although more may be drawn in alternative embodiments (different concentric rings representing sound fields for different layers of the sound field).

[0063] exist Figure 2b In FIG. 8 , the first layer or sound field may be within the concentric circles 80 with dashed lines and may be referred to as the foreground sound field. Figure 2b, the left view includes two audio areas (20B, 20C) in the foreground sound field, and two audio areas (20A, 20D) outside the foreground sound field. The sound field outside the foreground sound field can be designated as the background sound field. For a given amplitude and sound pressure, the sound decreases inversely proportional to the distance (sounds softer). Therefore, the farther the sound is from the person, the softer the sound is, and it sounds like it is in the background of the scene. But it should be noted that there are different types of sound. For example, there may be dominant sounds or diffuse sounds. The dominant sound can be directional in nature, that is, it seems to come from a certain direction. For example, a dog barking or a person talking is an example of a dominant sound. There may be a fan, wind, or the sound of a chair sliding on the floor. When heard by a person 40, these sounds may not sound from any particular direction and may therefore not be considered as dominant sounds. As described in the present invention, the layered sound field represented in FIG. 2b may include a dominant sound in one of the foreground sound field or the background sound field. Similarly, there may be diffuse sounds in one of the foreground sound field or the background sound field. The background sound field may also include ambient sounds. Ambient sound can be diffuse sound. In some embodiments, ambient sound can also be people talking, but, due to the distance and other sounds in the environment, it may not be possible to make out what they are saying, or if too many people are talking, there will be "multiple chatter", i.e., a lot of small talk by people.

[0064] like Figure 2b As shown in , when the device (102, 140) moves from the first device position to the second device position, the updated movement 17 is obtained by one or more processors. Therefore, after the movement of the device (102, 140) has changed from the first device position to the second device position, the one or more processors can obtain the updated first distance 68 between the first audio area 20C and the person 40. For the background sound field, there may be an ideal "environmental reference distance" that can be used for all audio objects or audio areas in the background sound field. Although it is possible to obtain the distance of each audio area, if they contain dominant sounds (such as the sound of dog 7A barking, or the sound of man 7D talking), it may be more computationally efficient to treat the distance of each audio area or audio object as having an environmental reference distance. Perceptually, it may be more desirable to have an environmental reference distance. Therefore, although the distance between the audio area 20A including the dog 7A and the audio area 20D including the man 7D may be different, in one embodiment, the same value of the environmental reference distance can be assigned to these distances. However, it is possible that there are different environmental reference distances, one for each audio area in the background sound field.

[0065] In other examples, the reference distance is static or defined as a syntax element of the audio bitstream. When defined using a syntax element, the reference distance can be static (such as sent once and therefore static during the experience) or dynamic (such as sent multiple times during the experience, for example, every audio frame or every some periodic or non-periodic number of audio frames).

[0066] In the same or alternative embodiments, audio zone 20A with dog 7A and / or audio zone 20D with man 7D may control these audio zones (e.g., audio zone 20A or audio zone 20D may be the second audio zone in a layered sound field, where the layered sound field is a background sound field).

[0067] One example of control may be loudness compensation for one or more audio regions. Another example of control may be adjusting the direction of one or more audio regions. Yet another example of control may be dynamic range control for one or more audio regions.

[0068] The control of the audio zones may be based on adjustment of loudness compensation, directionality, or dynamic range control of one or more audio zones based on an ambient reference distance. In addition, audio data in a first audio zone (e.g., audio zone 20A) may be configured to be adjusted based on a first loudness compensation based on a first ambient reference distance, and audio data in a second audio zone (e.g., audio zone 20D) may be configured to be adjusted based on a second loudness compensation based on a second ambient reference distance. The ambient reference distances may be the same or different.

[0069] In one embodiment, a first audio region (e.g., audio region 20B with woman 7B and audio region 20C with man 7C speaking to her) is in a layered sound field, and the layered sound field is a foreground sound field. In one embodiment, one or more processors in a device (102, 140) or vehicle may perform loudness compensation based on the foreground sound field, where each distance from an audio region (20B, 20C) to person 40 may be considered its own foreground reference distance. In an alternative embodiment, the foreground reference distance may be the same for each audio region in the foreground sound field.

[0070] In one embodiment, the first audio region of the device (102, 140) can be configured to switch from a first layer of a layered sound field to a second layer of the layered sound field. The switching of an audio region from one layered sound field of an audio region to another layered sound field of an audio region can be, for example, the switching of the audio region 20D from a background sound field to a foreground sound field 80, such as Figure 2b. The sound from the man 7D may change from an ambient sound (which may not be discernible when it is in the background sound field) to a dominant sound (a sound, such as speech, that is more pronounced in the foreground sound field). Another example is the transition of the audio region 20B with the woman 70B, which was originally in the foreground sound field 80, but is in the background sound field after the updated movement 17.

[0071] In one embodiment, a second audio region (e.g., audio region 20D) can be converted from a second layer (e.g., background sound field) of a layered sound field to a first layer (e.g., foreground sound field) of the layered sound field, and audio data in a first audio region (e.g., audio region 20B) can be adjusted based on a first loudness compensation based on a foreground distance, and audio data in a second audio region is configured to include adjustment based on a second loudness compensation based on a second foreground distance. That is, even though the second audio region is in the background sound field, after the updated movement 17, the second audio region has been converted to the foreground sound field. Both audio regions (the first and second audio regions) can be updated based on the foreground distance. The foreground distances are different. However, in some cases, it may be desirable to use only one foreground distance for all audio regions in the foreground sound field 80.

[0072] It should be noted that the display of the AR, MR, and / or VR device (102, 140) can depict a two-dimensional visual scene or a three-dimensional visual scene. Different geodesic dashed lines are used to emphasize that the visual scene and / or the audible scene can be three-dimensional. In some figures, for clarity, geodesic lines are not illustrated; however, a person of ordinary skill in the art will understand that the three-dimensional visual scene or the three-dimensional audible scene is part of the AR, MR, and / or VR device (102, 140).

[0073] The AR, VR and / or MR device (102, 140) or vehicle may include a display device configured to visualize one or more video objects included in at least one audio zone of the two or more audio zones. For example, audio zones 20A, 20B, 20C, and 20D each include video objects: the face of dog 7A, woman 7B, man 7C speaking to the woman, and another man 7D.

[0074] Person 40 can see the audio region displayed on the display device and can give interactive commands to manipulate or control the audio data in the selected audio region. For example, person 40 can touch the audio region, point to the audio region, give the command "audio region 1 (the audio region can be marked)", or can use a controller to touch or click the audio region. In addition, in some embodiments, person 40 can use gestures to grab or touch the audio region.

[0075] The interactive command may be an input modality. For example, a hand, finger, or gesture tracker (e.g., a jump motion sensor) may point to a scene element (i.e., a point of interest, or an audio region, or a video object). A finger, hand, or gesture may be used to tap, swipe, or grab one of the scene elements. In addition, the interactive command may also be in the form of an eye movement or a head gesture. For example, if the person 40 is looking at a particular scene element for a period of time, the eye movement may be used to interact with the scene element and select it. Nodding, shaking the head, or head movement may be used for a head gesture that also interacts with the scene element.

[0076] In addition, the interactive command may be a voice command. For example, an object may be identified in an AR, MR, and / or VR device (102, 140). Once a keyword or object name is associated with the video object or audio object, the voice command may be used to manipulate the video or audio object, for example, to make the object louder or to mute the object. In another embodiment, the interactive command may be a pointer controller, such as a mouse pointer, which may hover over a desired scene element, such as a video or audio object.

[0077] One or more processors may receive an interactive command to control audio data in a first audio region or a second audio region in a layered sound field, and generate one or more indicators that an interactive command has been received to control audio data in a first audio region or a second audio region in a layered sound field.

[0078] The generation of the one or more indicators may be received by one or more processors, and the one or more processors may control audio data in the first audio region, the second audio region, or both the first audio region and the second audio region in the layered sound field.

[0079] Additionally, the display device may be configured to visualize two or more points of interest of the layered sound field. For example, in each region there is an "ex" "X" that represents the visualization of one or more points of interest. The points of interest may be manipulated or controlled by interface input modalities similar to the audio regions, such as touch, pointing, eye movement, nodding, or clicking on one or more points of interest.

[0080] In addition, the display device can be configured to visualize the layered sound field. The foreground sound field 80 is described by a circle, although other shapes can also represent the sound field. For example, the foreground sound field can be represented by a sphere, a cube, a square, a rectangle or other closed surfaces to delimit the spatial boundaries between the sounds near the person 40. Proximity can be relative to a bounded space, and the distance of the sound source is relatively close to the sound detector so that the sound detector detects the level of the sound emitted by the sound source. For example, unless it is a very loud sound (for example, a jet or a very loud concert), the sound emitted by a person or electronic device 500 meters away from the sound detector is usually not heard. However, if the sound is very loud, then 20 meters away may be near-end. But the footsteps may only be more near-end at 1 meter, and the footsteps at 20 meters may be considered too far to enter the foreground sound field.

[0081] In an alternative embodiment, the interactive commands may also be part of a device (e.g., a smart phone) that is part of a mobile device, where the person 40 may hold or use an accessory to transform the smart phone into a wearable device and the display device 100 is integrated into the mobile device.

[0082] In an alternative embodiment, the interactive commands may also be part of a device in a vehicle, wherein the person 40 may be in the vehicle and the display device 100 is integrated into the vehicle.

[0083] The visualization of the foreground sound field can follow the movement of the person 40. Figure 2b In the right view of the foreground sound field, the foreground sound field moves with the person 40. In this example, the audio regions are static. Thus, as the device moves, audio region 20B falls outside the foreground sound field, while audio region 20D enters the foreground sound field. This mode supports an interactive mode in which the person 40 can hear (and visualize) the audio regions near the foreground sound field. In this interactive mode, the effect is more like a real immersive experience, and as you get closer to people or sound sources, they will sound closer.

[0084] For clarity, in Figure 2c , Figure 2d and Figure 2e Only one view is shown in FIG. 1 . Although the features described with reference to those figures are also applicable when the device is moved from one location to another. In addition, for simplicity, Figure 2c The audio area is also not illustrated.

[0085] exist Figure 2cIn the figure, the area around the point of interest has a slash mark in the circle around people 60C and 60D. This is done to illustrate that person 40 can give an interactive command to mute or block the audio in the area. If there is other audio data in the audio area (e.g., audio area 20C and audio area 20D), the audio data in the audio area (all audio data or a portion of the audio data) can be blocked. In addition, in one embodiment, the video objects in the audio area can also be blocked or deleted. For example, man 7C talking to woman 7B is partially blocked visually. However, in another embodiment, man 7C can also be completely blocked visually. Similarly, depending on the embodiment, man 7D can also be partially or completely blocked visually. In a video camera type application, person 40 may want to suppress or eliminate audio and video objects from the recording results, and may want to block (i.e. delete or suppress audio and / or video objects). Another application could be to eliminate interfering signals (e.g., unwanted audio sources) for an audio user interface, or some future modified version of a device such as a personal digital assistant or something like the Amazon spot (a device that currently has both an audio interface and camera capture capabilities).

[0086] exist Figure 2d and Figure 2e In one embodiment, one or more processors may be configured to control the first audio region (eg, 20B or 20D), which may include amplification of audio data in the first audio region. Figure 2d In FIG. 8 , a graphical representation of the amplification and / or attenuation of audio data in the audio region is shown in the foreground sound field 80. Figure 2e , the amplification and / or attenuation of audio data in the audio area is illustrated in the background sound field. These two figures illustrate that the amplification and / or attenuation of audio data (and / or video objects) in the audio area may be in the foreground sound field or in the background sound field. In addition, a person of ordinary skill will readily recognize that the amplification and / or attenuation of audio data and / or video objects can be pre-formed in the layered sound field. That is, the audio data (and / or video object) can be amplified in the foreground sound field 80, while the audio data (and / or video object) can be attenuated in the background sound field. Conversely, the reverse is also possible. That is, the audio data (and / or video object) can be attenuated in the foreground sound field 80, while the audio data (and / or video object) can be amplified in the background sound field.

[0087] Additional features and capabilities are described below. The display device 100 can be configured to visualize an enlarged representation of the audio data in the first audio region. The enlarged visual representation of the audio data in the first audio region can be illustrated by the increased size of the first audio region. Thus, as the size of the first audio region increases, the audio data in the first audio region can be enlarged.

[0088] Additionally, the one or more processors may be configured to control the second audio region (e.g., 20C or 20A) and reduce the size of the second audio region. In this case, the visual representation of the attenuation of the audio data in the second audio region may be illustrated by the reduced size of the second audio region. Thus, as the size of the second audio region decreases, the audio data in the second audio region may be attenuated.

[0089] In one embodiment, the visual representation of the amplification of audio data can be implemented as a flashlight sound application or a blocking sound application of a display device. In this way, the visualization of the amplification or attenuation of the audio data in the first audio area, the second audio area, or both the first audio area and the second audio area can be based on the change of the illumination of the display device screen. For example, in the case where the audio data is amplified or attenuated, the illumination of the area where the sound is amplified or attenuated can be accompanied by the amplification or attenuation of the audio data.

[0090] Although not explicitly illustrated, it is conceivable by those skilled in the art that a portion of the display device 100 may be illuminated more or less when there is amplification or attenuation of sound (i.e., audio data). In this way, changes in the illumination of the screen of the display device may represent changes in different amplification or attenuation levels of audio data in the first audio region, the second audio region, or both the first audio region and the second audio region. The color change used to represent the amplification or attenuation of audio data may not necessarily be the output from a flashlight (e.g., pale yellow) or a sound blocking application (e.g., black), but may be a different color and may be configurable.

[0091] Thus, other colors may be associated with amplification and / or attenuation of audio data. For example, the colors used for amplification and / or attenuation may be configurable. Thus, person 40 may have any of the colors orange, blue, green, etc. to represent amplification, and any of the colors brown, gray, red, etc. to represent attenuation.

[0092] exist Figure 2f, the left view of the figure shows the display device 100 and the layered sound field before the AR, VR and / or VR device moves (the area outside the foreground sound field 80 is the background sound field). The right view of the figure shows the display device 100 and the layered sound field after the AR, VR and / or MR device moves. One or more processors may be configured to track a first audio area (e.g., 20B or 20C) so that it moves in proportion to the movement of the device from the first device position to the second device position, but the second audio area (e.g., 20A or 20D) remains static relative to the second audio area before the device moves from the first device position to the second device position. An application that may require such tracking may be to purposefully modify the sound field. For example, person 40 may want to hear the sound of a dog directly behind the head of person 40 rather than in the direction of the left ear. Similarly, person 40 may want to make the voice of another person behind or deviate from the sound in the foreground sound field 80. Thus, as can be seen, the video objects and audio data within audio areas 20A and / or 20D track with the movement of the AR, VR, and / or MR device, moving with rotation (and translation).

[0093] Figure 3a and Figure 3b External objects that interact with the display device 100 are shown. Finger 311 of hand 310 can be used to point to an area on the display device 100 that can represent an audio element 344 and / or a video object to attenuate and / or amplify audio data or audio data associated with the video object. An audio element can be audio data and / or a video object associated with audio data. A pointer or other device with the ability to indicate direction can also be used. These external objects (hands, fingers, pointers, etc.) can be used to select audio data that is amplified and / or attenuated in the layered sound field. Figure 3b In the embodiment of the present invention, a representation of an external object (e.g., a hand) can be inserted into the sound field and / or enhanced into the visual representation of the sound field on the display device 100. Through such a representation, the person 40 can grab, touch, tap, etc. the audio element 344 and / or the video object to attenuate and / or amplify the audio data or the audio data associated with the video object. The external object can be visualized within the display device 100 and can be used for interactive commands to control the audio data and / or video objects within the layered sound field.

[0094] Figure 4 is a block diagram illustrating blocks that may be used to implement techniques according to various aspects of the present invention. Figure 4 In the embodiment, the AR, MR and / or VR device (102, 140) or the AR, MR and / or VR device integrated into the vehicle may include: Figure 4Components in the device 400 shown in . The device 400 may include one or more processors 402, which may also include a zone control unit 428 integrated into the one or more processors, wherein the zone control unit 428 is configured to receive the updated movement 17. The updated movement 17 can be used to determine the updated first distance or the updated second distance, or both the updated first distance and the updated second distance. The zone tracker 408 in the zone control unit 428 can operate in different audio domains.

[0095] In the same or alternative embodiments, the region control unit 428 includes a region tracker 408 configured to track two or more audio regions and configured to receive an updated movement 17, compare the updated movement 17 with the sound position of an associated audio point of interest, and adjust the translation or rotation, or both, of the sound field in each of the two or more audio regions. It should be noted that in alternative implementations, the audio region tracker functionality may be located in other blocks within the region control unit, but may be referred to by another name, or may be incorporated into the functionality of the region control unit itself. For example, in some implementations, the audio region tracker "tracks" the audio region, but may not necessarily compare the updated movement 17 with the sound position of an associated audio point of interest, and adjust the translation or rotation, or both, of the sound field in each of the two or more audio regions.

[0096] In the same or alternative embodiments, an audio spatialization unit 420 integrated with one or more processors 420 may be configured to output 424 one or more sound sources and corresponding positions of the one or more sound sources. For example, the barking of a dog 7A, or the speech of a woman 7B, or the speech of a man 7C, 7D may be considered as a sound source. The spatial positions of the sounds associated with 7A, 7B, 7C, 7D may be output by the audio spatialization unit 420. These spatial positions may be sound source positions and / or ambient / diffuse positions. In the same or alternative embodiments, spatial positions may be assigned to audio regions based on the size of the audio region and the specific implementation used to track the audio region. In one embodiment, the spatial position may be represented by Cartesian coordinates (x, y, z). In alternative embodiments, the spatial position may be represented by spherical coordinates (r, theta, phi) or (r, azimuth, elevation). In other embodiments, the spatial position may be represented by other coordinate representations.

[0097] In the same or alternative embodiments, the audio spatialization unit 420 is further configured to output ambient sound and a corresponding ambient sound reference distance. The ambient sound may come from a diffuse sound source. The ambient reference distances for different audio zones may be the same or different. It should be noted that throughout the present invention, the terms ambient reference distance, ambient reference, and ambient reference position may be used interchangeably. Although distance and position may be different, in some implementations, position and distance may have the same origin reference point and therefore may have the same meaning.

[0098] In the same or alternative embodiments, the audio spatialization unit 420 is coupled to the audio capture device 112. The audio capture device may include one or more microphones configured to capture an audio signal and output an audio signal representation of the audio signal. The audio capture device 112 may include a linear array of microphones, an array of microphones having different geometric shapes (e.g., a tetrahedron, a cross, an "L" shape). The microphone arrays may be placed on different planes of the AR, MR, and / or AR devices (102, 140). For example, one set of microphones may be located on one side of the device and another set of microphones may be located on the other side of the device (e.g., one side may be the top, bottom, left, or right, depending on the form factor of the device). The number of microphones on each side does not necessarily have to be the same. The audio capture device 112 may also include a circular array of microphones in one or more planes (i.e., there may be concentric circles of microphones in one plane and different concentric circles of microphones in another plane). The number of microphones in different planes does not necessarily have to be the same. Furthermore, the capture device 112 may include a spherical array of microphones, a hemispherical array of microphones (ie, half of a sphere), a quarter sphere of microphones (ie, one quarter of a sphere), or other fractional portions of a sphere.

[0099] In any embodiments described in the present invention, the audio capture device 112 can output an audio signal representation of the audio signal. Depending on the embodiment, the audio signal representation can be an audio waveform. In some embodiments, the audio waveform can be compressed, filtered or encoded. Depending on the embodiment, the audio signal representation can represent the audio signal in the time domain, the time-frequency domain, or the stereo domain.

[0100] In various embodiments, the audio spatialization unit 420 may be integrated into a decoder unit that is also integrated into one or more processors, wherein the decoder unit includes a parsing unit configured to obtain an audio signal representation of the audio signal. The audio signal representation may be an input to the audio spatialization unit 420. When the audio spatialization unit is integrated into the decoder unit, the audio signal representation of the audio signal may be part of a bitstream to be decoded, or in some implementations, the audio signal representation of the audio signal represents a decoded audio signal.

[0101] In the same or alternative embodiments, the audio spatialization unit 420 can be coupled to a storage unit including the memory 120. The audio spatialization unit 420 can be configured to store an audio signal representation, wherein the audio signal representation is an input to the audio spatialization unit 420. The audio signal representation of the audio signal can be a compressed audio signal, a filtered audio signal, and can be stored in different file formats. The one or more processors can be configured to retrieve the audio signal representation stored in the memory 120 in any format by reading (r) the memory 120. To store the audio signal representation of the audio signal, the one or more processors can write (w) the audio signal representation of the audio signal to the memory 120.

[0102] The audio spatialization unit 420 may operate in a channel format mode, an object format mode, and / or a scene-based format mode. In one embodiment, the audio spatialization unit 420 may not support more than one audio format. In different embodiments, the audio spatialization unit 420 may support two or more audio formats. For example, the audio spatialization unit 420 may support an object format and a scene-based format, or the audio spatialization unit 420 may support a channel format and an object format, or the audio spatialization unit 420 may support a channel format and a scene-based format. In another embodiment, all three audio formats may be supported, and there may (optionally) be a format selector 442 that may be used to switch between different formats and help support the audio spatialization unit 420 in the modes in which it may operate (e.g., channel format mode, object format mode, and / or scene-based format mode). Scene-based formats may include different types of stereo formats (e.g., higher-order stereo, mixed-order stereo, B-format (also known as first-order stereo), or some other variants.

[0103] In one embodiment, the one or more visual sensors 130 may output a visual signal representation of a video signal. In one embodiment, the video signal representation may include a set of pixels. The one or more visual sensors may be or be included in a camera and may be configured to capture one or more images and a region of interest associated with the first audio region and the second audio region.

[0104] In various embodiments, one or more processors 402 may receive a visual signal representation via a wireless connection with another device. A transceiver 480 coupled to one or more antennas 490 may be used to establish a wireless connection. In some embodiments, the visual signal representation may be compressed, filtered, or encoded. A bandwidth-encoded visual signal may be wirelessly sent and received between devices by including a video bitstream between the devices. In this case, the video bitstream may be referred to as a visual signal representation of a video signal. The video bitstream may be decoded by a video decoder 475. The video decoder 475 may be integrated with one or more processors 402. After the video decoder 475 decodes the video bitstream, the decoded video signal may be referred to as a reconstructed video signal. Therefore, the reconstructed video signal is also another visual signal representation of the video signal.

[0105] The visual signal representation may be input into the vision unit 410. In one embodiment, the vision unit 410 may include a visual object separator 414. In the same or alternative embodiments, the vision unit 410 may include a depth map unit 416. The depth map unit 416 may read in or accept a depth map received from another device (e.g., transmitted via the transceiver 48), i.e., the depth map may be included in a portion of the video bitstream. In various embodiments, the video decoder 475 may be capable of generating and / or estimating a depth map from the reconstructed stereo image on the fly. In one embodiment, the depth map may provide a grayscale representation of a 360-degree virtual reality scene, where black represents a great distance and white represents a closer distance, along with various grayscale tones between black and white indicating intermediate distances. The AR, VR, and / or MR device (102, 140) may utilize the depth map to formulate a view for the left eye or right eye from the respective right eye view or left eye view specified in the video bitstream (or captured by a visual sensor). The video decoder 475 can change the amount of lateral distance between the right eye view and the left eye view based on the depth map, scaling the lateral distance smaller based on darker shades of gray. In this way, a close object represented by white or light gray shades in the depth map may have a larger lateral distance between the left eye and right eye views, while a far object represented by black or dark shades in the depth map may have a smaller lateral distance between the left eye and right eye views (and thus appear closer like a distant point).

[0106] Although described as being performed with respect to grayscale depth maps, the technique may be performed with respect to other types of information that provide depth information, such as color images, color or grayscale stereo images, infrared camera images, etc. In other words, the technique may be performed with respect to any type of information that provides depth information of a scene associated with corresponding audio data.

[0107] The visual unit 410 coupled to the audio region control unit 428 can utilize the depth information provided by the depth map to adapt or adjust the position of the interest point relative to the head of the person 40. That is, the visual unit 410 coupled to the audio region control unit 428 can map the interest point to the depth map, and utilize the depth information of the depth map at the mapped position within the depth map to identify the more accurate reference distance of the environmental reference distance and / or the sound source position. As disclosed in the present invention, the audio region control unit 428 can utilize the output of the depth map to provide the video object foreground and / or video object background position. In other embodiments, it is possible that the depth map is directly input into the audio region control unit 428, and the varying depth of the depth map is associated with the mapped point to determine the size of the audio region and the interest point in the audio region.

[0108] The audio region control unit 428 may utilize the depth map to estimate the individual distance of each mapping point, pixel, anchor point, point of interest, or audio region. In this way, the audio region control unit 428 may determine an updated distance or an updated movement relative to each of the individually determined distances of the mapping point, pixel, anchor point (when the audio data is scene-based), point of interest, or audio region.

[0109] In embodiments where there are foreground and background sound fields, mapping points within the foreground sound field may be very close (e.g., within 2 meters), and mapping points within the background sound field may be very far (e.g., beyond 2 meters but within a range of no more than 50 meters).

[0110] In an alternative embodiment, the depth map may be loaded into the memory 120 of the AR, MR, and / or VR device (102, 140) and used in conjunction with the visual object separator 414. In an alternative embodiment, the visual unit 410 does not include a visual object separator and may not provide at least one video foreground object (VO_fgnd) and / or one video background object (VO_bgnd), and may include a depth map unit 416, which may help separate different video objects based on depth. Those video objects at greater depths may be referred to as video foreground objects (VO_fgnd), while those video objects at closer depths may be referred to as video background objects (VO_bgnd). Thus, there may be an alternative embodiment drawn that would illustrate four vertical arrows pointing downward from the depth map unit 416. The four arrows will represent (at least one of): (i) video foreground object (VO_fgnd); (ii) video background object; (iii) reference distance of video foreground object (VO_fgnd_ref_dist); (iv) reference distance of video background object (VO_bgnd_ref dist).

[0111] In the same or alternative embodiments, one or more points of interest are associated with a video object and are associated with audio data in a first audio region that is a video object. For example, in some implementations, a dog 7A, a woman 7B, and men 7C, 7D in different audio regions may be video objects. In the same or alternative embodiments, one or more points of interest are visual key points associated with a video object. A video object may be a foreground video object (VO_fgnd). For example, in FIG. 2a, a woman 7B and a man 7C who speaks to the woman 7B may be designated as foreground video objects. A dog 7A and a man 7D who does not speak to the woman 7B may be considered as a background video object (VO_bgnd). A video object may not necessarily be classified as a foreground video object or a background video object, but may simply be referred to as a video object.

[0112] In some embodiments, the vision unit 410 may not include the depth map unit 416. For example, a video object (i.e., a face of a person or dog) may be tracked relative to the person 40 using a facial detection algorithm. The vision unit 410 may include a visual tracker (not shown) to track the video object, and the video object, together with the relative distance of the video object to some other reference point of the person 40, may be the output of the video unit 410. In this case, the mapping point may be associated with the video object, and there may not necessarily be a distinction between associating the mapping point to the foreground video object and associating the mapping point to the video object, because there may be only one video object in those instances. Similarly, there may not necessarily be a distinction between associating the mapping point to the background video object and associating the mapping point to the video object, because there may be only one video object in those instances. In the present invention, those skilled in the art will recognize that even if a foreground video object (or background video object) is used in the example, in some cases, the vision unit 410 does not distinguish between foreground and background video objects, for example, taking a face detector as an example, the description of certain use cases still applies to the video object output by the face detector. In some embodiments, the face detector may be included in the vision unit 410, or the face detector may be outside the vision unit 410.

[0113] In one embodiment, the output of the vision unit 410 and / or the output of the video decoder 475 can be coupled to the display unit 465. The display unit 465 can process the output of the video decoder 475 and / or the vision unit 410. The visualization of the video object can be represented by a bounding box or some other marking and can be superimposed or enhanced to the reconstructed video signal. Thus, an arrow from the vision unit 410 connected to the display unit 465 can indicate that there is a marking or some other indication of a video object that has been detected by the vision unit 410. Similarly, an arrow from the video decoder 475 can indicate that the reconstructed video signal can be processed by the display unit 465. The output of the display unit 465 can be sent to the display device 100 viewed by the person 40. Figure 2a and 2b An example depicting what a person 40 may see on a display device 100 is shown in part in FIG.

[0114] Display unit 465 may be coupled to user interface unit 436. User interface unit 436 may be used to implement control logic associated with the techniques described in this disclosure.

[0115] In one embodiment, one or more points of interest are associated with a video object in the first audio region, and the one or more points of interest are associated with audio data in the first audio region. In the same or alternative embodiments, the one or more points of interest are visual key points associated with the video object. In the same or alternative embodiments, the video object is a foreground video object, or the video object is a background video object. In the same or alternative embodiments, if the video object is a foreground video object, the audio data in the first region is associated with the foreground video object. In the same or alternative embodiments, the audio data is also associated with diffuse sound.

[0116] In one embodiment, if the video object is a background video object, the audio data within the first region is associated with the background video object, and the audio data is also associated with the dominant sound.

[0117] In one embodiment, the AR, VR and / or MR (102, 140) device may also include one or more speakers configured to receive speaker feeds. The speaker feed may be generated by a speaker feed unit 460. The speaker feed unit 460 may convert the output of the audio rendering unit 458 into a speaker feed that may drive one or more speakers 470. The audio rendering unit 458 is configured to employ the output 456 of the audio region effect unit 432. The output 456 may be channel-based audio, object-based audio, or stereo-based audio. The speaker feed may be based on an audio region control unit using a first audio region and a second audio region. One or more speakers 470 may be configured to synthesize the speaker feed and be configured to play out audio data in the first audio region or the second audio region, or be configured to play out audio data in the first audio region or the second audio region. The one or more speakers 470 may be part of a speaker included in a vehicle. Alternatively, the one or more speakers 470 may be binaural speakers as part of the AR, VR and / or MR device 102, 140.

[0118] The binaural speakers may be headphones and coupled to the AR, VR, and / or MR device via a wired connection (such as a standard 3.5 mm audio jack, a universal system bus (USB) connection, an optical audio jack, or other form of wired connection) or wirelessly (such as through a Bluetooth connection, a wireless network connection, etc.). The headphones may recreate the sound field represented by the audio signal representation based on the left and right speaker feeds. The headphones may include a left headphone speaker and a right headphone speaker that are powered (or in other words, driven) by the corresponding left and right speaker feeds.

[0119] The distance from person 40 to the audio region can be expressed as d zone_distance(i) For example, the region distance for audio region 20C is labeled 62. After the updated movement 17 of the device from the first device position to the second device position, the region distance to the audio region may be updated and denoted as d zone_updated_distance(i) .d zone_distance(i) With d zone_updated_distance(i) The ratio of will affect the control of the audio zone. The zone control ratio (zcr) of the "i-th" zone is For each audio region, the ratio of the region distance to the updated region distance (or some other comparison, such as the difference between the numerator and denominator of the ratio, or the logarithmic difference between the numerator and denominator) can be used to control the audio region. The zcr(i) can be calculated in the region control unit 420, or in an alternative embodiment, can be calculated in the region effect unit 432.

[0120] exist Figure 5a , Figure 5band Figure 5c Different embodiments of the zone effect unit 432 are shown in . The audio effect unit 432 may allow independent control of the first audio zone and the second audio zone, ie, audio data in the first audio zone and audio data in the second audio zone may be controlled independently of each other.

[0121] The audio region effects unit 432 may be integrated into one or more processors 402 and may include an adjustment unit 515 configured to adjust audio data in a first audio region by a first audio region factor 525. The adjustment unit 515 adjusts the audio data in the first audio region by one or more first audio region factors 525 based on a change in movement of the device (102, 140) from the first position to the second position. Additionally, the audio effects unit 515 is configured to adjust audio data in a second audio region by a second audio region factor 525 based on a change in movement of the device (102, 140) from the first position to the second position.

[0122] In one embodiment (see FIG. 5 a), the audio region effects unit 432A may include an independent loudness compensation unit 515A, which functions as an adjustment unit 515 and may be configured to perform loudness compensation for a region (a first audio region and / or a second audio region) based on one or more audio region factors 525A used as audio region factors 525 as the device (102, 140) moves from a first position to a second position.

[0123] An example of controlling an audio region is loudness adjustment. A scaling factor (audio region intensity factor 525A) may be multiplied by zcr(i) to increase the intensity of an effect (e.g., the intensity of loudness). The audio region intensity factor 525A may control the intensity of the effect, which may be an intensity or magnitude of the effect intensity input by a person 40 to control. The audio region intensity factor 525A may be controlled by a person 40 (e.g., via a user interface with a button that changes based on touch or voice), or may also be dynamically or statically specified by a content creator.

[0124] The independent loudness adjustment unit 515A may include the application of a loudness factor that may be agnostic to the updated mobile 17. For example, in embodiments where the audio format is a channel format, the independent loudness adjustment unit 515A may include an all-pass filter that passes all audio and does not affect the amplitude of the audio. In the same or alternative embodiments, the loudness adjustment unit 515A may be in a pass-through mode, where the input to the independent loudness adjustment unit 515A is the same as the output of the independent loudness adjustment unit 515A.

[0125] In one embodiment, when the audio format is an object format, the metadata associated with the audio object may include the location of the audio source. In some implementations, the location of the audio source may be the location of the barking dog 7A, the speaking woman 7B, or the speaking men 7C, 7D. The locations of these audio sources may be used to calculate the distance to the audio region, which may be used to update the d zone_updated_distance(i) .

[0126] The updated first distance can be designated as d zone_updated_distance(1) , and the updated second distance can be designated as d zone_updated_distance(2) Similar to when the audio format is a channel format, the independent loudness adjustment unit 515 can include an all-pass filter that passes all audio and does not affect the amplitude of the audio. In the same or alternative embodiments, the loudness adjustment unit 515 can be in a pass-through mode, where the input to the independent loudness adjustment unit 515 is the same as the output of the independent loudness adjustment unit 515.

[0127] In one embodiment, when the audio format is a scene-based format, the mapping points may be mapped to anchor points. Anchor points represent points in space on a sphere centered at the head of the person 40. Anchor points may be used in a stereo domain, such as higher order stereo (HOA) or mixed order stereo (MOA). MOA may be a subset of HOA. For audio regions, a variable "K" is used instead of M (as described below) to distinguish between using all available points on the sphere and selectively using anchor points within a region or that may intersect a region. The variable "K" is greater than or equal to (N+1) 2 , where N represents the highest order associated with the HOA audio data. The K anchor points result in K spatial directions extending from the head of the person 40 to each of the K spatial points, and can be evenly spaced or non-evenly spaced. The K spatial directions can be represented by the subscript "k" k express.

[0128] The HOA coefficients or MOA coefficients can be associated with the K spatial directions as follows:

[0129]

[0130] in

[0131] The symbol can represent a pseudo-inverse matrix operation.

[0132] The symbol can represent a matrix consisting of K direction vectors S of HOA order O. Symbols can be represented by The matrix consisting of the modified (translated and rotated) direction vectors defined in .

[0133] In contrast to the prior art, when the audio format is a scene-based audio format, it is not necessary to consider all anchor points in all M spatial directions around the sphere in the distance-independent loudness adjustment. For example, the number of spatial directions can be limited to the number of spatial directions that intersect the audio region. For example, there may be up to K spatial directions extending from the head of the person 40 to K equidistant spatial points that intersect the audio region. The larger the size of the audio region, the more spatial points that may intersect the audio region. The size of the audio region can be based on the proximity of the person 40 to the audio region. The size of the audio region can also be based on the number of audio objects or audio sources in the audio region. As an example, each audio region can have an anchor point density associated with it. In FIG-2a, the audio region 20B with the woman 7B has a smaller number of points of interest than the audio region 20C with the man 7C. Therefore, the audio region marked 20C has a larger intra-region anchor point density than the audio region marked 20B, and the audio region marked 20B has a lower intra-region anchor point density. However, it should be noted that there may be some points of interest outside the audio region that affect the control of the audio region because they also intersect the audio region. In this embodiment, the points of interest illustrated by the "x" may be anchor points, or may be generated based on the intersection of anchor points to the audio region.

[0134] The total number of anchor points associated with the distance-independent loudness adjustment Azone(l) for each zone includes (a) anchor points intersecting the audio region that are located in the audio region, and (b) also includes anchor points at some number of angles spanning some sub-divisions of a sphere around the head of the person 40 that are not located in the audio region. The total number of anchor points K for each audio region included in the distance-independent loudness adjustment Azone(l) determination for each zone is as follows:

[0135] Where k = 1..K

[0136] The audio region effect unit 432 can use the matrix R (rendering matrix), (a version of the preliminary effects matrix), the distance-independent loudness adjustment Azone(l) for each zone, and the zone control rate to compensate for the updated movement 17. For example, the following equation can be used to compensate for the updated movement 17: Azone(l)*zcr(l).

[0137] The audio effects unit 432 may also feature interactive spatial effects, for example, to attenuate sounds from directions other than the primary view direction that the user is aware of through directional weighting of the M anchor points based on the von Mises distribution; or to "block out or block" sounds from a specific direction determined by the user interface device (e.g., AR, VR, and / or MR device (102, 140)). In this case, the additional weighting vector g fx can be added to the calculation of Azone(l). For example, the following equation can be used: Azone(l)*zcr(l)*g fx .

[0138] Vector fx An example of may be a binary weighting vector [1 0 1 1 0 0 1 0]. Other binary weighting vectors are possible. The zeros in the example binary weighting vector nullify any contribution from those audio regions that are masked or blocked. The ones in the example binary weighting vector allow the audio data in that sound field to be passed through and amplification may be achieved in an audio effects unit (e.g., a loudness adjustment unit).

[0139] In addition, the vector g fx Another example of may be a non-binary weighting vector. For example, for M=3, the original sound field may have anchor points weighted by [(1-.5), (1-.1), (1-.9)]. In another example, the vector may be [0.5 0.1 0.9].

[0140] To account for controlling different audio regions when the audio format is a scene-based audio format, an effects matrix may be applied prior to HOA synthesis and rendering. That is, synthesis unit 450 receives the output of audio region effects unit 432, which may include distance-independent loudness adjustments. Synthesis unit 450 may combine audio data from different layers of scene-based formats. For example, there may be foreground and background layers, each of which may include at least one audio region, and synthesis unit 450 may combine these audio regions. The combination, for example, may be based on adding stereo data within each region in the background layer and the foreground layer.

[0141] In the second embodiment (see Figure 5b), the audio region effects unit 432B may include an orientation / translation adjustment unit 515AB used as an adjustment unit 515, and may be configured to perform orientation and / or translation adjustments for the region (first audio region and / or second audio region) based on one or more audio orientation region factors and / or translation factors 525B used as audio region factors 525 as the device (102, 140) moves from a first position to a second position. An example of controlling the audio region to be an orientation adjustment is that the audio region 20A including the dog 7A can be rotated, that is, the audio data representing the dog's barking can be rotated in a direction toward or away from the person 40. Another example of controlling the audio region to be a translation adjustment is that the audio region 20A including the dog 7A can be translated, that is, the audio data representing the dog's barking can be translated in a direction up, down, forward, backward, or to either side of the user.

[0142] In the third embodiment (see Figure 5c ), the audio region effects unit 432C may include a dynamic range compression unit 515C, which functions as an adjustment unit 515 and may be configured to perform dynamic range compression (drc) for a region (a first audio region and / or a second audio region) based on one or more audio region dynamic range factors 525C used as audio region factors 525 as the device (102, 140) moves from a first position to a second position. Dynamic range compression for a region may limit the sound in the audio region and may desirably increase the overall gain of the sound field when rendered.

[0143] As previously described, in one embodiment, the first audio region and the second audio region may be located in a layered sound field, wherein the first audio region is in a first layer of the sound field, and the first layer of the sound field is a foreground sound field. The second audio region may be in a second layer of the sound field, and is a background sound field. In one embodiment, the audio data in the second audio region is configured to be adjusted based on an environment reference distance, and the audio data in the first audio region is configured to be adjusted based on a foreground reference distance.

[0144] In the same or alternative embodiments, the first audio region is configured to transition from a first layer of a layered sound field to a second layer of the layered sound field. In the same or alternative embodiments, the first audio region is configured to transition from a first layer of a layered sound field to a second layer of the layered sound field, and the audio data in the first audio region is configured to be adjusted with an adjustment unit based on an ambient reference distance, and the audio data in the second audio region is configured to be adjusted with an adjustment unit based on an ambient reference distance.

[0145] In the same or alternative embodiments, the first audio region is configured to transition from a first layer of a layered sound field to a second layer of the layered sound field. In the same or alternative embodiments, the first audio region is configured to transition from a first layer of a layered sound field to a second layer of the layered sound field, and the audio data in the first audio region is configured to be adjusted with an adjustment unit based on a foreground reference distance, and the audio data in the second audio region is configured to be adjusted with an adjustment unit based on a foreground reference distance.

[0146] Figure 6 6 is a block diagram showing an embodiment of an audio zone control unit 620. The audio zone control unit 620 may include a mapping point associator unit 625, which may associate an audio source or a video object with a mapping point. The mapping point associator block unit 625 includes a first mapping point associator unit 625A, which may associate a mapping point to an ambient sound reference position. The ambient sound reference position includes the position or distance of the ambient sound reference. There may be several mapping points associated with the ambient sound reference and the sound reference position. For ambient sounds that may be diffuse in nature, the mapping points may be more dispersed. Since the mapping points may represent samples of the sound field near the ambient source and / or ambient sound reference position, the output of the mapping point associator unit 625A may take a subset of the mapping points and generate audio interest points. In one embodiment, in the case of fewer mapping points, the full set of mapping points around the ambient source and / or ambient sound reference position may generate interest points. Depending on the diffuseness and intensity of the ambient sound, the density of the mapping points and the interest points may be different, where the interest points are fewer than the mapping points. In some cases, there may be mapping points that are ambient sounds and / or ambient positions of audio interest points. The collection of audio interest points generated by associating the mapping points with the ambient sources and / or ambient sound reference positions can form an audio region. The output of the mapping point associator unit 625A can be an audio interest point, an audio background sound (which can be an ambient sound, or if the ambient source is not entirely within the audio region, the portion of the ambient sound within the audio region) (which will be considered as audio data in this example), an audio region (which can include height, width, and depth along with the audio data), and the position of the audio region relative to the person 40 or some other fixed point in space. The position of the audio region can be determined in a variety of ways, and one or more points in the audio region that can be used to determine the position can be selected in a variety of ways (e.g., one of a corner of the audio region, the center of the audio region, the centroid of the sound field, a weighted combination of sound intensity and diffusivity).

[0147] In the same or alternative embodiments, the mapping point associator block unit 625 includes a second mapping point associator unit 625B, which can associate mapping points to audio source positions. There may be several mapping points associated with audio source positions. Since mapping points can represent samples of sound fields near audio sources and / or audio source positions, the output of mapping point associator unit 625B can adopt a subset of mapping points and generate audio interest points. In one embodiment, when mapping points are less, the full set of mapping points around audio sources and / or audio source positions can generate interest points. Depending on the strength of the audio source, interest points may be more or less than mapping points. In some cases, there may be an audio source and / or source position where the mapping point is an audio interest point. The set of audio interest points generated by the association of mapping points with audio sources and / or audio positions can form an audio region. The output of the mapping point associator unit 625B may be an audio point of interest, an audio foreground sound (which may be a sound source, or a portion of the sound source within the audio region if the sound source is not completely within the audio region) (which would be considered audio data in this example), an audio region (which may include height, width, and depth along with the audio data), and the location of the audio region relative to the person 40 or some other fixed point in space. The location of the audio region may be determined in a variety of ways, and one or more points in the audio region that may be used to determine the location may be selected in a variety of ways (e.g., one of a corner of the audio region, the center of the audio region, the centroid of the sound field, a weighted combination of sound intensities).

[0148] In the same or alternative embodiments, the audio region control unit 620 may include a first mapping point video object associator 635A. The mapping point video object associator 635A may associate mapping points to video object foregrounds and video object foreground distances. Mapping points may be pixels. Although pixels may be evenly spaced within a video object, the mapping to pixels may be evenly spaced or unevenly spaced. The mapping points mapped to video objects in the audio region are not necessarily mappings to dominant sounds (i.e., foreground sounds) or ambient sounds (i.e., background sounds). Mapping points in the audio region may also be mapped to video object foreground distances. The foreground distance of a video object may include using the centroid of a video object, the centroid of a video object, or a fixed point. In one embodiment, a connection may exist between the mapping point video object associator 635A and the mapping point audio associator 625B. The connection may include a bus or signaling mechanism by which the association of a video object with an appropriate audio source is coordinated or synchronized.

[0149] In the same or alternative embodiments, the audio region control unit 620 may include a video object associator 635B for a second mapping point. The mapping point video object associator 635B may associate the mapping point to a video object background and a video object background distance. The mapping point may also be a pixel. Although the pixels may be evenly spaced within the video object, the mapping to the pixels may be evenly spaced or unevenly spaced. The mapping points in the audio region that are mapped to the video object may not necessarily be a mapping to a dominant sound or an ambient sound. The mapping points in the audio region may also be mapped to a video object background distance. The background distance of the video object may include using the center of mass of the video object, the centroid of the video object, or a fixed point.

[0150] In one embodiment, an association of a video object (if any) with audio data in a corresponding audio region may occur. For example, if the audio data in a first region is associated with a foreground video object, and if the video object is a foreground video object, the audio data may also be associated with diffuse sound or dominant sound. Depending on the relative distance of the sound (the sound of a dog 7A or a man 7D not speaking to a woman 7B) to the person 40 of the device 400, among other factors, the audio data may be determined to be diffuse sound (or ambient sound) or dominant sound.

[0151] In the same or alternative embodiments, there may be a connection between the mapping point video object associator 635B and the first mapping point audio associator 625A, or between another mapping point video object associator 635A (if present) and the second mapping point audio associator 625B. The connection may include a bus or signaling mechanism by which the association of the appropriate video object with the appropriate audio source is coordinated or synchronized.

[0152] The audio region control unit 620 may include an audio region of interest determiner 650. For clarity, in the drawings and throughout some examples of the present invention, the term region may include audio points of interest as well as the size of the audio region. As an example, FIG-6, FIG-7a and FIG-7b illustrate "region" as an input to the audio region of interest determiner. The term region is used instead of writing "spatial size of the region" and "points of interest within the region", respectively.

[0153] In one embodiment, the layered sound field updater 660 is integrated into one or more processors 402. The layered sound field updater 660 is configured to receive the updated movement, and wherein the updated distance includes a third distance or a fourth distance, or both the third distance and the fourth distance. The third distance can be the same as the first distance 62 between the audio zone and the first device position. The third distance can also be the distance between the video object in the audio zone and the first device position. In one embodiment, the third distance can be some combination of the audio source (dominant or ambient) position and the video object to the audio zone and the first device position.

[0154] In the same or alternative embodiments, the fourth distance of the layered sound field updater 660 can be the same as the second distance between the audio zone and the second device position. The fourth distance can also be the distance between the video object in the audio zone and the second device position. In one embodiment, the fourth distance can be some combination of the audio source (dominant or ambient) position and the video object and the audio zone and the second device position.

[0155] The audio region of interest determiner 650 may also optionally include an audio region scheduler 670. In one embodiment, the audio region scheduler 670 may only serially pass one audio region for tracking by the audio region tracker 680. In a different embodiment, the audio region scheduler 670 may pass multiple audio regions in parallel for tracking by the audio region tracker 680.

[0156] Figure 7a and Figure 7b Other embodiments of the audio zone control unit are shown. Figure 7a , a block diagram shows an embodiment of a layered sound field updater 760A in an audio region control unit 750A. The layered sound field updater 760A may update the layered sound field determiner 764A based on the updated distance. The output of the layered sound field determiner 764A may be sent to a video object updater 768A for the layered sound field. The updated distance may allow the layered sound field determiner 764A to determine when an audio region transitions from one layered sound field to another. For example, an audio region may have initially been in a background sound field, and then after the device moves from a first position to a second position, the audio region becomes in a foreground sound field. Similarly, the audio region may have initially been in a foreground sound field, and then after the device moves from a first position to a second position, the audio region becomes in a background sound field.

[0157] exist Figure 7a, an embodiment of the audio region tracker 780 includes a comparator 782 configured to compare the distance (or position) of the point of interest with the updated movement. The comparator 782 can be coupled to an adjuster 784, which is configured to adjust the panning and / or orientation of the sound field by the updated movement or updated distance of each audio region. The audio region control unit output of the audio region tracker 780 may include audio data based on the updated movement and adjusted to take into account the panning and / or orientation of the sound field based on the updated movement or the updated distance. FIG-7b is an alternative embodiment of FIG-7a, except that the layered sound field updater 760B does not use the updated distance for the audio region control unit 750B. Therefore, in the aforementioned embodiment of the audio region control unit 750A, the layered sound field updater 750A outputs the audio region that has taken into account the updated movement, and accordingly, the comparator is configured to operate on the audio region that has transitioned from one layered sound field to another (if any). In the audio region control unit 750B, the comparator 782 is configured to operate on an audio region based on a previously layered sound field before the audio region transitions from one layer of the sound field to another layer, or in the same layered sound field but at a different location. For each embodiment, the output of the audio region control unit should produce the same perceptible result for the person 40.

[0158] Figure 8 is a flowchart illustrating an example process that can be performed according to various aspects of the present invention. A device (102, 140) or a mobile device (e.g., a mobile phone) or a vehicle (where a vehicle can be considered as an AR, MR, and / or VR device that can be located at different device positions (i.e., a first device position and moved to a second device position)) may include one or more processors that can receive interactive commands to control audio data in a first audio area or a second audio area in a layered sound field 819. In addition, the one or more processors may generate one or more indicators indicating that an interactive command to control audio data in a first audio area or a second audio area in a layered sound field 821 has been received.

[0159] It should be appreciated that, depending on the example, certain actions or events of any technique described in the present invention may be performed in a different order, may be added, combined, or omitted entirely (e.g., not all described actions or events are required to implement the technique). In addition, in some examples, actions or events may be performed concurrently rather than sequentially, such as through multithreading, interrupt handling, or multiple processors.

[0160] In some examples, a VR device (or streaming device) can communicate using a network interface coupled to a memory of the VR / streaming device to exchange messages to an external device, wherein the exchanged messages are associated with representations of multiple available sound fields. In some embodiments, the VR device can receive wireless signals using an antenna coupled to the network interface, including data packets, audio packets, video packets, or transport protocol data associated with representations of multiple available sound fields. In some examples, one or more microphone arrays can capture the sound field.

[0161] In some examples, one or more of the multiple available sound field representations may include at least one high-resolution region and at least one lower-resolution region, and wherein the presentation selected based on the steering angle provides higher spatial accuracy relative to the at least one high-resolution region and less spatial accuracy relative to the lower-resolution region.

[0162] In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted through a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media such as data storage media, or communication media, including, for example, any media that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this manner, computer-readable media may generally correspond to (1) non-temporary tangible computer-readable storage media, or (2) communication media such as signals or carrier waves. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures to implement the techniques described in the present invention. A computer program product may include a computer-readable medium.

[0163] As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, flash memory or any other medium that can be used to store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to send instructions from a website, server or other remote source, the definition of the medium includes coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals or other temporary media, but are directed to non-temporary tangible storage media. The disks and optical disks used in the present invention include compact disks (CDs), laser optical disks, optical disks, digital versatile disks (DVDs), floppy disks and blue-ray disks, wherein the disks usually reproduce data magnetically, while the optical disks reproduce data by laser optical methods. The above combination should also be included in the scope of computer-readable media.

[0164] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used in the present invention may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described in the present invention. In addition, in some aspects, the functions described in the present invention may be provided in dedicated hardware and / or software modules configured for encoding and decoding or incorporated in a combined codec. Similarly, the technology may be fully implemented in one or more circuits or logic elements.

[0165] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit, or provided by a collection of interoperable hardware units, including one or more processors as described above in combination with applicable software and / or firmware.

[0166] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A device for controlling an audio zone, comprising: a memory configured to store audio data within first and second audio regions, each region being associated with one or more sound sources in a layered sound field, wherein the layered sound field includes a foreground sound field including a first audio region within a relative radial distance from a user of the device and a background sound field including a second audio region outside the foreground sound field; a display device configured to visualize the first audio region and the second audio region and one or more video objects for each audio region, each video object corresponding to one or more respective sound sources included in the first audio region or the second audio region; One or more processors, coupled to the memory and configured to: receiving an interactive command to control the audio data in the first audio region or the second audio region in the layered sound field, wherein the display device displays one or more indicators that the interactive command has been received; generating one or more indicators of receiving the interactive command to control the audio data in the first audio region or the second audio region of the layered sound field; as well as a sensor unit coupled to the one or more processors, wherein the sensor unit comprises a visual sensor configured to track movement of the user's head or body; wherein the one or more processors are configured to track the first audio region and one or more video objects associated with the first audio region and one or more sound sources corresponding to each video object, moving in proportion to and in the same direction as the movement of the device based on movement of the device from a first device position to a second device position detected by the sensor unit, wherein the second audio region and one or more video objects associated with the second audio region and one or more sound sources corresponding to each video object remain static relative to the second audio region prior to the movement of the device from the first device position to the second device position; an audio effects unit coupled to the one or more processors, wherein the audio effects unit comprises an adjustment unit configured to adjust the audio data within the first audio region by a first audio region factor and adjust the audio data within the second audio region by a second audio region factor based on movement of the device from the first device position to the second device position; and An audio rendering unit coupled to the one or more processors, wherein the audio rendering unit is configured to receive the output of the audio effects unit and render the audio data to the user based on the movement of the device from the first device position to the second device position. 2 . The apparatus of claim 1 , further comprising a display device configured to visualize two or more points of interest of the layered sound field.

3. The device according to claim 1, wherein: The controlling of the first audio region and the second audio region includes amplification of the audio data in the first audio region, or amplification of the audio data in the second audio region, or amplification of the audio data in the first audio region and the second audio region.

4. The apparatus of claim 3, further comprising a display device configured to visualize a representation of an amplification or attenuation of the audio data within the first audio region, or the second audio region, or both the first audio region and the second audio region.

5. The device according to claim 4, wherein: The visualization of the representation of an amplification or attenuation of the audio data in the first audio region, or the second audio region, or both the first audio region and the second audio region is based on a change in illumination of a screen of the display device.

6. The device according to claim 5, wherein: The illumination of the screen of the display device changes color, and different colors represent different amplification levels of the audio data in the first audio region, or the second audio region, or both the first audio region and the second audio region.

7. The device according to claim 5, wherein: The illumination of the screen of the display device changes color, and different colors represent different attenuation levels of the audio data in the first audio region, or the second audio region, or both the first audio region and the second audio region.

8. The device according to claim 1, wherein: The controlling of the first audio region and the second audio region includes attenuation of the audio data in the first audio region, or attenuation of the audio data in the second audio region, or attenuation of the audio data in the first audio region and the second audio region.

9. The device of claim 1 , further comprising a display device configured to visualize a representation of the first audio region, wherein The second audio region remains static relative to the second audio region before the device was moved from the first device position to the second device position.

10. The apparatus of claim 1, further comprising a display device configured to visualize an external object for the interactive command for controlling audio data in the layered sound field.

11. The device according to claim 10, wherein: The external object is a representation of at least a hand, a finger or a pointer.

12. A method for controlling an audio zone, comprising: storing audio data within first and second audio regions, each region being associated with one or more sound sources in a layered sound field, wherein the layered sound field includes a foreground sound field including a first audio region within a relative radial distance from a user of the device and a background sound field including a second audio region outside the foreground sound field; visualizing the first audio region and the second audio region and one or more video objects for each audio region, each video object corresponding to one or more respective sound sources included in the first audio region or the second audio region; receiving an interactive command to control audio data in a first audio region or a second audio region in a layered sound field; as well as generating one or more indicators that the interaction command is received to control the audio data in the first audio region or the second audio region in the layered sound field; tracking the first audio region and one or more video objects associated with the first audio region and one or more sound sources corresponding to each video object, based on movement of the device from a first device position to a second device position detected by a sensor unit, moving in proportion to and in the same direction as the movement of the device, wherein the second audio region and one or more video objects associated with the second audio region and one or more sound sources corresponding to each video object remain static relative to the second audio region prior to the movement of the device from the first device position to the second device position; adjusting the audio data within the first audio region by a first audio region factor and adjusting the audio data within the second audio region by a second audio region factor based on movement of the device from the first device position to the second device position; receiving an output of an audio effects unit; and The audio data is rendered to the user based on movement of the device from the first device location to the second device location.

13. The method of claim 12, further comprising visualizing two or more points of interest of the layered sound field. The method of claim 12 , further comprising visualizing the layered sound field.

15. The method of claim 12, further comprising controlling the first audio region and the second audio region based on amplification or attenuation of the audio data in the first audio region, the second audio region, or both the first audio region and the second audio region.

16. The method of claim 12, further comprising visualizing a representation of an amplification or attenuation of the audio data in the first audio region, the second audio region, or both the first audio region and the second audio region.

17. The method of claim 16, further comprising wherein the visualization of the representation of amplification or attenuation of the audio data in the first audio region, or the second audio region, or both the first audio region and the second audio region is based on a change in illumination of a screen of a display device.

18. The method according to claim 17, wherein: The illumination of the screen of the display device changes color, and different colors represent different amplification levels of the audio data in the first audio region, or the second audio region, or both the first audio region and the second audio region.

19. The method according to claim 18, wherein: The illumination of the screen of the display device changes color, and different colors represent different attenuation levels of the audio data in the first audio region, or the second audio region, or both the first audio region and the second audio region.

20. The method of claim 12, further comprising visualizing external objects for the interactive commands for controlling audio data in the layered sound field.

21. An apparatus for controlling an audio zone, comprising: Means for performing the method according to any of claims 12-20.

22. A non-transitory computer readable medium storing code comprising instructions executable by a processor of a device to perform the method according to any one of claims 12-20.

Images