Masked regions in metadata for spatial audio rendering

By using masking area parameters specified by metadata in the sound program, combined with beamforming technology of HOA format and spherical harmonic function, the problem of difficulty in realizing sound attenuation in specific scenes in the sound program in the prior art is solved, and the masking area sound is realized Effective hiding improves privacy and audience experience.

CN119071712APending Publication Date: 2024-12-03APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410692896.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-05-22
Filing Date
2024-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art is difficult to achieve sound attenuation or omission in a specific scene in a sound program, especially when privacy protection is required, such as coaches and players discussing sound adjustments in sports events.

Method used

By providing metadata on the content creation side, instructing the decoding side to attenuate or omit audio objects in the specified masking area during spatial audio rendering, using the 3D position and masking area parameters specified by the metadata, combining HOA format and spherical harmonic function beamforming technology to achieve attenuation of specific sounds.

Benefits of technology

Effectively hide specified audio objects during playback, ensuring that listeners cannot hear the sound in the masked area, enhancing the privacy control capabilities of content creators and improving the authenticity and privacy of the listener's experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071712A_ABST
    Figure CN119071712A_ABST
Patent Text Reader

Abstract

The present disclosure relates to masked regions in metadata for spatial audio rendering, the various aspects enabling a content creation side to control how a sound program is spatial audio rendered by a decoding side such that audio scene components in a three-dimensional acoustic masked region specified by the metadata are not heard, and the other audio scene component in the unmasked area of the sound program is heard by the playback audience. Other aspects are also described and claimed to be protected.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This non-provisional patent application claims the benefit of an earlier filing date of U.S. Provisional Application No. 63 / 506,037, filed on June 2, 2023. Technical Field

[0002] One aspect of the disclosure herein relates to spatial audio rendering of a sound program by a decoder-side process based on metadata associated with the sound program, where the metadata is provided by a process executed in a content creation side. Other aspects are also described. Background Art

[0003] The sound program may be produced as a live recording, such as a recording of a concert or sporting event (with or without accompanying video), or it may be pre-recorded or pre-authored, for example, as a voiceover of a segment of a video game, using a software application development platform. In all cases, the sound program may be adjusted using digital signal processing on the content creation side to suit the taste of the content creator (e.g., someone acting as an audio mixer). The adjusted sound program may then be encoded to reduce the bit rate and then transmitted to the listener's playback device, for example, via the Internet. At the playback device, or on the decoding side, the sound program is decoded and then presented as a speaker driver signal suitable for the listener's sound subsystem (e.g., headphones, surround sound speaker arrangement).

[0004] The sound program may be digitally processed by a spatial audio renderer so that the resulting speaker driver signals produce a listening experience in which the listener perceives the program to be closer to what they would hear in the scene being recorded or synthesized. The spatial audio renderer will enable the listener to, for example, perceive the sound of birds chirping as coming from a few meters to their right, and the rustling of leaves on the ground by another animal as coming from a few meters to their left, or the sound of wind blowing in the trees as surrounding them. Summary of the invention

[0005] A content creator may decide to craft a sound program so that it carries the content creator's desire for how a particular scene is heard. For example, the creator may wish to mute or silence the voice of a particular person in the scene being recorded so that, for example, sensitive remarks made by that person are not heard by the audience. This could be, for example, the sound of a coach and players huddled together discussing confidential game strategies during a sporting event. Various aspects of the disclosure herein enable the content creation side to instruct the decoding side via the metadata of the sound program to attenuate or even omit any discrete audio objects in the playback sound program that may be located in a masked region (e.g., a three-dimensional region or a three-dimensional acoustic region) specified in the metadata when the sound program is being spatially audio rendered for playback by the decoding side.

[0006] The above summary does not include an exhaustive list of all aspects of the present disclosure. It is contemplated that the present disclosure includes all systems and methods that can be practiced by all suitable combinations of the various aspects summarized above and disclosed in the detailed description below and specifically pointed out in the claims section. Such combinations may have advantages not specifically set forth in the above summary of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Several aspects of the disclosure herein are illustrated in the accompanying drawings by way of example and not by way of limitation, in which like reference numerals indicate similar elements. It should be noted that reference to "one" or "an" aspect in the disclosure is not necessarily the same aspect, and it means at least one. In addition, for the sake of brevity and to reduce the total number of drawings, a given drawing may be used to illustrate features of more than one aspect of the disclosure, and for a given aspect, not all elements in the drawing may be required.

[0008] Figure 1 is a block diagram of encoding-side and decoding-side processes for spatial audio rendering using metadata specifying masking regions.

[0009] Figure 2 An example of a masked area is illustrated.

[0010] Figure 3 An example of camera positions and orientations of multiple feeds inserted into metadata of a captured scene video is shown. DETAILED DESCRIPTION

[0011] refer to Figure 1 , which is a block diagram of an example digital processing performed in the encoding side and the decoding side for spatial audio rendering using metadata. In the encoding side, a sound program containing at least one audio scene component is obtained (e.g., from the content creation side), which audio scene component can be in any of several different formats. For example, it can be composed of one or more audio stems or one or more audio objects (including, for example, a group of two or more objects), where each object or stem is an audio signal, such as a pulse code modulated PCM audio signal. Alternatively, the sound program (or its audio scene component) can be in the form of channels (e.g., a 5.1 surround format or a 7.1.4 surround format), or in the form of a high-order ambisonics (HOA). The sound program can also be a mixed scene format, for example, where the sound field includes not only audio objects but also channels and audio scene components represented by HOA. Although the techniques described below are in the context of objects, those techniques are equally applicable when the sound program is or contains segments with a multi-channel format, and they are equally applicable to segments of the sound program in the HOA representation.

[0012] For example, a sound program may contain dialogue, such as the individual voices of one or more actors in a play or movie dubbing with a backbone of dialogue, music, and effects. A sound program may be a live recording (recorded in real time) of, for example, a sporting event, a concert, a live news report, etc. Alternatively, it may be music or an audiovisual work, such as a video game or movie, pre-recorded or pre-composed, such as using a software development kit. It may also be a combination of a live recording and a synthesized work, such as a mixed reality work.

[0013] For the purpose of reducing the bit rate, the sound programs (audio scene components) are encoded into a bitstream by the encoding process (encoding 103) and are associated with metadata. The metadata may contain spatial descriptors about certain audio scene components. A single audio scene component (ASC) may be in the form of an audio object (or having an object representation), an HOA representation, or channel-based audio. The spatial descriptor may indicate the position of the object as a virtual sound source relative to an origin or reference, which may represent a listening position or a screen or display on which a video associated with the sound program is simultaneously presented. The position of the object may be given, for example, in polar coordinates comprising its distance from the origin, its azimuth, and its elevation. The metadata may also be encoded to reduce the bit rate. The metadata may be provided to the decoding side via a separate communication channel, or it may be incorporated into the bitstream together with the sound program, such as Figure 1 as shown in the example.

[0014] On the decoding side, the encoded sound program and metadata are provided to or obtained by the playback device, for example, in the form of a bitstream over the Internet. The playback device can be, for example, a digital media player in a console, a smart phone, a tablet computer, etc. One or more decoding side processes are executed by a programmed processor in the playback device (configurable according to instructions stored in a non-transitory machine-readable medium such as a solid-state memory). One of the decoding side processes (decoding 104) is used to undo the encoding to recover the ASCs that constitute the sound program and extract the metadata containing the masked areas.

[0015] The decoding side also includes a spatial audio renderer 101, which performs a spatial audio rendering process that produces speaker driver signals 109 suitable for the listener's sound subsystem, which can be depicted in the figure here with speaker symbols. The speaker driver signals 109 can be binaural headphone signals, or they can be speaker driver signals for a specific type of surround sound subsystem. The spatial audio renderer achieves this by converting the ASCs that make up the sound program into speaker driver signals using any suitable spatial audio rendering algorithm. The conversion can be performed by first converting the decoded audio objects into a high-order ambisonics HOA format and then converting the HOA format into speaker driver signals. Alternatively, the decoded audio objects can be directly converted into the channel format of the sound subsystem. The speaker driver signals will then drive the sound subsystem, which reproduces the sound program.

[0016] In one aspect of the disclosure herein, metadata specifies a three-dimensional masking region of a sound program and instructs a process in the decoding side (spatial audio renderer 101) to attenuate the sound program whenever an ASC is present in the masking region (while the process is spatially audio rendering the sound program for playback). This results in a user or listener of the sound subsystem being unable to hear any ASCs in the masking region during a playback session, while the listener can hear other ASCs in the unmasked region of the sound program during the playback session. In depicting an example playback in which the sound program is a live recording of a sporting event or a dubbing of a live broadcast, Figure 2 In the example shown in FIG. 2 , the content creator has chosen a masking zone limited to the following: the coach (element 202) and the referee or umpire (element 204) are face to face in an argument; and foul language is being used, which some viewers / listeners of the audio program may be sensitive to.

[0017] In one aspect, the decoding side process compares the 3D position (within the sound scene) of one or more ASCs (e.g., audio objects) with the masking regions indicated in the metadata. The position of the audio objects will be found in the metadata. Figure 2 In the example of , the result may be that one or two audio objects representing the voices of the coach and referee will be found to be within the masked region, so they will be sufficiently attenuated or simply omitted so as not to be heard during playback. At the same time, the voices of the crowd or the voices of the competitors visible in the same scene are outside the masked region (or are in the unmasked region).

[0018] If the sound program is created in the HOA format or has segments in the HOA format or has an HOA representation, then based on the masking areas specified in the metadata, digital signal processing-based sound field processing techniques are performed on the decoding side, which modify the HOA representation by attenuating (e.g., eliminating) the masking areas specified by the metadata of the sound scene (while not attenuating nearby unmasked areas). This can be achieved by using beamforming techniques of spherical harmonics to attenuate (e.g., eliminate) the acoustic energy of the areas specified by the masking areas. Figure 2 In the example of , the result should be that the HOA-based sound field processing technology only attenuates the sounds of the referee and coach enough, and makes these sounds inaudible to the audience.

[0019] If the sound program is created for example in a multi-channel surround format or has segments in a multi-channel surround format, a sound field analysis is performed by the decoding side to attenuate or even eliminate specified masking areas, for example by first converting the multi-channel surround format to HOA.

[0020] In one example, the variable AudioSceneMaskingZone in the metadata (see Figure 1 , masking regions) instructs the decoder-side process to attenuate only those ASCs (e.g., channels, objects, HOA) that are in one or more masking regions as specified in the metadata.

[0021] In one aspect, a dictionary or codebook of predefined zones is created (and shared with the decoding side) so that the content creator only needs to select one or more of the predefined zones. For each possible predefined zone, there is an entry in the dictionary containing the parameters or description of the zone. The metadata may include a switch or flag that specifies this, for example, if usePreDefinedZones==1. In this case, the masked zone is specified in the metadata only by the index of its entry in the dictionary. The renderer on the decoding side has access to the dictionary, and in this case will perform a table lookup in the dictionary to obtain the parameters or actual description of the selected masked zone. It then uses the parameters or description to determine which one or more ASCs are in the masked zone, and then only attenuates or omits those ASCs.

[0022] On the other hand, the content creator may specify any masking regions as a series of geometric coordinates in 3D space, such as a series of Cartesian coordinates in 3D space or a series of polar coordinates in 3D space, by inserting the description of the masking regions in the metadata directly into the metadata.

[0023] In yet another aspect, the metadata may specify that the spatial audio renderer 101 must implement a transition zone to avoid a sudden gain change between a rendered ASC in a masked area and a rendered ASC not in the masked area. The length of the transition zone (e.g., as a distance length) may be sent in the metadata, or it may be stored in the spatial audio renderer 101, for example, as a default value. In one aspect, the transition zone is adjacent to the masked area and between the masked area and the unmasked area, and the process in the decoding side performs a gain transition when rendering the sound program by: applying a low gain or attenuation (e.g., maximum attenuation to eliminate all sound sources located in the masked area); applying a gradual change in gain (e.g., an intermediate gain) to the transition zone, wherein the gradual change in gain (or intermediate gain) is greater than the low gain or attenuation; and applying a high gain to the unmasked area, wherein the high gain is greater than the gradual change in gain (e.g., no attenuation). Alternatively, the metadata may specify that the fade-in of the original ASC gain starts outside the masked area and the fade-in ends within the masked area.

[0024] Camera Position

[0025] Now go to Figure 3 , which illustrates an example of multiple cameras targeting a scene such as a live sporting event, where the camera positions and orientations are inserted into the metadata of several captured video feeds of the scene. A single video feed, referred to here as the reference feed, is encoded and sent along with its metadata, which includes the audio objects / HOAs and their corresponding positions relative to the reference camera. For all other video feeds, the same audio session is used, but only the relative positions of the cameras of that feed relative to the reference feed camera are sent. On the decoding side, during playback, the spatial audio renderer will use the repositioned objects / HOAs for the final rendering. The metadata also includes the camera position and angles or orientation, for example in the form of quaternions. This is in contrast to requiring a new audio session for each camera, even if the position of the objects remains the same.

[0026] The various aspects described herein (masking area, camera position) may be embodied at least in part in software. That is, the technical method operations described above and described below in the claims may be performed in an audio processing system in response to or through its processor executing instructions contained in a storage medium such as a non-transitory machine-readable storage medium (e.g., dynamic random access memory, static memory, non-volatile memory). Note that the phrase "processor" is generally used here to refer to one or more processors that may be in separate housings or devices and that may communicate with each other (e.g., in effect forming a distributed computing system). Moreover, in various aspects, hard-wired circuits may be used in combination with software instructions to implement the techniques described herein. Therefore, these techniques are not limited to any specified combination of hardware circuits and software, nor to any source of instructions executed by the audio processing system.

[0027] In this specification, certain terms are used to describe the features of various aspects. For example, in some cases, the terms "module", "processor", "unit", "renderer", "system", "device", "filter", "engine", "block", "detector", "simulation", "model" and "component" represent hardware and / or software configured to perform one or more processes or functions. For example, examples of "hardware" include, but are not limited to, integrated circuits such as processors (e.g., digital signal processors, microprocessors, application specific integrated circuits, microcontrollers, etc.). Therefore, as understood by those skilled in the art, different combinations of hardware and / or software can be implemented to perform the processes or functions described by the above terms. Of course, the hardware can alternatively be implemented as a finite state machine or even a combinational logic component. Examples of "software" include executable code in the form of applications, applets, routines, and even a series of instructions. As described above, the software can be stored in any type of machine-readable medium.

[0028] Certain portions of the foregoing detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits in a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the art of audio processing to most effectively convey the substance of their work to others skilled in the art. An algorithm is herein and is contemplated to refer to a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulation of physical quantities. However, it should be kept in mind that all of these and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specifically stated, it will be apparent from the foregoing discussion that it will be understood that throughout this specification, discussions using terms such as those given in the claims below refer to actions and processes of an audio processing system or similar electronic device that manipulates data represented as physical (electronic) quantities in the system's registers and memories and converts them into other data similarly represented as physical quantities in the system's memories or registers or other such information storage, transmission or display devices.

[0029] The processes and blocks described herein are not limited to the specific examples described, and are not limited to the specific order used as examples in this article. On the contrary, any processing box can be reordered, combined or removed, executed in parallel or serially as needed to achieve the above results. The processing blocks associated with the implementation of the audio processing system can be executed by one or more programmable processors to execute one or more computer programs stored on a non-transient computer-readable storage medium to perform the functions of the system. All or part of the audio processing system can be implemented as a dedicated logic circuit (e.g., FPGA (field programmable gate array) and / or ASIC (application-specific integrated circuit)). All or part of the audio system can be implemented using an electronic hardware circuit including at least one of an electronic device such as a processor, a memory, a programmable logic device or a logic gate. In addition, the process can be implemented in any combination of hardware devices and software components.

[0030] In some aspects, the present disclosure may include language such as “at least one of [element A] and [element B]”. The language may refer to one or more of these elements. For example, “at least one of A and B” may refer to “A”, “B”, or “A and B”. Specifically, “at least one of A and B” may refer to “at least one of A and at least one of B” or “at least either A or B”. In some aspects, the present disclosure may include language such as “[element A], [element B], and / or [element C]”. The language may refer to any one of these elements or any combination thereof. For example, “A, B, and / or C” may refer to “A”, “B”, “C”, “A and B”, “A and C”, “B and C”, or “A, B, and C”.

[0031] While certain aspects have been described and shown in the drawings, it is to be understood that such aspects are illustrative only and not restrictive, and that the disclosure is not limited to the particular construction and arrangements shown and described, since various other modifications may occur to those skilled in the art.

[0032] To assist the Patent Office and any reader of any patent issuing in this application in interpreting the appended claims, applicants wish to note that they do not intend to invoke 35 U.S.C. 112(f) with respect to any of the appended claims or claim elements unless the words “means for” or “step for” are expressly used in a claim.

[0033] It is understood that the use of personally identifiable information should be subject to privacy policies and practices that are deemed to meet or exceed industry or government requirements for maintaining user privacy. Personally identifiable information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use, and the nature of the authorized use should be made clear to users.

Claims

1. An encoding-side method for spatial audio rendering using metadata, the method comprising: Encoding a sound program into a bitstream; as well as Metadata of the sound program is provided, wherein the metadata specifies a masking area and indicates whether a process in a decoding side attenuates the sound program when the process performs spatial audio rendering on the sound program for playback, so that one or more audio scene components of the sound program located in the masking area are not heard, while one or more other audio scene components of the sound program located outside the masking area or in an unmasked area are heard by a listener of the playback.

2. A method according to claim 1, wherein the metadata specifies the masked area as an index to a dictionary of predefined geometric three-dimensional areas, wherein the decoding side uses the index to perform a lookup on the dictionary to obtain one of the predefined geometric three-dimensional areas.

3. The method according to claim 2, wherein each of the predefined geometric three-dimensional regions comprises a series of Cartesian coordinates in 3D space or a series of polar coordinates in 3D space.

4. A method according to any one of the preceding claims, wherein the metadata further specifies a transition region adjacent to the masked region and between the masked region and an unmasked region, and the process in the decoding side performs gain transition when rendering the sound program by applying: The low gain of the masked area, a gradual change in gain in the transition region, wherein the gradual change in gain is greater than the low gain, and A high gain in the unmasked region, wherein the high gain is greater than the gradual change in gain. The method of claim 4 , wherein the metadata specifies a distance length of the transition region.

6. The method of claim 1, wherein the metadata further specifies that a fade-in of the original audio scene gain begins outside the masking region and that the fade-in ends within the masking region.

7. The method of claim 1, wherein the metadata specifies the masking region directly as a geometric three-dimensional region comprising a series of Cartesian coordinates in 3D space or a series of polar coordinates in 3D space.

8. The method according to any one of the preceding claims, wherein the one or more audio scene components of the sound program and the one or more other audio scene components of the sound program adopt or have i) a channel-based representation, ii) an object representation, or iii) a Higher Order Ambisonics (HOA) representation.

9. A method for spatial audio rendering using metadata, performed in a decoding side, the method comprising: Decoding a sound program from a bitstream; as well as Receive metadata for the sound program, wherein the metadata specifies a masking region and indicates whether a process attenuates the sound program when the process performs spatial audio rendering of the sound program for playback, so that one or more audio scene components of the sound program located in the masking region are not heard, while one or more other audio scene components of the sound program located outside the masking region or in an unmasked region are heard by a listener of the playback.

10. The method according to claim 9, wherein the one or more audio scene components of the sound program located in the masked area are represented using a higher-order Ambisonics representation (HOA), the method further comprising: In response to the metadata, the HOA representation is processed using a sound field processing technique that modifies the HOA representation by attenuating acoustic energy in the masked region employing the HOA representation.

11. The method of claim 9, wherein the one or more audio scene components of the sound program located in the masked region are audio objects, the method further comprising: In response to the metadata, determining whether the audio object [having an associated 3D or 2D position given for it in the metadata] is in the masked region, and omitting the audio object from the playback or attenuating the audio object so that the audio object can no longer be heard by the listener during the playback.

12. The method of claim 9, wherein the one or more audio scene components of the sound program located in the masked region are channel-based audio representations, the method further comprising In response to the metadata, the channel-based audio representation is processed using a sound field processing technique that modifies the channel-based audio representation by attenuating acoustic energy in the masked regions.

13. The method of any one of claims 9-12, wherein the metadata specifies a country or product market for the masked area.

14. A method according to any one of claims 9-13, wherein the metadata specifies the masked area as an index to a dictionary of predefined geometric three-dimensional areas, and wherein the decoding side uses the index to perform a lookup on the dictionary to obtain one of the predefined geometric three-dimensional areas as the masked area.

15. The method of claim 14, wherein each of the predefined geometric three-dimensional regions comprises a series of Cartesian coordinates in 3D space or a series of polar coordinates in 3D space.

16. The method according to any one of claims 9 to 15, wherein the metadata further specifies a transition region adjacent to the masked region and between the masked region and an unmasked region, and the process in the decoding side performs gain transition when rendering the sound program by applying: The low gain of the masked area, an intermediate gain in the transition region, wherein the intermediate gain is greater than the low gain, and A high gain in the unmasked region, wherein the high gain is greater than the intermediate gain. The method of claim 16 , wherein the metadata specifies a distance length of the transition region.

18. The method according to any one of claims 9-15, wherein the metadata further specifies that a fade-in of an original audio scene component gain starts outside the masking region and that the fade-in ends within the masking region.

19. The method of any one of claims 9-18, wherein the metadata specifies the masking region directly as a geometric three-dimensional region comprising a series of Cartesian coordinates in 3D space or a series of polar coordinates in 3D space.