Audio rendering method and system, and electronic device
By introducing distance encoding filters and spherical harmonics into Ambisonics audio rendering technology, combining low-frequency processing and phase adjustment, the problem of neglecting near-field effect of sound sources in traditional technology is solved, and a more accurate sense of distance and realistic 3D sound field is achieved. Render.
Patent Information
- Application Number
- PCT/CN2024/097045
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-01
- Filing Date
- 2024-06-03
- Publication Date
- 2024-12-05
AI Technical Summary
Traditional Ambisonics audio rendering technology ignores the near-field effect of sound sources in virtual reality (VR) and augmented reality (AR) scenarios, resulting in unreal distance and unable to effectively render the sound field in 3D space, especially in the distance between sound sources. Recent time.
The distance encoding filter and low-frequency processing are introduced, the low-frequency amplitude of the audio signal is increased to render the distance sense more accurately, and the phase information of the sound wave is reasonably adjusted, combined with spherical harmonic function is used to encode and decode, taking into account the distance and direction characteristics of the sound source. Improve the Ambisonics codec solution.
Improves the distance accuracy and fidelity of audio rendering, and can better render 3D sound field in virtual reality and augmented reality scenes, providing a more realistic listening experience.
Smart Images

Figure CN2024097045_05122024_PF_FP_ABST
Abstract
Description
Audio rendering method, system and electronic device
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims the benefit of international patent application number PCT / CN2023 / 097866, filed on June 1, 2023, which is incorporated herein by reference. Technical Field
[0003] The present disclosure relates to the technical field of audio signal processing, and in particular to an audio rendering method, system, electronic device, and non-transitory computer-readable storage medium. Background Art
[0004] Audio rendering refers to the appropriate processing and playback of sound signals from a sound source to provide users with a desired listening experience, particularly an immersive experience, in their specific application scenarios. Ambisonics is an audio playback technology that uses a limited number of channels to record, render, and restore the characteristics of spatial sound, providing listeners with a more realistic sense of presence.
[0005] Summary of the Invention
[0006] According to some embodiments of the present disclosure, an audio signal encoding device for audio rendering is provided, which is configured to: obtain metadata-related information associated with an audio signal, the metadata-related information including spatial position information related to the audio signal; and spatially encode the audio signal based on the metadata-related information associated with the audio signal to obtain an encoded audio signal.
[0007] According to some embodiments of the present disclosure, a method for encoding an audio signal for audio rendering is provided, comprising: obtaining metadata-related information associated with an audio signal, the metadata-related information including spatial position information related to the audio signal; and spatially encoding the audio signal based on the metadata-related information associated with the audio signal to obtain an encoded audio signal.
[0008] According to some embodiments of the present disclosure, an audio signal decoding device for audio rendering is provided, configured to: obtain an encoded audio signal, wherein the encoded audio signal includes an encoded audio signal obtained by an audio signal encoding device according to an embodiment of the present disclosure; and spatially decode the encoded audio signal to obtain a decoded audio signal for audio rendering.
[0009] According to some embodiments of the present disclosure, an audio decoding method for audio rendering is provided, comprising: obtaining an encoded audio signal, wherein the encoded audio signal comprises an encoded audio signal obtained by an audio signal encoding device according to an embodiment of the present disclosure; and spatially decoding the encoded audio signal to obtain a decoded audio signal for audio rendering.
[0010] According to some embodiments of the present disclosure, an audio rendering system / audio rendering device is provided, including: an audio signal encoding device according to an embodiment of the present disclosure, and / or an audio signal decoding device according to an embodiment of the present disclosure.
[0011] According to some embodiments of the present disclosure, an audio rendering method is provided, including: the audio signal encoding method according to some embodiments of the present disclosure, and / or the audio signal decoding method according to some embodiments of the present disclosure.
[0012] According to some further embodiments of the present disclosure, a chip is provided, comprising: at least one processor and an interface, wherein the interface is configured to provide computer-executable instructions to the at least one processor, and the at least one processor is configured to execute the computer-executable instructions to implement the method of any embodiment described in the present disclosure.
[0013] According to some further embodiments of the present disclosure, a computer program is provided, comprising: instructions / program codes, which, when executed by a processor, cause the processor to perform the method of any one of the embodiments described in the present disclosure.
[0014] According to some further embodiments of the present disclosure, an electronic device is provided, including: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the method of any embodiment described in the present disclosure based on instructions stored in the memory device.
[0015] According to some further embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method of any embodiment described in the present disclosure is implemented.
[0016] According to still further embodiments of the present disclosure, a computer program product is provided, comprising instructions, which, when executed by a processor, implement the method of any embodiment described in the present disclosure.
[0017] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of this application. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:
[0019] FIG1A shows an exemplary schematic diagram of some embodiments of an audio signal processing process;
[0020] FIG1B shows an exemplary block diagram of an audio rendering system according to some embodiments of the present disclosure;
[0021] FIG1C shows an exemplary flow chart of an audio rendering method according to some embodiments of the present disclosure;
[0022] FIG2 shows an exemplary schematic diagram of a core processing framework in an audio rendering solution according to some embodiments of the present disclosure;
[0023] FIG3A shows an exemplary block diagram of an audio encoding device according to some embodiments of the present disclosure;
[0024] 3B and 3C illustrate exemplary implementations of audio encoding according to some embodiments of the present disclosure;
[0025] 3D and 3E show exemplary flow charts of audio encoding methods according to some embodiments of the present disclosure;
[0026] FIG4A shows an exemplary block diagram of an audio decoding device according to some embodiments of the present disclosure;
[0027] FIG4B shows an exemplary flow chart of an audio decoding method according to some embodiments of the present disclosure;
[0028] FIG5 shows a block diagram of some embodiments of an electronic device of the present disclosure;
[0029] FIG6 shows a block diagram of another embodiment of the electronic device of the present disclosure;
[0030] FIG7 shows a block diagram of some embodiments of a chip of the present disclosure.
[0031] It should be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not necessarily drawn to scale. The same or similar reference numerals are used throughout the drawings to indicate the same or similar parts. Therefore, once an item is defined in one drawing, it may not be discussed further in subsequent drawings. DETAILED DESCRIPTION
[0032] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.
[0033] Unless otherwise specified, the relative arrangement of components and steps, numerical expressions, and numerical values described in these embodiments do not limit the scope of this disclosure. Techniques, methods, and devices known to those skilled in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of this disclosure. In all examples shown and discussed herein, any specific values should be interpreted as merely illustrative and not limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0034] It should be understood that the various steps described in the method embodiments of the present disclosure can be performed in different orders and / or performed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values of the parts and steps set forth in these embodiments should be interpreted as being merely exemplary and do not limit the scope of the present disclosure.
[0035] As used in this disclosure, the term "include" and its variations are intended to be open-ended terms that include at least the elements / features listed therein, but do not exclude other elements / features, i.e., "including but not limited to." Furthermore, the term "comprise" and its variations are intended to be open-ended terms that include at least the elements / features listed therein, but do not exclude other elements / features, i.e., "including but not limited to." Therefore, "include" and "include" are synonymous. The term "based on" means "based, at least in part, on."
[0036] Reference throughout this specification to "one embodiment," "some embodiments," or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. For example, the term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Furthermore, the appearances of the phrases "in one embodiment," "in some embodiments," or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment, but may.
[0037] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules, or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules, or units. Unless otherwise specified, concepts such as "first" and "second" are not intended to imply that the objects described in such a manner must be in a given order in time, space, ranking, or any other manner.
[0038] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".
[0039] Figure 1A illustrates some conceptual diagrams of audio signal processing, particularly the process / system from acquisition to rendering. As shown in Figure 1A , in this system, audio signals undergo audio processing or production after acquisition. The processed / produced audio signals are then distributed to a rendering end for rendering. The audio rendering / playback end then processes the audio signals for playback / presentation to the user. Specifically, the audio signals are rendered and presented to the user with the desired effect, satisfying the user experience. It should be noted that this audio signal processing process is applicable to various application scenarios, particularly virtual reality audio content presentation.
[0040] Currently, there are many audio application scenarios, such as audio rendering scenarios and XR virtual scenes, which can refer to highly realistic virtual scenes created through at least one of virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies. Ambisonics is particularly promising as an audio transmission method in XR scenarios. Ambisonics technology is commonly used in audio rendering and playback. Ambisonics corresponds to a mathematical representation of a three-dimensional sound field. Unlike traditional channel-based and object-based transmission methods, Ambisonics supports arbitrary head rotation, making it ideal for representing three-dimensional rotatable sound fields. It is also a standard mathematical codec solution with the advantages of low theoretical error and orthogonal basis sets.
[0041] Traditional Ambisonics approaches to virtual source imaging or sound coding primarily focus on coding based on the far-field assumption, while ignoring the near-field effects of finite-distance sources. In other words, the sound sources are assumed to be located far from the head, thus focusing only on the two spatial dimensions related to direction. The core logic of this approach is to assume that the sound source is far enough away to ignore the changes in auditory perception caused by changes in distance. Therefore, the distance of the sound image does not change the auditory perception. This approach is applicable to sound scenes such as traditional movie theaters, because they all have relatively distant sound image positions, so ignoring the impact of changes in the distance of the sound image on the auditory perception will not be too obvious. However, for XR device scenarios, since many sound images are very close to the human ear and even have changes in distance, it is necessary to pay attention to changes in the distance of the sound image.
[0042] In some traditional implementations and operations, the sense of distance is increased by leveraging low-frequency characteristics. Some strategies aim to increase low frequencies to make the human ear perceive sounds as closer, and this approach has some basis. The main reason is that high-frequency sounds are attenuated more than low-frequency sounds. Therefore, when the sound source is farther away, high-frequency attenuation is generally more severe. In other words, adding more low frequencies will make the listener perceive the distance as closer. In some practical applications, the sense of distance is increased by leveraging reverberation. Some solutions enhance the sense of distance by increasing the reverberation of sound. This approach primarily exploits the psychoacoustic property that the perception of reverberation creates a sense of spaciousness, thereby creating a sense of distance in the listener. This approach primarily utilizes a psychoacoustic dislocation and does not truly render distance.
[0043] In view of this, the present disclosure proposes an improved audio processing, in particular an improved audio codec, which enables the three dimensions (such as (x, y, z), As well as other three-dimensional or N-dimensional encodings known or applicable in the art, this can be achieved, increasing the sense of distance in rendering and, additionally or alternatively, restoring the sound field over a wider range. In the context of this disclosure, codecs generally refer to the process of converting ordinary audio signals (e.g., ordinary time-domain signals) into and from ambisonic signals, and can be referred to as ambisonic codecs, as opposed to traditional codecs performed for transmitting compressed audio.
[0044] In particular, the present disclosure proposes introducing near-field compensation during the encoding stage, such as distance-based encoding, including but not limited to designing a distance coding filter. This can obtain a feasible Ambisonic format, thereby making the Ambisonics sound field carry more complete information and providing listeners with a more accurate listening experience.
[0045] In addition, the present disclosure also proposes improved low-frequency processing, in which the low-frequency increase amplitude is precisely calculated and is more accurate, so that the audience perceives the distance more accurately.
[0046] In addition, the present disclosure also proposes reasonable adjustment of the phase, which reasonably presents the time delay problem of sound wave transmission, thereby bringing a more realistic effect.
[0047] The following describes an exemplary implementation of an embodiment of the present disclosure using an Ambisonics audio signal as an example. It should be noted that the concepts and solutions of the present disclosure can also be appropriately applied to other types of audio signals. Audio rendering according to an embodiment of the present disclosure will be described in detail with reference to the accompanying drawings.
[0048] FIG1B shows a block diagram of some embodiments of an audio rendering system according to embodiments of the present disclosure. The audio rendering system 1 includes an audio signal acquisition module 11 configured to acquire an input audio signal, an audio signal encoder 12 configured to encode the input audio signal into an audio signal in a specific spatial format; and an audio signal decoder 13 configured to spatially decode the encoded audio signal in the specific spatial format to obtain a decoded audio signal for audio rendering, thereby enabling audio to be presented / played back to a user based on the spatially decoded audio signal.
[0049] In some embodiments, the audio signal in a specific spatial format may be in any appropriate format, as long as it can be used for playback in a user playback environment. In some embodiments, the input audio may be appropriately processed, such as encoding, format conversion, etc., to obtain an audio signal in a specific spatial format. In some embodiments, when the input audio signal can derive audio signals in two or more audio content formats, the obtained audio signal in a specific spatial format has a common format, that is, audio signals in various audio content formats all obtain audio signals in the same format for playback. As an example, the audio signal in the specific spatial format is an Ambisonics audio signal, or the audio signal in the specific spatial format is any one of FOA (First Order Ambisonics), HOA (Higher Order Ambisonics), and MOA (Mixed-order Ambisonics).
[0050] According to embodiments of the present disclosure, the input audio signal may be in a variety of formats, and an audio signal in a specific spatial format may be appropriately obtained based on the format of the input audio signal. In some embodiments, the input audio signal may be in a distributed spatial audio exchange format, which may be obtained from various captured audio content formats. Spatial audio processing, particularly encoding, is then performed on such input audio signal to obtain an audio signal in the specific spatial format. In particular, in some embodiments, the spatial audio processing may include appropriate processing of the input audio, particularly including parsing, format conversion, information processing, encoding, etc., to obtain an audio signal in the specific spatial format. In other embodiments, the audio signal in the specific spatial format may be directly obtained from the input audio signal without performing at least some of the spatial audio processing. In some embodiments, the input audio signal may contain or directly be a signal in a specific audio content format, such as a specific audio representation signal, so that encoding can be performed directly on the audio signal to obtain an audio signal in the specific spatial format. In still other embodiments, the input audio signal may also include an audio signal directly in the specific spatial format. Such an audio signal is directly transmitted / transparently transmitted to the decoding side without passing through an intermediate processing stage.
[0051] In some embodiments, the input audio signal may include an original input signal, such as an input signal in different formats, which is applied to different application scenarios. In some embodiments, a pre-processing module may be included. Taking into account different application scenarios and input formats, the pre-processing module can be used to be compatible with inputs of different formats. In some embodiments, the pre-processing module is used to obtain an audio signal having an audio content format compatible with an audio rendering system and at least one of the metadata information associated with the audio signal from the input audio signal, so that the audio signal of the specific spatial format can be obtained based on the audio signal having an audio content format compatible with the audio rendering system. The pre-processing module may be included in the audio signal acquisition module 11, as shown in the pre-processing module 111 in Figure 1. In other embodiments, the input signal may directly be an audio signal having an audio content format compatible with an audio rendering system. In this case, the pre-processing operation can be processed by other devices in the system, and the pre-processing module may be located outside the audio signal acquisition module of the present disclosure, or even outside the audio processing system. In some embodiments, the audio signal acquisition module can obtain an audio signal in a specific audio content format and metadata information associated with the audio signal by parsing the input signal, or directly receive an audio signal in a specific audio content format and metadata information associated with the audio signal.
[0052] According to an embodiment of the present disclosure, audio signal encoding can be performed based on metadata-related information, which is associated with the input audio signal and can be obtained together with the audio signal, or can be extracted from the input audio signal. As an example, the metadata-related information can be obtained by an acquisition module, or obtained by other devices / modules and provided to the audio signal encoder. In some embodiments, the metadata-related information can be in various forms and can include various information, such as spatial properties, channel properties, and so on. The metadata-related information can generally depend on the signal type of the audio signal, and in particular can correspond to the signal representation method of the signal. For example, for object-based signal representation, the metadata information can be related to the properties of the audio object, especially the spatial properties; for scene-based signal representation, the metadata information can be related to the properties of the scene; for channel-based signal representation, the metadata information can be related to the properties of the channel. As an example, the spatial position information of the sound source can be included in the metadata-related information.
[0053] In some embodiments, the acquisition module 11 may further include an audio information processing module 112, which is configured to extract audio parameters of an audio signal in a specific audio content format based on metadata associated with the audio signal in the specific audio content format, so that the audio signal encoding module may be further configured to spatially encode the audio signal in the specific audio content format based on the metadata associated with the audio signal and at least one of the audio parameters. As an example, the audio information processing module may be referred to as a scene information processor, which may provide the audio parameters extracted based on the metadata to the audio signal encoding module for encoding. The audio information processing module is not required for the audio rendering of the present disclosure, for example, its information processing function may not be performed, or it may be outside the audio rendering system, or the audio information processing module may be included in other modules, such as an audio signal acquisition module or an audio signal encoding module, or its function may be implemented by other modules, and is therefore indicated by a dotted line in the accompanying drawings.
[0054] In some embodiments, additionally or optionally, the audio rendering system may include a signal adjustment module, which is configured to perform signal processing on the decoded audio signal. The signal processing performed by the signal adjustment module can be referred to as a signal post-processing, especially the post-processing performed on the decoded audio signal before being played back by the playback device. Therefore, the signal adjustment module may also be referred to as a signal post-processing module. In particular, the signal adjustment module may be configured to adjust the decoded audio signal based on the characteristics of the playback device in the user application scenario, so that the adjusted audio signal can present a more appropriate acoustic experience when rendered by the audio rendering device. It should be noted that the audio signal adjustment module is not required for the audio rendering of the present disclosure, for example, the signal adjustment function may not be performed, or it may be outside the audio rendering system, or the audio signal adjustment module may be included in other modules, such as in an audio signal decoding module or its function is implemented by a decoding module, and is therefore indicated by a dotted line in the accompanying drawings.
[0055] Additionally, the audio rendering system 1 may further include or be connected to an audio input port for receiving an input audio signal, which may be distributed and transmitted to the audio rendering system in the audio system, as described above, or directly input by the user at the user end or consumer end, as will be described later. Additionally, the audio rendering system 1 may further include or be connected to an output device, such as an audio rendering device or an audio playback device, which may present the spatially decoded audio signal to the user. According to some embodiments of the present disclosure, the audio rendering device or audio playback device according to an embodiment of the present disclosure may be any appropriate audio device, such as a speaker, a speaker array, headphones, and any other appropriate device capable of presenting an audio signal to the user.
[0056] Figure 1C shows a flowchart of some embodiments of the audio rendering method according to the present disclosure. The various steps / processing in the method can be performed as described above, for example, as the corresponding modules are performed. Although not shown, the audio rendering method according to the present disclosure may also include other steps to implement the processing / operations in the pre-processing, audio information processing, audio signal spatial coding, etc. described above, which will not be described in detail here. It should be noted that the audio rendering method according to the present disclosure and the steps therein can be performed by any appropriate device, such as a processor, an integrated circuit, a chip, etc., for example, it can be performed by the aforementioned audio rendering system and the various modules therein, and the method can also be embodied in a computer program, an instruction, a computer program medium, a computer program product, etc.
[0057] The following will summarize the audio rendering process of some embodiments of the present disclosure with reference to Figure 2, which shows an exemplary schematic diagram of the core processing framework in the audio rendering scheme according to some embodiments of the present disclosure. Generally speaking, the encoding and decoding process of the spatial audio signal according to the present disclosure can be embodied in the framework shown in Figure 2. The audio encoding scheme involved in the present disclosure, especially the encoding of the distance, belongs to the sub-step of spatial encoding or the sub-module of the spatial encoding module, which is particularly used for the object-based spatial sound encoding method. The audio decoding scheme involved in the present disclosure, especially the decoding of the distance, belongs to the sub-step of spatial decoding or the sub-module of the spatial decoding module, which is particularly used for the object-based spatial sound decoding method. Unless otherwise specified, the encoding and decoding of the audio signal mentioned here and below refers to the Ambisonics encoding and decoding as described above.
[0058] For example, the audio rendering system primarily consists of a rendering metadata system and a core rendering system. The metadata system contains control information describing the audio content and rendering techniques, such as whether the audio input format is single-channel, dual-channel, multi-channel, object, or sound field (HOA), as well as dynamic sound sources and listener spatial location information, and rendered acoustic environment information such as room shape, size, and wall quality. The core rendering system performs rendering for the corresponding playback devices and environments based on different audio signal representations and metadata.
[0059] First, an input audio signal is received. Specifically, processing can be performed based on the format of the input audio signal, such as parsing or direct transmission. On the one hand, if the input audio signal is in any spatial audio exchange format, the input audio signal can be parsed to obtain an audio signal with a specific spatial audio representation, such as an object-based spatial audio representation, a scene-based spatial audio representation, or a channel-based spatial audio representation, along with associated metadata. The parsed result is then passed to subsequent processing stages. On the other hand, if the input audio signal is directly in a specific spatial audio representation, parsing is not required and the signal is passed directly to subsequent processing stages. For example, such an audio signal can be passed directly to the audio encoding stage. For example, it can be a narrative soundtrack within an object-based audio representation, a scene-based audio representation, or a channel-based audio representation that requires encoding. Even if the audio signal with the specific spatial representation is of a type / format that does not require encoding, it can be passed directly to the audio decoding stage. For example, it can be a non-narrative soundtrack within the parsed channel-based audio representation, or a narrative soundtrack that does not require encoding.
[0060] Then, information processing can be performed based on the acquired metadata to extract and obtain audio parameters related to each audio signal. Such audio parameters can also be included in the metadata information. This information processing can be performed separately for either the parsed audio signal or the directly transmitted audio signal. Of course, as mentioned above, such information processing is optional and not required.
[0061] Next, the audio signal of the specific spatial audio representation is encoded. On the one hand, the audio signal of the specific spatial audio representation can be encoded based on the metadata information, and the resulting encoded audio signal can be directly transmitted to the subsequent audio decoding stage, or an intermediate signal can be generated and then transmitted to the subsequent audio decoding stage. On the other hand, if the audio signal of the specific spatial audio representation does not need to be encoded, such audio signal can even be directly transmitted to the audio decoding stage.
[0062] Then, in the audio decoding stage, the received audio signal can be decoded to obtain an audio signal suitable for playback in the user application scenario as an output signal. Such an output signal can be presented to the user through the user application scenario, such as an audio playback device in an audio playback environment.
[0063] It should be noted that the various elements of the audio rendering system described above are merely logical modules divided according to the specific functions they implement, and are not intended to limit specific implementation methods. For example, they can be implemented in software, hardware, or a combination of software and hardware. In actual implementation, the above-mentioned units can be implemented as independent physical entities, or can also be implemented by a single entity (for example, a processor (CPU or DSP, etc.), an integrated circuit, etc.). For example, the encoder, decoder, etc. can be implemented as a chip (such as an integrated circuit module including a single chip), a hardware component, or a complete product. In addition, the elements shown with dotted lines in the drawings indicate that these elements may exist, but do not need to actually exist, and the operations / functions they implement can be implemented by the processing circuit itself. In addition, the above-mentioned modules are shown with dotted lines in the drawings to indicate that these units may not actually exist, and the operations / functions they implement can be implemented by other modules or systems or devices containing the modules themselves. For example, at least one of the pre-processing module 111 and the information processing module 112 shown in Figure 1A can be located outside the acquisition module 11 and exist in the audio rendering system 1, for example, it can be located between the acquisition module 11 and the encoder 12, and process the input audio signal in sequence to obtain the audio signal to be processed by the decoder. It can even be located outside the audio rendering system.
[0064] In addition, the audio rendering system 1 may optionally include other components not shown, such as interfaces, memory, and a communication unit. For example, the interface and / or communication unit may be used to receive an input audio signal to be rendered and output the resulting audio signal to a playback device in the playback environment for playback. In one example, the communication unit may be implemented in a suitable manner known in the art, for example, including communication components such as an antenna array and / or a radio frequency link, various types of interfaces, communication units, and the like. These will not be described in detail here. Furthermore, the device may include other components not shown, such as a radio frequency link, a baseband processing unit, a network interface, a processor, a controller, and the like. These will not be described in detail here. For example, the memory may store various information generated by the various modules included in the system or device during operation, programs and data used for operation, data to be transmitted by the communication unit, and the like. For example, it may store various data, information, programs, and the like used in and / or generated during the audio rendering process. The memory may be volatile memory and / or non-volatile memory. For example, the memory may include, but is not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. Of course, the memory may also be located external to the device.
[0065] The following describes in detail an exemplary implementation of audio signal encoding and decoding according to an embodiment of the present disclosure with reference to the accompanying drawings. In the embodiment of the present disclosure, the audio signal encoding and decoding processing takes into account audio spatial position information. In particular, this audio spatial position information can be included in or equivalent to spatial orientation information, especially including spatial distance information, thereby further improving the accuracy of spatial encoding and decoding and improving audio rendering effects.
[0066] In some embodiments, an audio coding scheme for audio rendering is proposed. For an input audio signal in a specific audio content format, the audio signal in the specific audio content format can be spatially encoded based on metadata associated with the audio signal in the specific audio content format to obtain an encoded audio signal. Specifically, the metadata can include audio spatial position information, which can refer to spatial position information related to the audio signal, including, but not limited to, spatial position information of the audio signal itself, spatial position information of the sound source of the audio signal, spatial propagation information of the audio signal, spatial position information of the listener, and information related to the listening environment. For example, the spatial position information of the sound source includes at least one of the distance, vertical angle, and horizontal angle of the sound source. Specifically, the spatial position information of the object-based audio representation signal includes at least one of position information of each audio element in the audio representation signal in a coordinate system, distance information of each audio element, various angles, or relative position information of the sound source related to the audio signal relative to the listener.
[0067] According to some embodiments of the present disclosure, the audio signal in a specific audio content format may include at least one of an object-based spatial audio representation signal, a channel-based spatial audio representation signal, a scene-based spatial audio representation signal, etc., and the encoded audio signal in the specific spatial format may include an Ambisonics audio signal.
[0068] According to some embodiments of the present disclosure, an audio signal encoder (also referred to as an audio signal encoding device, an audio signal encoding module, etc.) for audio rendering may be configured to obtain metadata-related information associated with an audio signal, the metadata-related information including spatial position information associated with the audio signal; and spatially encode the audio signal based on the metadata-related information associated with the audio signal to obtain an encoded audio signal. In embodiments of the present disclosure, the audio signal encoder may be implemented in various suitable ways.
[0069] Figure 3A shows an exemplary block diagram of an audio signal encoder (also referred to as an audio signal encoding device, audio signal encoding module, etc.) for audio rendering according to some embodiments of the present disclosure. The audio signal encoder 300 shown in Figure 3A includes an acquisition unit 310 for acquiring metadata-related information associated with an audio signal, including spatial position information associated with the audio signal; and an encoding unit 320 for spatially encoding the audio signal based on the metadata-related information associated with the audio signal to obtain an encoded audio signal. Here, the audio signal may be an audio signal in a specific audio content format. The spatial position information associated with the audio signal may, in particular, include spatial position information of a sound source of the audio signal, and more specifically, distance information associated with the audio signal. In some embodiments, the encoding unit 320 may include a determination unit 321 configured to determine encoding parameters based on the metadata-related information; and a sub-encoding unit 322 configured to encode the audio signal into an encoded audio signal in a specific spatial format based on the encoding parameters. Of course, the determination unit and sub-encoding unit may not be present, with their functions being implemented by the encoding unit itself.
[0070] According to embodiments of the present disclosure, the acquisition unit can be implemented in various appropriate ways and can receive various appropriate audio signals. In some embodiments, the acquisition unit can directly receive an audio signal in a specific audio content format. As an example, the acquisition unit can be a specific input port / input device. In other embodiments, the acquisition unit can further parse or process the input audio signal to obtain an audio signal in a specific audio content format, similar to the aforementioned pre-processing, which will not be described in detail here.
[0071] According to some embodiments of the present disclosure, the acquired audio signal in a specific audio content format may be referred to as an audio signal to be encoded. As an example, the acquired audio signal may be a non-direct / transparent audio signal and may have various audio content formats or audio representations, such as at least one of the three types of audio signal representations described above, or other suitable audio signals. As an example, such an audio signal may be, for example, an object-based audio representation signal, a scene-based audio representation signal, or a narrative soundtrack that has been pre-specified to be encoded for a specific application scenario, such as the channel-based audio representation signal described above. In particular, the acquired audio signal may be directly input, such as a signal without signal parsing, as described above, or may be extracted / parsed from an input audio signal, such as an audio signal obtained by the signal parsing module described above and requiring audio encoding. As an example, the audio signal in the specific audio content format is an object-based audio representation signal, and the audio signal encoding module is configured to spatially encode the object-based audio representation signal based on spatial position information associated with the object audio representation signal. In particular, encoding may be performed via matrix multiplication.
[0072] In some examples, the acquisition unit may be implemented to only receive the audio signal to be encoded, such as the audio signal to be encoded that is directly input or obtained from the audio signal parsing module, as described above.
[0073] In other embodiments, the audio signal encoder can be implemented to receive audio signals or audio representation signals in various audio content formats. In this way, in implementation, it can be determined whether the audio signal received by the audio signal encoder is an audio signal that needs to be encoded, and if it is determined that it is an audio signal that needs to be encoded, the audio signal is transmitted to the acquisition unit and the encoding unit; if it is determined that it is an audio signal that does not need to be encoded, the audio signal is directly transmitted to the decoding module without audio encoding. Such a determination operation can be implemented by a separate unit. In addition to the acquisition unit and the encoding unit, for example, the audio signal encoder can also include a determination unit to implement the determination operation. Alternatively, such a determination operation can be performed by the acquisition unit itself. As an example, the acquisition unit can include a determination subunit to perform the above-mentioned determination operation. In some embodiments, the determination can be performed in various appropriate ways. For example, a comparison can be performed with reference to the audio content format or audio signal representation of the audio. When the format or representation of the input audio signal matches the format or representation of the audio signal that needs to be encoded, it is determined that the input audio signal needs to be encoded. For another example, the determination unit may also receive other reference information, such as application scenario information, predefined rules for specific application scenarios, and the like, and may make a determination based on this reference information. As previously described, upon obtaining the predefined rules for specific application scenarios, the audio signal to be encoded may be selected based on the rules. For another example, the determination unit may also obtain an identifier related to the signal type and determine whether the signal requires encoding based on the identifier. This identifier may take various appropriate forms, such as a signal type identifier or any other appropriate indication of the signal type.
[0074] In yet another example, the acquisition unit may not even be included in the audio signal encoder, but merely provides the acquired audio signal and metadata-related information to the calculation unit for subsequent calculation and encoding.
[0075] According to embodiments of the present disclosure, metadata-related information associated with an audio signal may include at least one of metadata associated with the audio signal and audio parameters of the audio signal derived based on the metadata. In some embodiments, the metadata-related information may include metadata associated with the audio signal, such as metadata obtained together with the audio signal, for example, directly input or obtained through signal parsing, as described above. In some embodiments, the metadata-related information may be directly input when the audio signal is input. For example, if the input audio signal can be directly input via an API without performing the aforementioned audio signal parsing, the metadata-related information may be input along with the audio signal when the audio signal is input, or separately from the audio signal. In other embodiments, the metadata-related information may also include audio parameters of the audio signal derived based on the metadata, as described above with respect to the operation of the information processing module. In particular, the metadata information may be obtained through signal parsing. In some examples, the metadata-related information may be metadata associated with a specific audio representation signal obtained by parsing an input signal in a spatial audio exchange format, as described above. In other embodiments, the metadata of the parsed audio signal or the directly input metadata may be further processed, such as information processing, to obtain appropriate audio parameters / information as metadata information for audio encoding. This information processing may be referred to as scene information processing, and in this information processing, metadata associated with the audio signal may be processed to obtain appropriate audio parameters / information.
[0076] Figure 3B shows an exemplary implementation of spatial coding according to some embodiments of the present disclosure. First, for object-based spatial coding, an object-based spatial audio representation signal can be obtained, and metadata of the object can also be obtained. The metadata may at least include spatial position information related to the audio object, such as spatial position information of the sound source of the audio object, such as spatial distance information. Then, coding parameters are calculated based on the metadata of the object, and audio object HOA signals are encoded from the object-based spatial audio representation signal based on the coding parameters for use in audio rendering.
[0077] According to some embodiments of the present disclosure, the coding parameters may include parameters related to the coding coefficients, which may be expressed as a coding coefficient matrix, particularly including a distance-related coding coefficient matrix, and the calculation unit may be further configured to construct a distance-related coding coefficient matrix based on the spatial position information of the sound source, particularly to construct a distance-related coding coefficient matrix based on the distance information of the sound source, and the coding unit may be further configured to encode the audio signal into a coded audio signal of a specific spatial format based on the distance-related coding coefficient matrix. Of course, it should be noted that the coding parameters may also include coding coefficients related to other directional components, such as a directionality-related coding coefficient matrix. For simplicity, the coding coefficient matrix may also be referred to as a coding matrix.
[0078] In some embodiments, the encoding parameters may include encoding parameters corresponding to properties of the audio signal. In particular, the properties of the audio signal may relate to channels of the encoded audio signal of a specific spatial format to be encoded, and the encoding parameters may include encoding parameters corresponding to the channels, in particular, a distance-dependent encoding coefficient matrix corresponding to the channels of the encoded audio signal of the specific spatial format to be encoded. Therefore, the calculation unit may be further configured to calculate the corresponding distance-dependent encoding coefficient matrix based on the channels of the encoded audio signal of the specific spatial format to be encoded.
[0079] In some embodiments, the coding coefficient matrix may be related to the directional properties of the audio signal, and in particular may be further calculated based on the directional characteristics of the sound source. In some embodiments, the calculation unit may be further configured to calculate the coding coefficient matrix based on the directional characteristics of the sound source. In some embodiments, the coding coefficient matrix may also be calculated based on both the directional characteristics and the distance characteristics of the sound source, for example, based on the directional characteristics of the sound source and the distance-related coding coefficient matrix. It should be understood that the construction / calculation of the above-mentioned coding matrix may also be implemented before the encoding step, or may be performed during the encoding step.
[0080] In some embodiments of the present disclosure, the encoding of the audio signal can be performed in various appropriate ways, in particular, can be implemented using various appropriate matrix multiplications. In some embodiments, the encoding unit can be further configured to multiply or convolve the audio signal with a coding matrix for encoding.
[0081] As an example, according to an embodiment of the present disclosure, encoding of an audio signal based on spatial position information may include obtaining the position information of the sound source of the audio signal, including distance information and directionality information, and the directionality information may include vertical and horizontal plane angles (such as elevation angles, horizontal angles, etc.); calculating different distance-related coding coefficient matrices according to Ambisonics channels of different orders, and in particular, calculating the corresponding distance-related coding coefficient matrix for each channel; calculating the coding matrix according to the directional information of the sound source, such as characteristics in different directions, and finally multiplying or convolving the signal with the coding matrix for encoding.
[0082] FIG3C illustrates an exemplary implementation of a spatial coding method according to an embodiment of the present disclosure. Here, for an input audio signal S, encoding is performed based on the input position information. Specifically, distance coding is performed based on the distance information in the input position information to construct a distance coding matrix. Angle coding is also performed based on the angle information in the input position information to obtain a position coding matrix. This matrix can be obtained based on a combination of distance coding and angle coding.
[0083] According to an embodiment of the present disclosure, encoding of an audio signal based on spatial position information can be implemented in various appropriate ways. In some exemplary implementations, encoding of an audio signal based on spatial position information can be performed by solving an equation. In some embodiments, an acoustic equation can be constructed in a specific coordinate system. The acoustic equation is constructed for the audio signal based on or taking into account the spatial position information, and contains or can derive a sub-equation or term related to the distance information of the sound source, and the acoustic equation is solved using a specific function. In particular, the sub-equation or term related to the distance information of the sound source in the acoustic equation can be solved using a specific function, thereby implementing encoding of the audio signal based on the distance information of the sound source. Here, equivalently, the solution of the equation, in particular the correlation coefficient in the equation result, can correspond to the encoding parameters for audio encoding. In some embodiments, the encoding parameters can be generated by solving the sound wave equation based on a specific function, in particular, they can be generated by solving the sub-equation or term related to the distance information of the sound source in the acoustic equation based on a specific function. In this way, by considering the distance information based on the sound source, the resulting coding matrix and subsequent coding results are more accurate and comprehensive, and can better reflect the spatial distance characteristics in the audio rendering environment, making the rendering or playback effect more realistic, especially suitable for application scenarios that take into account the near-field effect, such as XR scenarios.
[0084] Additionally, solving the equation may also involve solving sub-equations related to directional information that may be included in or derived from the acoustic equation. This can be combined with solving sub-equations related to sound source distance information, for example, performed simultaneously during the equation solving process. By considering the combination of sound source distance information and directional information, more comprehensive and accurate encoding of the audio signal can be achieved.
[0085] In some embodiments, the specific equation system may be spherical coordinates, and the specific equation may be a specific sound wave equation for the audio signal in spherical coordinates. In particular, the specific equation is an equation for a sound pressure signal, and in spherical coordinates, the sound pressure signal can generally be characterized by spatial distance, elevation angle, horizontal angle, etc., and the equation can be constructed accordingly.
[0086] In some examples, during the solution process, variable separation can be used to obtain a sub-equation related to spatial distance from a specific sound wave equation. The solution to this sub-equation is then obtained based on a specific function, thereby encoding the audio signal. This variable separation can be achieved using Laplace transforms, but can also be achieved using other appropriate methods, algorithms, etc. As an example, equation construction and transformations used in Ambisonics technology in the art can be employed, and will not be described in detail here.
[0087] In some embodiments, the specific function may include a function corresponding to the acoustic equation or the acoustic wave equation. For example, the specific function may include a function of the same type as the acoustic equation or the acoustic wave equation or other compatible / applicable functions. In particular, the specific function may include a spherical harmonic function, such as a Neumann function, a Hankel function, a Bessel function, etc., such as a first-class Hankel function, a second-class Hankel function, etc., and additionally, it may also include a numerical solution expression form and a matrix expression form of the spherical harmonic function, such as various numerical solution expressions and matrix expressions of the above various functions. In some embodiments, the solution of the equation can be represented based on the specific function, especially the solution of the sub-equation related to the distance information of the sound source, so that the term related to the distance information of the sound source can be appropriately included or reflected in the solution of the acoustic wave equation obtained, and then also reflected in the encoding result, that is, the encoding coefficient matrix or encoding matrix corresponding to the solution of the acoustic wave equation can include a component related to the distance information of the sound source. As an example, the solution of the acoustic wave equation, especially the solution of the sub-equation related to the distance information of the sound source, may be characterized as a standing wave form or a traveling wave form based on a specific function.
[0088] The following briefly describes the algorithm logic for encoding based on spatial distance according to an embodiment of the present disclosure.
[0089] In traditional audio coding schemes, especially traditional Ambisonics schemes without distance, two spatial dimensions are mainly concerned. The rendering and encoding of , where the former indicates the elevation angle and the latter indicates the horizontal angle, can generally be represented by constructing the acoustic equations for the audio playback / rendering environment and then solving them, particularly solving the equations for the directional components. Specifically, the acoustic wave equations for the acoustic signal are typically constructed in spherical coordinates, followed by equation transformations (e.g., Laplace transforms) and solutions. The algorithm logic is briefly described below.
[0090] In particular, the acoustic wave equation in spherical coordinates can be constructed using the Laplace operator and the partial derivatives of the sound pressure signal as follows:
[0091] Acoustic wave equation in spherical coordinates
[0092] The Laplace operator in spherical coordinates is
[0093] The acoustic wave equation can be considered as an equation constructed for the acoustic / sound pressure signal p of the audio, which can be expressed as r indicates the spatial distance-related component / dimension, and That Indicates the direction-related components / dimensions, namely the horizontal angle and the vertical angle, c0 indicates the speed of sound, t indicates the time, Indicates the partial derivative of the sound pressure signal.
[0094] Then, the above-mentioned acoustic wave equation can be subjected to variable separation. In particular, variable separation can be performed on the three-dimensional components of the sound pressure signal and on time, as follows:
[0095] Where, in formula (3), T indicates the equation of the relationship between sound pressure and time after the separation of variables, k is the separation variable coefficient, in formula (4), R indicates the equation of the relationship between sound pressure and distance r after the separation of variables, in formula (5), θ indicates the equation of the relationship between sound pressure and θ after the separation of variables, λ is the separation variable coefficient, in formula (6), Ф indicates the relationship between sound pressure and The unknown equation is , and v is the coefficient of the separation variable.
[0096] In conventional solutions, the solution of equation (4) involving the distance r has poor convergence, and equation (4) is often ignored due to the far-field assumption of the sound field. Therefore, conventional solutions often do not consider or ignore the solution of equation (4). On the other hand, considering that equations (5) and (6) already include the two dimensions in spatial coding and It can represent the sound field characteristics in different directions in space. Therefore, the traditional Ambisonics encoding method is mainly used to solve equations (5) and (6) in the above equations and is widely used in various engineering applications.
[0097] In view of this, the present disclosure proposes an improved spatial coding scheme. As an example, the scheme of the present disclosure specifically solves the distance term equation. In particular, a suitable distance term equation can be constructed using a specific function (such as Bessel function, Neumann function, Hankel function, etc.), and a specific form of the equation solution is set, such as the acoustic signal waveform form, such as the standing wave form, the traveling wave form, etc. of the acoustic signal, thereby appropriately solving the distance term equation, thereby realizing distance-related rendering in audio rendering. The following is a brief description of the algorithm logic of the present disclosure as an example. The algorithm logic can transform and solve the formula in spherical coordinates as above, and the formula can be represented as various appropriate function equations.
[0098] In the radial r part, let x = kr, equation (4) becomes
[0099] The above equation is called the spherical Bessel equation, and the spherical Bessel function is defined as
[0100] Where J and N are the cylindrical Bessel and Neumann functions. The solution of equation (7) can be expressed as a standing wave form R(r) = A l j l (kr)+B l n l (kr) (9)
[0101] Define the spherical Hankel function of the second kind and
[0102] Then the solution of equation (9) can also be expressed in traveling wave form.
[0103] In the above formula, R is the distance function, l is the separation variable constant of the partial differential equation, A and B are the constants / coefficients of the combined functions when the distance function is characterized by a combination of various functions, where formula (10) indicates that the Hankel function can be in the form of a complex function, which includes a real part and an imaginary part.
[0104] Thus, the distance can be rendered by solving the distance term in the equation. In particular, it is equivalent to encoding the distance of the sound field by using spherical harmonic functions, such as Hankel functions and their approximate solutions, so that more accurate encoding can be achieved. It should be noted that the solution for the directional component is not excluded in the embodiment of the present disclosure, for example, the solution of the aforementioned equations (5) and (6) can be performed. As an example, the equations (4)-(6) can be solved separately or together as described above during operation, which is equivalent to obtaining coding parameters that comprehensively consider the distance and directional characteristics, thereby achieving more comprehensive and realistic rendering.
[0105] The following describes an implementation of an audio signal encoding method for audio rendering according to embodiments of the present disclosure. Specifically, FIG3D illustrates a flowchart of an audio signal encoding method for audio rendering according to some embodiments of the present disclosure. The method may include: obtaining metadata-related information associated with an audio signal, the metadata-related information including spatial position information associated with the audio signal; and spatially encoding the audio signal based on the metadata-related information associated with the audio signal to obtain an encoded audio signal.
[0106] In some embodiments, spatially encoding the audio signal to obtain the encoded audio signal may further include: determining encoding parameters based on the metadata related information; and encoding the audio signal into an encoded audio signal in a specific spatial format based on the encoding parameters.
[0107] In some embodiments, the spatial position information related to the audio signal includes distance information of a sound source of the audio signal, and determining the encoding parameters based on the metadata-related information may further include constructing a distance-related encoding coefficient matrix based on the distance information of the sound source of the audio signal. Here, the distance-related encoding coefficient matrix may be included in the encoding parameters or used as the encoding parameters.
[0108] In some embodiments, determining the encoding parameters based on the metadata-related information may further include: determining the encoding parameters based on a combination of spatial position information of the sound source and reference spatial position information.
[0109] In some embodiments, the spatial position information of the sound source includes the distance of the sound source, the reference spatial position includes the reference distance, the combination of the spatial position information of the sound source and the reference spatial position includes the ratio of the distance of the sound source to the reference distance, and wherein, determining the encoding parameters based on the combination of the spatial position information of the sound source and the specific reference spatial position may also include: constructing a distance-related encoding coefficient matrix based on the ratio of the distance of the sound source to the reference distance.
[0110] In some embodiments, the metadata-related information also includes directional information of the sound source of the audio signal, and determining the encoding parameters based on the metadata-related information may also include: determining the encoding matrix based on the directional information of the sound source of the audio signal, where the encoding coefficient may be included in the encoding parameter or used as the encoding parameter, and encoding the audio signal into a coded audio signal of a specific spatial format based on the encoding parameter may also include: multiplying or convolving the audio signal with the encoding matrix for encoding.
[0111] In some embodiments, audio signal encoding is performed by solving an acoustic equation constructed for the audio signal, and determining the encoding parameters based on the metadata-related information may also include: determining the encoding parameters by solving a sub-equation related to the distance information of the sound source that is contained in or can be derived from the acoustic equation based on a specific function.
[0112] In some embodiments, audio signal encoding is performed based on solving an acoustic equation constructed for the audio signal, and the method further includes applying a specific filter to the solution of the acoustic equation. As an example, the filter includes at least one of an IIR filter and an FIR filter, wherein the IIR filter is constructed by performing spectrum fitting based on frequency information in the digital domain using a least squares method; and the FIR filter is obtained by performing a time-frequency transform to obtain zero and pole characteristics and then using a bilinear transformation.
[0113] In some embodiments, determining the encoding parameters based on the metadata-related information may further include: determining a corresponding distance-related encoding coefficient matrix for a channel of the encoded audio signal in a specific spatial format.
[0114] FIG3D illustrates an exemplary implementation of an audio encoding method according to some embodiments of the present disclosure.
[0115] It should be noted that the audio encoding method according to the embodiments of the present disclosure can be implemented by various appropriate devices and in various appropriate ways. In particular, it can be performed by the audio encoding device according to the embodiments of the present disclosure, for example, by corresponding units in the audio encoding device, or by other appropriate devices. Detailed description will not be given here.
[0116] The following describes an audio signal decoding scheme according to an embodiment of the present disclosure. In the audio signal decoding method of the present disclosure, decoding can be implemented using operations or steps corresponding to audio signal encoding, which can be considered, to some extent, the inverse process of encoding. In particular, for encoded audio signals of a specific spatial format obtained by encoding, such as Ambisonics signals, decoding can be performed using a decoding matrix. For example, the Ambisonics signal can be multiplied or convolved with the decoding matrix to obtain the decoded signal. In the embodiments of the present disclosure, the decoding matrix can be obtained in an appropriate manner. As an example, the decoding matrix can also be determined based on spatial position information related to the audio signal, for example, it can be a predefined decoding matrix for distance and / or position, especially including a decoding matrix based on spatial distance. The determination of the decoding matrix can be similar to the determination of the encoding matrix described above, and can be used as the inverse of encoding. As another example, the decoding matrix can also be determined based on the rendering / playback environment, for example, based on the device configuration of the rendering / playback environment. In particular, for orthogonal type encoding, the encoding and decoding matrices are transposed of each other.
[0117] In some embodiments of the present disclosure, the encoded audio signal (i.e., the audio signal to be decoded) can be further processed during the decoding process, including rotation, amplitude adjustment, and so on. For example, the audio signal to be decoded can be adjusted as needed. For example, if the sound image is required to rotate with the listener's head, a sensor is required to detect the listener's head direction, thereby pre-rotating the Ambisonics signal and adjusting its amplitude accordingly, thereby obtaining a new Ambisonics signal before decoding.
[0118] According to an embodiment of the present disclosure, an audio signal decoding device for audio rendering is provided, the device being configured to: obtain an encoded audio signal, wherein the encoded audio signal includes an encoded audio signal obtained by an audio signal encoding device according to an embodiment of the present disclosure; and spatially decode the encoded audio signal to obtain a decoded audio signal for audio rendering. The audio signal decoder in the present disclosure can be implemented in various suitable ways.
[0119] Figure 4A shows an exemplary block diagram of an audio decoding device (audio signal decoder) according to some embodiments of the present disclosure. As shown in Figure 4A, the audio signal decoder 400 includes an acquisition unit 410, configured to acquire an encoded audio signal, wherein the encoded audio signal includes an encoded audio signal obtained by the audio signal encoding device according to an embodiment of the present disclosure, and a decoding unit 420, which is configured to spatially decode the encoded audio signal to obtain a decoded audio signal for audio rendering.
[0120] Here, in some embodiments, the decoding matrix may be pre-defined based on attribute information of the sound rendering device, wherein the attribute information includes at least one of spatial position information or type information of the sound rendering device.
[0121] In particular, the spatial position information of the audio rendering device may include, for example, at least one of a distance to the audio rendering device, a directional angle, and the like. For example, when the audio rendering device is an external rendering device or a device that is remote from the listener, such as various speakers, a decoding matrix may be constructed based on the spatial position information of such speakers, thereby enabling decoding of the encoding matrix.
[0122] According to an embodiment of the present disclosure, a preprocessing operation may be performed before decoding: the encoded signal is preprocessed based on the listener position information, and then the preprocessed encoded signal is decoded. In some embodiments, the listener position information includes the listener's head direction, and the encoded signal is rotated and / or amplitude-adjusted based on the listener's head direction. The listener here may include a real listener, or an auxiliary listening device worn by the listener, and so on. Accordingly, the audio decoding device may include a preprocessing unit for performing the preprocessing operation. It should be noted that such a preprocessing unit and preprocessing operation are optional.
[0123] According to embodiments of the present disclosure, post-processing operations may be performed after decoding to adjust the spatially decoded audio signal for a specific playback device in a user application scenario, particularly adjusting the audio signal characteristics, so that the adjusted audio signal can present a more appropriate acoustic experience when rendered by an audio rendering device. Accordingly, the audio decoding device may include a post-processing unit for performing the post-processing operations. It should be noted that such a post-processing unit and post-processing operations are optional.
[0124] The following describes an implementation of an audio signal decoding method for audio rendering according to an embodiment of the present disclosure. In particular, FIG4B shows a flowchart of an audio signal decoding method for audio rendering according to some embodiments of the present disclosure, which may include obtaining an encoded audio signal, wherein the encoded audio signal includes an encoded audio signal obtained by an audio signal encoding device / audio signal encoding method according to an embodiment of the present disclosure, and spatially decoding the encoded audio signal to obtain a decoded audio signal for audio rendering.
[0125] In some embodiments, spatially decoding the encoded audio signal may further include constructing a decoding matrix, and applying the decoding matrix to the encoded audio signal to spatially decode the encoded audio signal.
[0126] In some embodiments, the audio signal decoding method may further include: pre-processing the encoded audio signal based on the listener position information, and then spatially decoding the pre-processed encoded audio signal. It should be noted that the pre-processing here is optional.
[0127] In some embodiments, after decoding, the audio signal decoding method may further include post-processing operations to adjust the spatially decoded audio signal for the specific playback device in the user application scenario, particularly adjusting the audio signal characteristics, so that the adjusted audio signal can present a more appropriate acoustic experience when rendered by an audio rendering device. It should be noted that this post-processing is optional.
[0128] It should be noted that the audio decoding method according to the embodiments of the present disclosure can be implemented by various appropriate devices and in various appropriate ways. In particular, it can be performed by the audio decoding device according to the embodiments of the present disclosure, for example, by corresponding units in the audio decoding device, or by other appropriate devices. Detailed description will not be given here.
[0129] Some preferred embodiments of the coding and decoding scheme according to the embodiments of the present disclosure will be further described below, in which the divergence problem that may be encountered when implementing spatial coding by solving equations is particularly taken into consideration, and the implementation of the coding scheme is optimized.
[0130] Generally, the sound pressure signal can be expanded into
[0131] Where J is the encoding function of the sound source distance, Y is the encoding function of the sound source angle, and a and b are the encoded signals. For different sound field characteristics, the sound waves change differently with distance. The symbols here represent all common representations that can represent sound field characteristics, but no matter what representation is used, the closer the distance to the sound source, the greater the sound pressure will be, causing the distance encoding function to not converge in some areas. In particular, the above mentioned logic of using spherical harmonic functions such as Hankel functions to solve equations to encode distance according to the embodiment of the present disclosure, but it should be pointed out that in actual use, in some special cases, the Hankel function will diverge rapidly when the coordinate r→0, which may face mathematical infinity problems and cannot be solved at that time. For this, based on the possibility that the ratio of two infinite numbers in mathematics may be a finite number, the embodiment of the present disclosure proposes selecting a reference point in space and optimizing the solution of the equation with the help of the ratio achieved by the reference point, which can effectively alleviate the divergence problems that may be encountered in spatial coding, such as non-convergence problems, infinity problems, etc. In particular, for the possible problem of infinite coding signals, by selecting a reasonable reference point and recording the ratio of the signal to the reference point, the infinity is converted into a finite problem, which effectively solves the problem and improves the stability of the coding scheme.
[0132] According to some embodiments of the present disclosure, reference spatial position information is also set for audio encoding, and based on the combination of the spatial position information of the sound source and the specific reference spatial position, the audio signal encoding is further optimized. This can be equivalent to adjusting or optimizing the encoding matrix to some extent, making the encoding process or the encoding matrix more stable and accurate.
[0133] According to some embodiments of the present disclosure, the combination of the spatial position information of the sound source and the specific reference spatial position includes a ratio relationship between the spatial position information of the sound source and the reference spatial position, and / or wherein the spatial position information of the sound source includes the distance of the sound source, and the reference spatial position includes the reference distance. In some embodiments, the reference spatial position or the reference distance may be set in various appropriate ways. According to some embodiments of the present disclosure, the reference spatial position may be the spatial position of an audio rendering device, and the audio rendering device includes at least one of a speaker and a virtual speaker on the audio rendering side. It should be noted that the reference spatial position or the reference distance may also be set in other appropriate ways, and may be set to other appropriate values, which will not be described in detail here.
[0134] Thus, in an embodiment of the present disclosure, by introducing a reference value, relative position information is constructed based on the spatial position information related to the audio signal (especially, for example, the distance information of the sound source of the audio signal), and equations are solved based on this, thereby effectively solving the divergence problem. In particular, in the distance coding process, the ratio of the spatial position information to the reference value is used as relative position information to replace the spatial position information related to the audio signal in the equation, so that the infinite problem that the equation may face can be converted into a finite problem, and some divergent problems can be solved, so that the equation can be solved effectively, stably and accurately, and then the audio coding and rendering can be further improved. Optionally, in some examples, the information of the reference point can be retained after encoding; thereby, the inverse is performed during decoding based on this information, such as constructing the inverse matrix of the coding matrix, thereby avoiding the divergence problem in the coding process.
[0135] The following example illustrates the spherical harmonics used in Ambisonics. Based on the solution of equation (1), multiplying the distance term by the spherical harmonics gives the solution to the overall acoustic wave equation (also known as the vibration equation), which yields:
[0136] where j l (kr) is the spherical harmonic Bessel function family (or Neumann function, or Hankel function, no specific distinction is made in the following text), k = 2πf / c. According to the relevant mathematical relationship, the Ambisonics encoding equation with distance can be obtained. For a point source position at Here ρ indicates the spatial distance, and the encoded signal is:
[0137] in
[0138] is a divergent spherical Hankle function of the second kind. Here is a simple equation of the spherical Hankle function:
[0139] in, The encoded audio signal may be indicated, for example an audio signal in various suitable formats, such as an Ambisonics audio signal, wherein a distance parameter and additionally also a directionality parameter are taken into account.
[0140] It can be seen that the spherical Hankel equation contains two parts, one of which is the derivative of the distance, and the other is the delay term of the phase.
[0141] ·For the distance term, it can be seen that it contains multiple powers of the reciprocal of the distance. Compared with traditional Ambisonics that only considers the energy attenuation of the inverse proportional first term, the Ambisonics scheme with distance takes into account higher-order distance attenuation terms and is more accurate.
[0142] ·For the phase delay term, that is, it brings the phase delay term of kρ, which is manifested as the delay of the overall time.
[0143] To sum up, Γ l contains the phase delay and amplitude attenuation terms. Compared with the approximate logic of traditional use of inverse ratio relationship, it is more accurate in amplitude. Especially the increased amplitude at low frequencies is obtained by accurate calculation and is more accurate. In addition, it also contains phase information, reasonably adjusts the phase, reasonably presents the time delay problem of sound wave transmission, so it encodes the sound field more accurately and brings a more realistic effect. It should be noted that in the context, Γ l essentially corresponds to the encoding coefficient matrix, especially the distance-related encoding coefficient matrix. essentially corresponds to the encoding coefficient matrix obtained by further considering the directionality, which indicates that on the basis of obtaining the distance-related encoding coefficient matrix, the direction characteristics are further considered to construct the encoding coefficient matrix.
[0144] It can be seen from Equation (17) that the Hankel function diverges rapidly at the origin. At this time, the scheme of the present disclosure can assume that there is a reference sound source position (ref). Assuming that the distance of the sound source is ρ and the distance of the reference point is R, its distance transfer function matrix is
[0145] where F is the distance equation corresponding to different orders of spherical harmonic functions and the acoustic wave vibration equation, which is in the form of a combination of spherical Bessel functions or its同族 functions under the Ambisonics encoding logic. Of course, other types of distance-related encoding functions / distance equations are also included here. It can be seen that when ρ < R, the absolute value of this correction function is greater than 1; it can be seen that when ρ > R, the abs of this correction function is less than 1. For the spherical Bessel function, it can be seen that this ratio does not diverge.
[0146] Therefore, the final encoding form under Ambisonics encoding can be reduced to
[0147] where S is the signal to be encoded, H is the distance-direction spherical harmonic function encoding basis set with a reference point, which can correspond to the encoding coefficient matrix, especially the distance-related encoding coefficient matrix, that is, the encoding coefficient matrix considering the reference point; Y is the basis set of the spherical harmonic equation. Similarly, This corresponds to a coding matrix, indicating that, based on the distance-related coding coefficient matrix that takes reference points into account, the coding matrix is constructed by further considering directional characteristics. It should be noted that this embodiment demonstrates the distance coding logic for the Ambisonics encoding method; for other distance coding methods that use the sound field as a representation, strategies that can improve divergence issues by selecting reference points should also be included.
[0148] The concept of setting a reference point is particularly beneficial for audio encoding, decoding and rendering in scenarios close to the sound source. It should be noted that the concept of a reference point is not mandatory. In some implementations, the configuration of the reference point may depend on the settings on the decoding side. For example, if a decoding method using other optimization methods is used on the decoding side, and if a far-field assumption is made during decoding, it can be assumed that the decoding speaker is at a farther position and substituted into the formula to calculate the coefficients. In this case, an appropriate decoding scheme can be used for decoding, and / or some optimization strategies can be used to directly optimize the decoded signal without assuming the speaker position. In this way, there is no need to set a reference point for encoding in the encoding process on the encoding side.
[0149] The following further describes other embodiments according to the present disclosure, in which an optimized coding scheme is proposed to achieve fast calculation. In some cases, the final solution of the Hankel function will be relatively complex, so it is desirable to propose an improved solution method to accelerate or facilitate the solution of the equation function, for example, so that the solution can be implemented quickly or even in real time. In particular, according to the logic of the embodiment, since the appearance of distance r is often accompanied by the appearance of wave number k, and for a sound wave with a speed of sound c, the corresponding frequency is Therefore, the final distance correction equation is inherently frequency-dependent. Given this, the present disclosure proposes optimizing filter design to optimize the solution. The filter herein may involve at least one of an infinite impulse response (IIR) filter and a finite impulse response (FIR) filter. Furthermore, different filter amplitude and phase responses exist for sound sources of different orders and distances. Therefore, it is desirable to further optimize the filter design to optimize the solution.
[0150] In view of this, the present disclosure proposes optimized FIR filter design strategies and IIR filter design strategies. The following describes the encoding method under Ambisonics logic, but similar strategies can be adopted for encoding methods under other sound field representation types.
[0151] According to some embodiments of the present disclosure, a filter design optimized for encoding is proposed, wherein the filter includes at least one of an IIR filter and a FIR filter. In particular, in some embodiments, a coding matrix can be determined based on at least one of the optimized IIR filter and the FIR filter.
[0152] As an example, substituting the spherical Bessel polynomial into equation (19), equation (19) can be rewritten as follows:
[0153] That is, the core is to solve the function
[0154] The general solution can be expressed as follows
[0155] At this time, in the encoding process, such as the solution process of the above encoding formula, a filter is introduced to solve the problem. In particular, the filter design is introduced to optimize the design and simplification of the coefficient matrix. The design can be performed using an IIR filter or a FIR filter. The FIR filter can be designed directly in the digital domain using, but not limited to, the least squares method; the IIR filter can be implemented using a polynomial approximation method.
[0156] FIR filter: According to formula (20), the frequency information can be brought in, and then the spectrum can be fitted using the least squares method, so that the filter can be implemented using a lower-order FIR filter, and then implemented using the frequency domain algorithm and overlapped convolution scheme.
[0157] IIR filter: According to Equation (20), Laplace transform is used to perform time-frequency domain conversion to obtain zero and pole characteristics. Then, bilinear transformation is used to obtain its iterative equation. The iterative equation is then decomposed into parallel or series forms, and then implemented using a second-order IIR filter bank. The SIMD instruction set can be used to optimize and speed up the calculation process.
[0158] Therefore, according to an embodiment of the present disclosure, filters are used to simplify the solution of the Hankel function, including IIR and FIR filters, so that the signal can be optimized using a filter optimization strategy, thereby utilizing a lower-order iterative strategy and utilizing existing fast algorithms for digital signal processing for calculation.
[0159] Still other embodiments according to the present disclosure will be further described below, in which optimization of the decoding process, especially the independence of encoding and decoding, is mainly considered.
[0160] Generally, encoding is followed by decoding, but decoding is often performed on the client side. Whether a new encoding method is compatible with the client's existing decoding solution significantly impacts the overall effectiveness of the solution. Compatibility facilitates solution promotion with minimal impact on customers. Therefore, improved encoding and decoding optimization is needed to improve solution compatibility.
[0161] As mentioned above, for near-field decoding, the decoding matrix converges and is stable. However, for near-field encoding, the divergence of the Hankel function makes the encoding matrix unusable. This divergence issue can be circumvented by integrating encoding and decoding, taking advantage of the fact that the Hankel function is infinitely large when its order is consistent. Specifically, the encoding scheme design can directly utilize the decoder's parameter configuration, particularly using the decoder's position parameters as reference values for the encoder, thereby achieving a combined encoding and decoding solution.
[0162] In view of this, in some embodiments, the speaker position information during playback can be applied to the encoding end, combining encoding and decoding. In particular, for a state where the source distance is ρ and the speaker position during playback is R, both information can be directly incorporated into the aforementioned equation (19) for calculation. In this case, since the source distance ρ and the speaker distance R are already included in the encoding process, the decoding process only requires the use of the traditional Ambisonics decoding matrix, which can be implemented by reusing the traditional Ambisonics decoding scheme.
[0163] For example, for decoding, if actual speakers are available, the distance of the speakers used in decoding can be directly incorporated during encoding. If virtual speakers are used for decoding and then post-processed, the distance of the virtual speakers used in decoding can also be directly incorporated during encoding. For decoding methods using other optimization methods, if far-field assumptions are made during decoding, the coefficients can be calculated by assuming that the speakers on the decoding end are located farther away.
[0164] Therefore, during the encoding process, the reference position is directly selected as the decoded speaker position for decoding possible speaker positions (including virtual speakers). At this time, distance encoding and decoding can be integrated into the encoding process, so that the traditional decoding strategy can be reused in the decoding process, and the client does not need to make changes.
[0165] Some exemplary implementations of audio signal encoding processing / audio signal encoders according to embodiments of the present disclosure will be described below, which may be implemented in combination with the aforementioned audio signal encoding and decoding embodiments.
[0166] According to an embodiment of the present disclosure, the specific spatial format of the encoded audio signal can be a spatial format that can be supported by the audio rendering system, for example, it can be played back to the user in different user application scenarios, such as different audio playback environments. In a sense, the encoded audio signal in this specific spatial format can be used as an intermediate signal medium, that is, it indicates that an intermediate signal in a common format is encoded from an input audio signal that may contain various spatial representations, and the intermediate signal is decoded and processed for rendering. The encoded audio signal in this specific spatial format can be an audio signal in a specific spatial format as described above, such as FOA, HOA, MOA, etc., which will not be described in detail here. Thus, for an audio signal that may have at least one of multiple different spatial representations, it can be spatially encoded to obtain an encoded audio signal in a specific spatial format that can be used for playback in a user application scenario. In other words, even if the audio signal may contain different content formats / audio representations, an audio signal in a common or shared spatial format can still be obtained through encoding. In some embodiments, the encoded audio signal can be added to the intermediate signal, for example, encoded into the intermediate signal. In another embodiment, the encoded audio signal can also be directly transmitted / transparently transmitted to the spatial decoder without being added to the intermediate signal. In this way, the audio signal encoding module can be compatible with various types of input signals to obtain encoded audio signals in a common spatial format, thereby enabling the audio rendering process to be performed efficiently.
[0167] In some embodiments, spatial encoding of an object-based audio signal may be performed separately for each of one or more sound objects contained in the audio signal, and the encoding process for each sound object may be performed as described above. In some embodiments, the audio signal encoder is further configured to weightedly combine the encoded signals of each object-based audio representation signal based on the weights of the sound objects defined in the metadata, wherein the weights may be set in various appropriate ways, such as pre-set, empirically set, set based on the strength of the signal object, priority, etc., which will not be described in detail here. In particular, when the audio signal contains multiple sound objects, the object-based audio representation signal may be spatially encoded for each sound object in the audio signal based on information related to the spatial propagation of the sound object in the audio signal, for example, after the audio representation signal is spatially encoded for the spatial propagation path of each sound object as described above, the encoded audio signals of each sound object may be weightedly combined using the weights of each sound object contained in the metadata associated with the audio representation signal.
[0168] In some embodiments, spatial encoding of object-based audio signals (such as the object-based audio signal spatial encoding described above) can be performed separately for one or more spatial propagation paths from a sound object to a listener. Specifically, if there is a single spatial propagation path from a sound object to a listener, spatial encoding of the object-based audio signal is performed for that spatial propagation path. If there are multiple spatial propagation paths from a sound object to a listener, spatial encoding of the object-based audio signal can be performed for at least one, or even all, of the multiple spatial propagation paths. Specifically, information related to each spatial propagation path from a sound object to a listener can be considered separately, and the audio signal corresponding to that spatial propagation path can be encoded accordingly. The encoding results for each spatial propagation path can then be combined to generate an encoding result for the sound object. The spatial propagation path from a sound object to a listener can be determined by various suitable means, particularly by obtaining, but not limited to, spatial position information. In some embodiments, the position information of the spatial propagation path can include the azimuth angle of the spatial propagation path to the listener or the azimuth angle of the spatial propagation path relative to a coordinate system. In some embodiments, the spherical harmonic function based on the position angle of the spatial propagation path can be any suitable form of spherical harmonic function.
[0169] In some embodiments, spatial audio coding of object-based audio signals may further utilize at least one of a near-field compensation function and a diffuseness function to encode the audio signal based on the length of the spatial propagation path from the sound object in the audio signal to the listener. For example, depending on the length of the spatial propagation path, at least one of the near-field compensation function and the diffuseness function may be applied to the audio signal of the sound object in the propagation path to perform appropriate audio signal compensation and enhance the audio effect.
[0170] The audio encoding according to the present disclosure can be performed separately for various audio signals, in particular, encoding can be performed for each audio signal, and in the case of containing multiple audio signals, the encoded audio signals can also be combined, and in particular weighted processing can be performed. According to some embodiments, the audio signal encoding module can be configured to weight the audio signal based on metadata information. In particular, the audio signal encoding module can be configured to weight according to the weight in the metadata. The metadata can be associated with the audio signal to be encoded obtained by the audio signal encoding module, for example, associated with signals with various audio content formats / audio representation signals, as described above. In particular, in some embodiments, the audio signal encoding module can also be configured to weight the audio signal obtained, in particular, an audio signal with a specific audio content format, based on the metadata associated with the audio signal. In other embodiments, the audio signal encoding module can also be configured to further perform additional processing on the encoded audio signal, such as weighting, rotation, etc. In particular, the audio signal encoding module can be configured to convert an audio signal in a specific audio content format into an audio signal in a specific spatial format, and then weight the obtained audio signal in the specific spatial format based on metadata to obtain an intermediate signal. In some embodiments, the audio signal encoding module can be configured to further process the audio signal in the specific spatial format obtained by conversion based on metadata, such as format conversion, rotation, etc. In some embodiments, the audio signal encoding module can be configured to convert the audio signal in the specific spatial format obtained by encoding or directly input to meet the format supported and constrained by the current system, for example, conversion can be performed in terms of channel arrangement method, regularization method, etc. to meet system requirements.
[0171] In some embodiments, the audio signal encoder is configured to spatially encode the object-based audio signal based on at least one of a filter function and a spherical harmonic function. The filter function may be a filter function that filters the audio signal based on the path energy intensity of the spatial propagation path from a sound object in the audio signal to a listener, and the spherical harmonic function may be a spherical harmonic function based on position information of the spatial propagation path. In some embodiments, the audio signal encoding may be performed based on a combination of the filter function and the spherical harmonic function. For example, the audio signal encoding may be performed based on the product of the filter function and the spherical harmonic function.
[0172] In some embodiments, spatial audio encoding of object-based audio signals can further be based on the delay of the sound object in spatial propagation, for example, based on the propagation duration of the spatial propagation path. In this case, the filter function that filters the audio signal based on path energy intensity is a filter function that filters the audio signal of the sound object before it propagates along the spatial propagation path, based on the path intensity energy of the path. In some embodiments, the audio signal of the sound object before it propagates along the spatial propagation path refers to the audio signal at a moment before the time required for the sound object to reach the listener along the spatial propagation path, for example, the audio signal of the sound object before the propagation duration.
[0173] In some embodiments, the position information of the spatial propagation path may include the direction angle of the spatial propagation path reaching the listener or the direction angle of the spatial propagation path relative to the coordinate system.
[0174] Some exemplary implementations of audio signal decoding processing / audio signal decoders according to embodiments of the present disclosure will be described below, which may be implemented in combination with the aforementioned audio signal encoding and decoding embodiments.
[0175] In some embodiments, when the audio rendering device is a device adjacent to the listener, such as a device worn by the listener, such as headphones, earbuds, etc., the type information of the device can be directly considered to set the decoding matrix. For example, the device ID indicates the device type and has a corresponding decoding matrix, etc., so that an appropriate decoding matrix can be selected to implement decoding.
[0176] According to an embodiment of the present disclosure, after the audio signal is audio-encoded or directly / transparently transmitted as described above, such encoded audio signal or directly / transparently transmitted audio signal will be audio-decoded to obtain an audio signal suitable for playback / rendering in the user application scenario. In particular, such an encoded audio signal or directly / transparently transmitted audio signal may be referred to as a signal to be decoded, which may correspond to the audio signal of the specific spatial format described above, or an intermediate signal. As an example, the audio signal of the specific spatial format may be the aforementioned intermediate signal, or may be an audio signal directly / transparently transmitted to the spatial decoder, including an unencoded audio signal, or an encoded audio signal that has been spatially encoded but not included in the intermediate signal, such as a non-narrative channel signal or a binaural signal after reverberation processing. The audio decoding process may be performed by an audio signal decoding module.
[0177] According to an embodiment of the present disclosure, the audio signal decoding module can decode the intermediate signal and the transparent signal onto the playback / playback device according to the playback mode. Thus, the audio signal to be decoded can be converted into a format suitable for playback by a playback device in a user application scenario, such as an audio playback environment or an audio rendering environment. According to an embodiment of the present disclosure, the playback mode can be related to the configuration of the playback device in the user application scenario. In particular, depending on the configuration information of the playback device in the user application scenario, such as the identifier, type, and arrangement of the playback device, a corresponding decoding method can be adopted. In this way, the decoded audio signal can be suitable for a specific type of playback environment, especially for the playback device in the playback environment, thereby achieving compatibility with various types of playback environments. As an example, the audio signal decoder can perform decoding based on information related to the type of the user application scenario. This information can be a type indicator of the user application scenario, for example, a type indicator of the rendering device / playback device in the user application scenario, such as a renderer ID, so that a decoding process corresponding to the renderer ID can be performed to obtain an audio signal suitable for playback by the renderer. As an example, the renderer IDs can correspond to specific renderer arrangements, playback scenarios, playback device arrangements, and the like, as described above. This allows decoding to produce an audio signal suitable for playback of the renderer arrangement, playback scenario, playback device arrangement, and the like corresponding to the renderer ID. In some embodiments, the playback mode, such as the renderer ID, can be pre-specified, transmitted to the rendering end, or input via an input port. In some embodiments, the audio signal decoder decodes the audio signal in a specific spatial format using a decoding method corresponding to the playback device in the user application scenario.
[0178] In some embodiments, the playback device in the user application scenario may include a speaker array, which may correspond to a speaker playback / rendering scenario. In this case, the audio signal decoder may use a decoding matrix corresponding to the speaker array in the user application scenario to decode the audio signal in the specific spatial format. As an example, such a user application scenario may correspond to a specific renderer ID, such as the aforementioned renderer ID2. In particular, for example, corresponding identifiers may be set based on the type of speaker array to more accurately indicate the user application scenario. For example, corresponding identifiers may be set for standard speaker arrays, custom speaker arrays, and so on.
[0179] The decoding matrix can be determined based on the configuration information of the speaker array, such as the type and arrangement of the speaker array. In some embodiments, when the playback device in the user application scenario is a predetermined speaker array, the decoding matrix is a decoding matrix corresponding to the predetermined speaker array built into the audio signal decoder or received from the outside. In particular, the decoding matrix can be a preset decoding matrix, which can be pre-stored in the decoding module, for example, stored in a database in association with / corresponding to the type of the speaker array, or provided to the decoding module in other ways. Thus, the decoding module can call the corresponding decoding matrix according to the known predetermined speaker array type to perform decoding processing. The decoding matrix can be in various appropriate forms, for example, it can include gains, such as HOA track / channel to speaker gain values, so that the gains can be directly applied to the HOA signal to generate output audio channels so as to render the HOA signal to the speaker array.
[0180] As an example, for a standard speaker array defined in the standard, such as 5.1, the decoder will have built-in decoding matrix coefficients, and the playback signal L can be obtained by multiplying the intermediate signal by the decoding matrix.
[0181] L=DS N ,
[0182] Where L is the speaker array signal, D is the decoding matrix, S N is an intermediate signal, obtained as described above. On the other hand, for direct / transparent audio signals, the signal can be converted into the speaker array according to the definition of the standard speaker, for example, it can be multiplied by the decoding matrix as described above, and other suitable methods can also be used, such as vector-based amplitude panning (VBAP), etc. As another example, in the case of special speaker array spatial decoding, for Sound Bar or some more special speaker arrays, the speaker manufacturer is required to provide a correspondingly designed decoding matrix. The system provides a decoding matrix setting interface to receive decoding matrix related parameters corresponding to the special speaker array, so that the received decoding matrix can be used for decoding processing, as described above.
[0183] In other embodiments, when the playback device in the user application scenario is a custom speaker array, the decoding matrix is a decoding matrix calculated according to the arrangement of the custom speaker array. As an example, the decoding matrix is calculated based on the position angle and pitch angle of each speaker in the speaker array or the three-dimensional coordinate value of the speaker. As an example, in the custom speaker array spatial decoding, in the scenario of the custom speaker array, such speakers usually have a spherical, hemispherical design or a rectangle, which can surround or semi-surround the listener. The decoding module can calculate the decoding matrix according to the arrangement of the custom speakers, and the required input is the position angle and pitch angle of each speaker, or the three-dimensional coordinate value of the speaker. The calculation method of the speaker decoding matrix can include SAD (Sampling Ambisonic Decoder), MMD (Mode Matching Decoder), EPAD (Energy Preserved Ambisonic Decoder), AllRAD (All Round Ambisonic Decoder), etc.
[0184] According to some embodiments of the present disclosure, when the playback device in the user application scenario is a headset, it may correspond to scenarios such as headset rendering / playback, binaural rendering / playback, etc., and the audio signal decoder is configured to directly decode the audio signal to be decoded into a binaural signal as a decoded audio signal, or to obtain a decoded signal as a decoded audio signal through speaker virtualization. As an example, such a user application scenario may correspond to a specific renderer ID, such as the aforementioned renderer ID1. As an example, there may be a variety of appropriate decoding methods for the playback environment of headphones. In some embodiments, for example, the signal to be decoded, such as the aforementioned intermediate signal, may be directly decoded into a binaural signal. In particular, the signal to be decoded may be directly decoded, for example, the HOA signal may be converted by determining a rotation matrix based on the listener's posture, and then the HOA channel / track may be adjusted, such as performing convolution (for example, convolution using a gain matrix, a harmonic function, HRIR (head-related impulse response), spherical harmonic HRIR, etc., such as frequency domain convolution), so that a binaural signal may be obtained. In other words, this process can be viewed as directly multiplying the HOA signal by a decoding matrix, which can include rotation matrices, gain matrices, harmonic functions, and so on. Typical methods include LS (least squares), Magnitude LS, and SPR (Spatial resampling). For transparent signals, typically binaural signals, playback is performed directly. As another example, indirect rendering can also be performed, where a speaker array is first used, and then HRTF convolution is performed based on the speaker's position to virtualize the speaker and obtain the decoded signal.
[0185] In some embodiments, during the audio decoding process, the audio signal to be decoded can also be processed based on metadata information associated with the audio signal to be decoded. In particular, the audio signal to be decoded can be spatially transformed based on the spatial transformation information in the metadata information. For example, when the metadata information indicates that rotation is required, the sound field rotation operation can be performed on the audio representation signal to be decoded based on the rotation information indicated in the metadata. As an example, first, according to the processing method of the previous module and the rotation information in the metadata, the intermediate signal is multiplied by the rotation matrix as needed to obtain the rotated intermediate signal, so that the rotated intermediate signal can be decoded. It should be noted that the spatial transformation here, such as spatial rotation, can be performed alternatively with the spatial encoding, such as spatial rotation, in the spatial encoding process described above.
[0186] The following describes exemplary implementations of audio processing solutions, particularly audio rendering solutions, according to some embodiments of the present disclosure. These solutions involve, based on an input audio signal, obtaining an output audio signal suitable for rendering in a user application scenario, particularly for presentation / playback to a user via a device in a playback environment. The audio rendering solutions according to embodiments of the present disclosure may include the aforementioned audio codec solutions according to embodiments of the present disclosure, and may also include other appropriate audio processing, which will be described in further detail below.
[0187] In this process, an audio signal in a specific spatial format that can be used for playback in a user application scenario can be obtained based on the input audio signal. In the audio rendering process according to the present disclosure, an audio signal in a specific spatial format is obtained based on the input audio signal. Depending on the signal format of the input signal, such as the signal representation method, encoding method, audio content format, etc., for certain signal formats, the input signal or some signals contained in the input signal can be directly transmitted / transparently transmitted to the decoding block for decoding without spatial audio processing. In this case, these signals can be directly used as audio signals in a specific spatial format without spatial audio processing. In particular, the audio signal in a specific spatial format includes audio signals that do not require spatial encoding in an audio signal with an audio content format compatible with the audio rendering system, which is derived from the input audio signal. In some embodiments, the audio signal with an audio content format compatible with the audio rendering system may include at least one of an object-based audio representation signal, a scene-based audio representation signal, and a channel-based audio representation signal. In some embodiments, the audio signal that does not require spatial encoding includes at least one of a non-narrative channel track and a narrative channel track that does not require spatial audio processing in the channel-based audio signal. As an example, when the format of the input audio signal directly corresponds to the specific spatial format, the input audio signal can be directly used as the audio signal of the specific spatial format, that is, directly transmitted / transparently transmitted to the audio signal spatial decoder.
[0188] In some embodiments, for other signal formats, the input signal or some signals contained in the input signal may undergo spatial audio processing to obtain an audio signal in a specific spatial format. For example, spatial audio processing may include, but is not limited to, pre-processing, audio information processing, and spatial encoding of audio signals. These processes may be implemented using appropriate devices and components, which may be implemented in various appropriate ways, such as hardware, firmware, and software.
[0189] Then, the audio signal in the specific spatial format can be decoded, and the decoded audio signal can be presented to the user in a user application scenario, such as in an audio playback environment, through an audio rendering device / audio playback device to meet the user's needs.
[0190] Such a process can be implemented by the audio rendering system according to the embodiment of the present disclosure, in particular, by various appropriate modules in the audio rendering system. Some exemplary implementations of the above process will be described below.
[0191] (Input) Audio signal acquisition
[0192] Signals suitable for rendering processing by the audio rendering system can be acquired through various appropriate means. According to embodiments of the present disclosure, signals suitable for rendering processing by the audio rendering system can have an audio content format compatible with the audio rendering system, such as an audio signal in a specific audio content format. The audio content format can correspond to an audio representation method, which can include a specific audio representation method that includes at least one of an object-based audio representation signal, a scene-based audio representation signal, and a channel-based audio representation signal. In some embodiments, an audio signal in a specific audio content format can be directly input into the audio rendering system, that is, the audio signal in a specific audio content format can be directly input as an input signal and thus directly acquired. In other embodiments, the audio signal in a specific audio content format can be acquired from an audio signal input into the audio rendering system. As an example, the input audio signal may be an audio signal in another format, such as a specific combination of audio signals in a specific audio content format or a signal in another format. In this case, the audio signal in the specific audio content format can be acquired by parsing the input audio signal. In this case, the input signal acquisition module can be referred to as or include an audio signal parsing module, and the signal processing performed by the module can be referred to as a type of signal pre-processing, particularly processing performed before the audio signal is encoded. As an example, the audio signal analysis module may be or may be included in the pre-processing module, and the operations performed by the audio signal analysis module may also be performed by the pre-processing module.
[0193] Audio signal analysis
[0194] According to some embodiments of the present disclosure, taking into account different application scenarios, audio signals may be input in different input formats. Therefore, audio signal parsing can be performed before audio rendering processing to be compatible with inputs of different formats. Such audio signal parsing processing can be considered as a kind of pre-processing / pre-processing, for example, it can be performed by an audio signal parsing module. In some embodiments, the audio signal parsing module or the pre-processing module can be configured to obtain an audio signal having an audio content format compatible with the audio rendering system from the input audio signal, and in particular, can parse any input spatial audio exchange format signal to obtain an audio signal having an audio content format compatible with the audio rendering system.
[0195] According to other embodiments of the present disclosure, the audio signal parsing module or pre-processing module is configured to parse any input spatial audio exchange format signal and convert the signal into a format agreed upon by the audio rendering system according to the signal format type. In particular, in some embodiments, the obtained audio signal having an audio content format compatible with the audio rendering system can be further converted so that the audio signal has a predetermined format, in particular, a predetermined format of the audio rendering system, for example, the signal is converted into a format agreed upon by the audio rendering system according to the signal format type. In particular, the predetermined format can correspond to predetermined configuration parameters of an audio signal in a specific audio content format, so that in the audio signal parsing operation, the audio signal in a specific audio content format can be further converted into predetermined configuration parameters. In some embodiments, when the audio signal having an audio content format compatible with the audio rendering system is a scene-based audio representation signal, the audio signal parsing module or pre-processing module is configured to convert a scene-based audio signal with different channel orderings and normalization coefficients into channel orderings and normalization coefficients agreed upon by the audio rendering system.
[0196] As an example, the pre-processing module can be used to pre-process any spatial audio exchange format signal being distributed. For example, an audio signal having a specific audio representation can be obtained from an input audio signal. Specifically, for example, any spatial audio exchange format signal for distribution, whether non-streaming or streaming, can be classified into three types of signals based on the spatial audio signal representation method through an input signal parser: at least one of a scene-based audio representation signal, a channel-based audio representation signal, and an object-based audio representation signal, as well as metadata corresponding to each type of signal. Alternatively, the spatial audio signal can be further converted into a predetermined format, such as a format pre-defined or predetermined by an audio rendering system or even an audio system. Of course, this format conversion is not required. As an example, the pre-processing can also include converting the signal into a system-constrained format based on the format type. For example, for the scene-based spatial audio representation signal HOA, different channel orderings (such as ACN, Ambisonic Channel Number, FuMa, Furse-Malham and SID, Single index designation) and different normalization coefficients (N3D, SN3D, FuMa) are used in different data exchange formats. In this step, they can be converted into a certain agreed channel ordering and normalization coefficient, such as (ACN+SN3D).
[0197] In some embodiments, the input audio signal is directly used as the audio signal in the specific spatial format. That is, the input audio signal does not require pre-processing and is directly passed / transparently transmitted to the audio signal decoder for processing. In some embodiments, the pre-processing module includes an application programming interface (API) for directly receiving the input audio signal and transmitting it directly to the audio signal decoder.
[0198] For example, this pre-processing module can implement pre-processing of input signals from custom applications. In some custom applications, this pre-processing can be performed directly on the consumer side, eliminating the need for spatial audio exchange format conversion. By providing an API within the rendering system, the application can directly pass the corresponding audio data and metadata into the system.
[0199] According to an embodiment of the present disclosure, an audio signal in a specific spatial format may be obtained by spatially encoding an input audio signal, particularly by spatially encoding an audio signal obtained through pre-processing and parsing. Here, spatial encoding may be implemented using an encoding scheme according to an embodiment of the present disclosure.
[0200] Obtaining metadata information
[0201] In embodiments of the present disclosure, metadata information related to the audio signal may also be obtained. In embodiments of the present disclosure, the metadata information may include various appropriate data. As an example, the metadata information may include information corresponding to the audio content format of the audio signal, such as at least one of a scene-based audio representation signal, a channel-based audio representation signal, and an object-based audio representation signal, depending on the audio representation of the parsed audio signal. The metadata information may be corresponding information.
[0202] In an embodiment of the present disclosure, the metadata information may include metadata directly input together with the input signal, and / or may also include related information further derived from the metadata, which may include audio parameters related to the audio signal, which can be used for further processing of the audio signal, such as spatial coding.
[0203] In some embodiments, metadata information associated with the audio signal can be obtained in various appropriate ways. In some embodiments, the metadata information can be directly input when the audio signal is input, or obtained and provided by other devices in the system or third-party devices. In one example, the metadata information can be separated from the input audio signal and input into the audio rendering system separately or together with the input audio signal. In other embodiments, the information processing module can be used to extract signals of different formats and calculate corresponding parameters. In particular, the information processing module can process the metadata associated with the audio signal to obtain appropriate information as metadata information for subsequent signal processing, including signal space encoding and decoding processing. In one implementation, for the received input audio signal, the audio signal parsing module can obtain an audio signal having an audio content format compatible with the audio rendering system, as well as metadata information related to the audio signal from the input audio signal.
[0204] In some embodiments, metadata information may be obtained through audio information processing, which may be performed by an audio information processing module.
[0205] Audio information processing
[0206] According to some embodiments of the present disclosure, an audio rendering system may include an audio information processing module configured to obtain audio parameters of an audio signal of a specific type based on the audio content format of the audio signal of the specific type, particularly based on the audio content format of an audio signal obtained from an input audio signal in an audio content format compatible with the audio rendering system. For example, the audio parameters may be parameters of specific types corresponding to the audio content formats, as described above. In some embodiments, for example, the audio information processing module may be configured to obtain audio parameters of an audio signal of a specific audio content format based on metadata associated with the audio signal of the specific audio content format, particularly based on metadata associated with the audio signal of the specific audio content format as metadata information that can be used for encoding. In particular, in some embodiments, the audio information processing module may be configured to perform information extraction of the audio signal based on the audio content format of the audio signal obtained from the input audio signal in an audio content format compatible with the audio rendering system. The audio information processing module may be included in the audio signal acquisition module and may receive metadata output by the pre-processing module and perform processing based thereon.
[0207] According to some embodiments of the present disclosure, the audio signal is an object-based audio representation signal, and the audio information processing module is configured to obtain spatial position information of the object-based audio representation signal as an audio parameter that can be used for spatial audio encoding processing, for example, obtaining the spatial position information of the audio signal from metadata information associated with the object-based audio representation signal. In some embodiments, the spatial position information of the audio signal includes position information of each audio element in a coordinate system, or relative position information of a sound source associated with the audio signal relative to a listener. In some embodiments, the spatial position information of the audio signal further includes distance information of each sound element of the audio signal in a coordinate system. As an example, in metadata processing of object-based audio representation, position information of each sound element in a coordinate system, such as azimuth and elevation, can be obtained, and optionally distance information can also be obtained, or relative position information of each sound source relative to the listener's head can be obtained.
[0208] According to some embodiments of the present disclosure, an audio signal is a scene-based audio representation signal, and an audio information processing module is configured to obtain rotation information related to the audio signal from metadata associated with the audio signal for use in spatial audio coding processing. In some embodiments, the rotation information related to the audio signal includes at least one of rotation information of the audio signal and rotation information of a listener of the audio signal. As an example, in metadata processing of a scene-based audio representation, scene audio rotation information and listener rotation information are read from the metadata.
[0209] According to some embodiments of the present disclosure, the audio signal is a channel-based audio signal, and the audio information processing module is configured to obtain audio parameters based on the channel track type of the audio signal. In particular, the audio encoding process will mainly target a specific type of channel-based audio signal that requires spatial encoding, especially a narrative channel track of a channel-based audio signal, and the audio information processing module may be configured to perform at least one of the following: splitting the audio representation of the channel into audio elements by channel to convert into metadata as audio parameters, or directly outputting the audio signal without performing spatial audio processing on the audio signal. In yet another embodiment, for non-narrative channel tracks, the audio information processing module is configured to directly output the audio signal without performing spatial audio processing on the audio signal. It should be noted that, for example, a narrative channel track of a channel-based audio signal may not perform spatial audio encoding depending on the specific application scenario. Such a track may be directly passed to the decoding stage or further processed depending on the playback method.
[0210] As an example, in metadata processing based on channel-based audio representation, for narrative channel audio tracks, the audio representation of the channel can be split into audio elements according to the standard definition of the channel, and converted into metadata for processing. Depending on the needs of the application scenario, spatial audio processing can be omitted, and mixing can be performed for different playback modes in subsequent links. For non-narrative channel audio tracks, since dynamic spatialization processing is not required, mixing can be performed for different playback modes in subsequent links. In other words, non-narrative channel audio tracks will not be processed by the audio information processing module, that is, spatial audio processing will not be performed, but can bypass the audio information processing module and be directly transmitted / transparently transmitted.
[0211] According to an embodiment of the present disclosure, the audio information processing module may be referred to as a scene information processing module / processor, and the audio parameters acquired by the audio information processing module may be input into an audio signal encoding module, thereby further configuring the audio signal encoding module to spatially encode the audio signal in the specific audio content format based on the audio parameters. Here, the audio signal in the specific audio content format may include an audio signal in an audio content format compatible with the audio rendering system obtained from the input audio signal, such as at least one of the aforementioned scene-based audio representation signal, object-based audio representation signal, and channel-based audio representation signal, and more particularly, at least one of a specific type of channel signal among the object-based audio representation signal, scene-based audio representation signal, and channel-based audio representation signal. As an example, this specific type of channel signal may be referred to as a first specific type of channel signal, which may include a non-narrative channel / track in the channel-based audio representation signal. In another example, this specific type of channel signal may also include a narrative channel / track that does not require spatial encoding depending on the application scenario.
[0212] Audio signal processing
[0213] For the acquired audio signal, such as the input audio signal or the audio signal with a specific spatial format, further processing, such as spatial audio processing, may be performed on the audio signal to obtain an audio signal with a specific spatial audio exchange format.
[0214] In some embodiments, the audio signal spatial encoder may perform appropriate encoding processing on the non-direct / transparent audio signal to obtain an encoded audio signal in a specific spatial format that can be used for playback in a user application scenario. The encoded audio signal in the specific spatial format may be the audio signal in the specific spatial format described above and will not be described in detail here. The encoded audio signal can be added to the intermediate signal or directly / transparently transmitted to the spatial decoder. As an example, in the spatial encoding module, all audio signals except the non-narrative channels are converted into FOA, HOA or MOA signals through spatial encoding as intermediate signals.
[0215] In some embodiments, the audio signal spatial encoder can appropriately encode audio signals in different audio content formats. The audio signals in different audio content formats can be directly input, or can be extracted / parsed from the input audio signal, such as obtained by the pre-processing module described above.
[0216] As an example, if the input audio signal includes an audio signal in a spatial audio exchange format that is distributed to the audio rendering system, spatial audio processing can be performed on the input audio signal to obtain an audio signal in a specific spatial format. Specifically, the spatial audio exchange format can be any known suitable format for audio signals in signal transmission, such as the audio distribution format described above for audio signal distribution, and will not be described in detail here. As another example, spatial audio processing can include at least one of parsing, format conversion, information processing, and encoding of the input audio signal. Specifically, audio parsing can be used to derive audio signals in various audio content formats from the input audio signal. The parsed signal can then be encoded to obtain an audio signal in a spatial format suitable for rendering in a user application scenario, i.e., a playback environment, for playback. Furthermore, format conversion and signal information processing can optionally be performed prior to encoding. Thus, an audio signal in a specific spatial audio representation can be derived from the input audio signal, and an audio signal in the specific spatial format can be obtained based on the audio signal in the specific spatial audio representation.
[0217] As an example, further, for the obtained audio signal of the specific audio representation, audio processing is performed based on the audio representation of the audio signal. Specifically, spatial audio encoding is performed on at least one of the narrative channels in the scene-based audio representation signal, the object-based audio representation signal, and the channel-based audio representation signal to obtain an audio signal with a specific spatial format. That is, although the format / representation of the input audio signal may be different, the input audio signal can still be converted into a common audio signal with a specific spatial format for decoding and rendering. The spatial audio encoding process can be performed based on metadata-related information associated with the audio signal. The metadata-related information here can include metadata of the audio signal directly obtained, such as derived from the input audio signal during the parsing process, and / or optionally, can also include audio parameters corresponding to the spatial audio signal obtained by information processing of the metadata information of each signal obtained, and the spatial audio encoding process can be performed based on the audio parameters.
[0218] In some embodiments, the audio signal of the specific audio content format is an object-based audio representation signal, and the audio signal spatial encoder is configured to spatially encode the object-based audio signal based on information related to the spatial propagation path from the sound object to the listener of the audio signal. In some embodiments, the information related to the spatial propagation path from the sound object to the listener includes at least one of the propagation duration, propagation distance, position information, path intensity energy, and nodes along the way of each spatial propagation path from the sound object to the listener. In some embodiments, the audio signal spatial encoder is configured to spatially encode the audio signal for each spatial propagation path of each sound object according to a filter function that filters the audio signal based on the path energy intensity of the spatial propagation path and a spherical harmonic function based on the position angle of the spatial propagation path. In some embodiments, the audio signal spatial encoder is configured to spatially encode the audio signal for each spatial propagation path of each sound object by calculating the product of the filter function that filters the audio signal based on the path energy intensity of the spatial propagation path and the spherical harmonic function based on the position angle of the spatial propagation path. In some embodiments, for each spatial propagation path of each sound object, the filter function for filtering the audio signal based on the path energy intensity is a filter function for filtering the audio signal of the sound object before it propagates along the spatial propagation path based on the path intensity energy of the path. In some embodiments, the audio signal of the sound object before it propagates along the spatial propagation path refers to the audio signal at a moment before the time required for the sound object to reach the listener along the spatial propagation path. In some embodiments, the position angle of the spatial propagation path includes the direction angle of the spatial propagation path to the listener or the direction angle of the spatial propagation path relative to a coordinate system. In some embodiments, the audio signal spatial encoder is configured to further encode each spatial propagation path of each sound object using at least one of a near-field compensation function and a spread function based on the length of the spatial propagation path. For example, a near-field compensation function and / or a spread function may be further applied to the encoded HOA signal. Specifically, if the path distance is less than a threshold, the near-field compensation function may be applied, and if it is greater than the threshold, the spread function may be applied, and vice versa, to further optimize the encoded HOA signal. In some embodiments, the audio signal spatial encoder is further configured to perform weighted superposition on the encoded signals of the object-based audio representation signals based on the weights of the sound objects defined in the metadata.
[0219] As an example, in the spatial encoding process of object-based audio representation, for each audio object, the audio signal will be written into a delay device taking into account the delay of sound propagation in space. According to the results obtained by the audio information processing module, each sound object will have one or more propagation paths to the listener. Based on the length of each path, the time t1 required for the sound object to reach the listener is calculated. The spatial encoding starts by obtaining the audio signal s of the sound object before time t1 from the delay device of the audio object and filtering it according to the path energy intensity E and the direction θ of the path to the listener (optionally, the direction of the path relative to the coordinate system can be used instead of the direction to the listener, so that the target sound field signal can be obtained by multiplying it with the rotation matrix in subsequent steps). The signal is spatially encoded and converted into an HOA signal s_N, where N is the number of channels of the HOA signal. A typical encoding method is as follows, where Y_N is the spherical harmonics of the corresponding channel.
[0220] s N =E(s(t-t1))Y N (θ)
[0221] The encoding operation can be performed in the time domain or frequency domain, and near-field compensation and source spread functions can be added based on the path distance to enhance the effect. Finally, the HOA signals obtained after each signal conversion are weighted and superimposed according to the weights of the sound objects defined in the metadata. The weighted sum of all object-based audio signals is obtained as the encoded signal, which can be used as an intermediate signal.
[0222] In some embodiments, the spatial encoding of the audio signal may also generate a reverberation-related signal to be included in the output encoded signal. In particular, the reverberation-related signal may be directly passed to a spatial decoder for decoding, or may be added to an intermediate signal included in the encoder output.
[0223] In some embodiments, the audio signal in the specific audio content format is an object-based audio representation signal, and the audio signal spatial encoder is further configured to obtain a reverberation-related signal of the object-based audio signal. In some embodiments, the audio signal spatial encoder is further configured to obtain a spatial reverberation response of the scene and convolve the audio signal based on the spatial reverberation response to obtain a reverberation-related signal of the audio signal. In some audio signal spatial encoders, the encoder is further configured to obtain reverberation parameter information and perform reverberation processing on the audio signal to obtain a reverberation-related signal of the audio signal. As an example, for more advanced scene information processors, the spatial room reverberation response of the scene may be generated, including but not limited to RIR (Room Impulse Response), ARIR (Ambisonics Room Impulse Response), BRIR (Binaural Room Impulse Response), and MO-BRIR (Multi-Orientation Binaural Room Impulse Response). For this type of information, a convolver may be added to this module to obtain a processed signal. Depending on the reverberation type, the generated result may be an intermediate signal (ARIR), an omnidirectional signal (RIR), or a binaural signal (BRIR, MO-BRIR). The output is added to the intermediate signal or transparently transmitted to a subsequent step for corresponding playback decoding. Optionally, the scene information processor may also provide reverberation parameter information such as reverberation duration. An artificial reverberation generator (such as a feedback delay network) can be added to this module to perform artificial reverberation processing. The result is output to the intermediate signal or transparently transmitted to the decoder for processing.
[0224] In some embodiments, the audio signal of a specific audio content format is a scene-based audio representation signal, and the audio signal spatial encoder is further configured to weight the scene-based audio representation signal based on the weight information in the metadata. In this way, the weighted signal can be used as an encoded audio signal for spatial decoding. In some embodiments, the audio signal of a specific audio content format is a scene-based audio representation signal, and the audio signal spatial encoder is further configured to perform a sound field rotation operation on the scene-based audio representation signal based on the rotation information indicated in the metadata. In this way, the rotated audio signal can be used as an encoded audio signal for spatial decoding. As an example, for a scene audio signal, it is itself an FOA, HOA or mix order Ambisonics signal, which can be weighted according to the weight information in the metadata, that is, the intermediate signal that is desired to be obtained. In addition, if the metadata indicates that the sound field needs to be rotated, the sound field rotation operation can be performed at this step or at spatial decoding (next step) depending on different implementations.
[0225] In some embodiments, the audio signal of the specific audio content format is a channel-based audio representation signal, and the audio signal spatial encoder is further configured to convert the channel-based audio representation signal into an object-based audio representation signal and encode it when the channel-based audio representation signal needs to be converted. The encoding operation here can be performed in the same manner as described above for encoding the object-based audio representation signal. As an example, in the spatial encoding process of channel-based audio representation, channel-based audio processing is performed according to the agreement in the previous step of scene information processing. If conversion to object-based audio is required, it is performed in the object-based audio processing manner. If spatial audio processing is not required, the intermediate signal is not added and it is directly transmitted to the next module.
[0226] Alternatively, the input audio signal may be in a suitable format other than the spatial audio exchange format, specifically, a specific spatial representation signal or even a specific spatial format signal. In this case, at least some of the aforementioned spatial audio processing can be bypassed to obtain an audio signal in the specific spatial format. In some embodiments, if the input audio signal is not a distributed audio signal in the spatial audio exchange format, but rather a directly input audio signal in the specific spatial audio representation, the aforementioned audio parsing process can be omitted, and format conversion and encoding can be performed directly. Furthermore, if the input audio signal has a predetermined format, the aforementioned format conversion can be omitted, and encoding can be performed directly. In other embodiments, if the input audio signal is directly in the specific spatial format, such an input audio signal can be directly transmitted / transparently passed to the audio signal spatial decoder without undergoing spatial audio processing, such as parsing, format conversion, information processing, encoding, etc. For example, if the input audio signal is a scene-based spatial audio representation signal, such an input audio signal can be directly transmitted to the spatial decoder as a signal in the specific spatial format, without requiring the aforementioned spatial audio processing. According to some embodiments, when the input audio signal is not a distributed audio signal having a spatial audio exchange format, for example, it may be an audio signal of the aforementioned specific spatial audio representation or an audio signal of a specific spatial format, then it may be directly input at the user end / consumer end, for example, it may be directly obtained from an application programming interface (API) directly provided in the rendering system.
[0227] For example, in the case of a signal with a specific representation directly input from the user end / consumer end, such as one of the three audio representations mentioned above, the aforementioned parsing process may not be required, and the signal may be directly converted into the format specified by the system. For another example, when the input audio signal is already in the format specified by the system and in a representation that the system can process, the signal may be directly passed to the spatial encoding processing module without the aforementioned parsing and code conversion. For another example, if the input audio signal is a non-narrative channel signal, a binaural signal after reverberation processing, etc., the input audio signal may be directly transmitted to the spatial decoding module for decoding without performing the aforementioned spatial audio encoding process. In this case, a judgment unit / module may be present in the system to determine whether the input audio signal meets the above conditions.
[0228] In some embodiments, when the input audio signal is not a signal in the distributed spatial audio exchange format, it may not be necessary to perform at least some of the spatial audio processing on the input audio signal. As an example, the input specific audio signal can directly be at least one of the three signal representations described above, thereby eliminating the aforementioned signal parsing processing, and the audio signal and its associated metadata can be directly passed to the audio signal encoding module. In other embodiments, the input audio signal can even be an audio signal in the specific spatial format described above. Such an input audio signal can be directly / transparently passed to the audio signal decoding module without performing the aforementioned spatial audio processing including parsing, format conversion, audio encoding, etc.
[0229] In some embodiments, for such an input audio signal, the audio rendering system may further include a specific audio input device for directly receiving the input audio signal and transmitting it directly / transparently to the audio signal encoding module or the audio signal decoding module. It should be noted that such a specific input device may be, for example, an application programming interface (API), and the format of the input audio signal it can receive has been pre-set, for example, corresponding to the specific spatial format mentioned above, for example, it may be at least one of the three signal representation methods mentioned above, and so on, so that when the input device receives the input audio signal, the input audio signal can be directly transmitted / transparently without performing at least some of the spatial audio processing. It should be noted that such a specific input device may also be part of the audio signal acquisition operation / module, or even included in the audio signal parsing module.
[0230] It should be noted that the aforementioned implementations of the audio signal parsing module and specific audio input device are merely exemplary and non-restrictive. According to some embodiments of the present disclosure, the audio signal parsing module can be implemented in various suitable ways. In some embodiments, the audio signal parsing module may include a parsing submodule and a direct transmission submodule. The parsing submodule may only receive audio signals in a spatial audio exchange format for audio parsing, while the direct transmission submodule may receive audio signals in a specific audio content format or a specific audio representation signal for direct transmission. Thus, the audio rendering system can be configured so that the audio signal parsing module receives two inputs: an audio signal in a spatial audio exchange format and an audio signal in a specific audio content format or a specific audio representation signal. In other embodiments, the audio signal parsing module may include a determination submodule, a parsing submodule, and a direct transmission submodule. This allows the audio signal parsing module to receive any type of input signal and perform appropriate processing. The determination submodule may determine the format / type of the input audio signal and, if it determines that the input audio signal is in a spatial audio exchange format, transfer the parsing submodule to perform the aforementioned parsing operations. Otherwise, the direct transmission submodule may directly / transparently transmit the audio signal to the format conversion, audio encoding, audio decoding, and other stages, as described above. Of course, the judgment submodule can also be outside the audio signal analysis module. Audio signal judgment can be implemented in various known appropriate ways, which will not be described in detail here.
[0231] Audio signal spatial decoding
[0232] Spatial decoding can then be performed on the obtained audio signal with a specific spatial format. Specifically, the obtained audio signal with a specific spatial format can be referred to as an audio signal to be decoded, and spatial decoding of the audio signal aims to convert the audio signal to be decoded into a format suitable for playback by a user application scenario, such as an audio playback environment, a playback device in an audio rendering environment, or a rendering device. According to embodiments of the present disclosure, decoding can be performed based on the audio signal playback mode. The playback mode can be indicated in various appropriate ways, such as by an identifier, and can be communicated to the decoding module in various appropriate ways, such as along with the input audio signal, or input and communicated to the decoding module by other input devices. For example, the renderer ID described above can be used as an identifier to indicate whether the playback mode is binaural playback or speaker playback, etc. In some embodiments, audio signal decoding can utilize a decoding method corresponding to the playback device in the user application scenario, particularly a decoding matrix, to decode the audio signal in the specific spatial format and convert the audio signal to be decoded into audio of a suitable format. In other embodiments, audio signal decoding can also be performed using other appropriate methods, such as virtual signal decoding.
[0233] In some embodiments, the audio signal spatial decoder uses a decoding matrix corresponding to the playback device in the user application scenario to decode the audio signal of the specific spatial format. In this way, the decoded audio signal can be suitable for a specific type of playback environment, especially for the playback device in the playback environment, so that compatibility with various types of playback environments can be achieved. Here, the audio signal of the specific spatial format can be the aforementioned intermediate signal, or it can be an audio signal directly transmitted / transparently transmitted to the spatial decoder, including an unencoded audio signal, or an encoded audio signal that has been spatially encoded but not included in the intermediate signal. In particular, after the spatial audio signal is encoded, the system obtains an intermediate medium signal that has been spatially encoded, and transparently transmitted signals such as non-narrative channel signals and binaural signals after reverberation processing. The spatial audio decoding module decodes the intermediate signal and the transparent signal onto the playback device according to the playback type of the user application scenario.
[0234] In some embodiments, the audio signal spatial decoder is configured to perform a sound field rotation operation on the audio representation signal to be decoded based on rotation information indicated in the metadata. For example, the intermediate media signal is first multiplied by a rotation matrix or rotated as needed based on the processing method of the previous module and the rotation information in the metadata, thereby decoding the rotated intermediate media signal.
[0235] In some embodiments, the decoding matrix is a built-in or externally provided decoding matrix corresponding to the speaker array in the user application scenario.
[0236] As an example, in the case of standard speaker array spatial decoding, for a standard speaker array defined in the standard, such as 5.1, the decoder will have built-in decoding matrix coefficients, and the playback signal L can be obtained by multiplying the intermediate signal with the decoding matrix. L = DS N , where L is the speaker array signal, D is the decoding matrix, S N The intermediate signal is also converted to an intermediate signal based on the definition of standard speakers. Specific methods include VBAP. For spatial decoding of special speaker arrays, such as sound bars or other more specialized speaker arrays, the speaker manufacturer must provide a correspondingly designed decoding matrix. The system provides a decoding matrix setup interface, which allows decoding processing by specifying the decoding matrix.
[0237] In some embodiments, when the speaker array is a custom speaker array, the decoding matrix is a decoding matrix calculated according to the arrangement of the custom speaker array. In some embodiments, the decoding matrix is calculated based on the position angle and pitch angle of each speaker in the speaker array or the three-dimensional coordinate values of the speaker. As an example, in the custom speaker array spatial decoding, in the scenario of the custom speaker array, such speakers usually have a spherical, hemispherical design or a rectangle, which can surround or semi-surround the listener. The decoding module can calculate the decoding matrix according to the arrangement of the custom speakers, and the required input is the position angle and pitch angle of each speaker, or the three-dimensional coordinate values of the speaker. The calculation methods of the speaker decoding matrix can include SAD (Sampling Ambisonic Decoder), MMD (Mode Matching Decoder), EPAD (Energy Preserved Ambisonic Decoder), AllRAD (All Round Ambisonic Decoder), etc.
[0238] In some embodiments, when the user application environment is a headphone playback environment, the audio signal spatial decoder is configured to directly decode the signal into a binaural signal, or perform speaker virtualization to obtain a decoded signal. As an example, for the headphone playback environment, there are several optional decoding methods. The first is to directly decode the intermediate medium signal into a binaural signal. Typical methods include LS (least squares), Magnitude LS, SPR (Spatial resampling), etc. For the transparent signal, it is usually a binaural signal, which is played back directly. Indirect rendering can also be performed, that is, first using a speaker array, and then performing HRTF convolution according to the position of the speaker to virtualize the speaker.
[0239] According to some embodiments of the present disclosure, optionally, after the audio signal is decoded, the decoded output may be post-processed, in particular, signal adjustment may be performed to adjust the spatially decoded audio signal for a specific playback device in a user application scenario, especially to adjust the audio signal characteristics, so that the adjusted audio signal can present a more appropriate acoustic experience when rendered by an audio rendering device.
[0240] Audio signal post-processing
[0241] According to an embodiment of the present disclosure, optionally or additionally, the spatially decoded audio signal can be adjusted on a specific playback device used for a user application scenario, aiming to enable the adjusted audio signal to present a more appropriate acoustic experience when rendered by an audio rendering device. In particular, the audio signal adjustment may be mainly intended to eliminate the inconsistencies that may exist between different playback types, or different playback modes, etc., so that the adjusted audio signal can maintain a consistent playback experience when played back in the application scenario, thereby improving the user experience. In the context of the present disclosure, audio signal adjustment processing may be referred to as a post-processing, which refers to post-processing of the output signal obtained by audio decoding, which may be referred to as output signal post-processing. In some embodiments, the signal post-processing module is configured to perform at least one of frequency response compensation and dynamic control range on the decoded audio signal for a specific playback device.
[0242] In some embodiments, the audio processing system may further include a signal post-processing module, which may be configured to perform at least one of frequency response compensation and dynamic range control on the decoded audio signal for a specific playback device. For example, the post-processing module accounts for inconsistencies in playback modes, such as varying frequency response curves and gain across different playback devices, and performs post-processing adjustments on the output signal to provide a consistent acoustic experience. Post-processing operations include, but are not limited to, device-specific frequency response compensation (EQ) and dynamic range control (DRC).
[0243] In the audio rendering system disclosed in the present invention, the audio information processing module, audio signal spatial encoder, audio signal spatial decoder and output signal post-processing mentioned above can constitute the core rendering module of the system, which is responsible for processing the signals in the three audio representation formats and their metadata obtained after pre-processing and playing them back through the playback device in the user application environment.
[0244] It should be noted that the processing of audio data and / or metadata in the above-mentioned rendering process can be performed in various appropriate formats. According to some embodiments, audio signal processing can be performed in blocks, and the block size can be set. For example, the block size can be pre-set and not changed during the processing. For example, the block size can be set when the audio rendering system is initialized. In some embodiments, metadata can be parsed in blocks and then the scene information can be adjusted based on the metadata. This operation can be included in the operation of the scene information processing module according to an embodiment of the present disclosure.
[0245] FIG5 shows a block diagram of some embodiments of an electronic device of the present disclosure.
[0246] As shown in Figure 5, the electronic device 5 of this embodiment includes: a memory 51 and a processor 52 coupled to the memory 51, and the processor 52 is configured to execute the reverberation duration estimation method or audio signal rendering method in any embodiment of the present disclosure based on instructions stored in the memory 51.
[0247] The memory 51 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.
[0248] Reference is now made to FIG6 , which illustrates a schematic diagram of the structure of an electronic device suitable for implementing an embodiment of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device illustrated in FIG6 is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.
[0249] FIG6 shows a block diagram of some other embodiments of the electronic device of the present disclosure.
[0250] As shown in Figure 6, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device are also stored in RAM 603. The processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0251] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 6 illustrates an electronic device having various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may alternatively be implemented or present.
[0252] According to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.
[0253] In some embodiments, a chip is further provided, comprising: at least one processor and an interface, wherein the interface is configured to provide computer-executable instructions to the at least one processor, and the at least one processor is configured to execute the computer-executable instructions to implement the reverberation duration estimation method or audio signal rendering method of any of the above-mentioned embodiments.
[0254] FIG7 shows a block diagram of some embodiments of a chip of the present disclosure.
[0255] As shown in Figure 7, the chip's processor 70 is mounted as a coprocessor on the host CPU, which assigns tasks to it. The core of processor 70 is the arithmetic circuit. Controller 704 controls arithmetic circuit 703 to extract data from memory (weight memory or input memory) and perform calculations.
[0256] In some embodiments, the arithmetic circuit 703 includes multiple processing units (PEs). In some embodiments, the arithmetic circuit 703 is a two-dimensional systolic array. The arithmetic circuit 703 may also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some embodiments, the arithmetic circuit 703 is a general-purpose matrix processor.
[0257] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 702 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 701 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 708.
[0258] The vector calculation unit 707 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc.
[0259] In some embodiments, the vector calculation unit 707 can store the processed output vector to the unified buffer 706. For example, the vector calculation unit 707 can apply a nonlinear function to the output of the operation circuit 703, such as a vector of accumulated values, to generate an activation value. In some embodiments, the vector calculation unit 707 generates a normalized value, a merged value, or both. In some embodiments, the processed output vector can be used as an activation input to the operation circuit 703, for example, for use in a subsequent layer in a neural network.
[0260] The unified memory 706 is used to store input data and output data.
[0261] The memory unit access controller 705 (Direct Memory Access Controller, DMAC) moves the input data in the external memory to the input memory 701 and / or the unified memory 706, stores the weight data in the external memory into the weight memory 702, and stores the data in the unified memory 706 into the external memory.
[0262] The bus interface unit (BIU) 510 is used to implement interaction between the main CPU, DMAC and instruction fetch memory 709 through the bus.
[0263] An instruction fetch buffer 709 connected to the controller 704 and used to store instructions used by the controller 704;
[0264] The controller 704 is used to call the instructions cached in the memory 709 to control the working process of the computing accelerator.
[0265] Generally, the unified memory 706, the input memory 701, the weight memory 702 and the instruction fetch memory 709 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.
[0266] In some embodiments, a computer program is further provided, comprising: instructions, which, when executed by a processor, cause the processor to execute the audio signal rendering method, audio signal encoding method, etc. of any of the above embodiments.
[0267] Those skilled in the art will appreciate that the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. A computer program product includes one or more computer instructions or computer programs. When a computer instruction or computer program is loaded or executed on a computer, a process or function according to an embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0268] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art will appreciate that the above examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Those skilled in the art will appreciate that modifications may be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.
Claims
1. An audio signal encoding device for audio rendering, in, The audio signal encoding device is configured to: Acquiring metadata related information associated with the audio signal, wherein the metadata related information includes spatial position information associated with the audio signal; and Based on metadata related information associated with the audio signal, the audio signal is spatially encoded to obtain an encoded audio signal.
2. The audio signal encoding device according to claim 1, in, The method of spatially encoding an audio signal to obtain an encoded audio signal comprises: Determining encoding parameters based on the metadata related information; and The audio signal is encoded into an encoded audio signal in a specific spatial format based on the encoding coefficients.
3. The audio signal encoding device according to claim 1 or 2, in, The audio signal is an audio signal in a specific audio content format, including at least one of an object-based audio representation signal, a scene-based audio representation signal, and a channel-based audio representation signal, and / or The encoded audio signal in the specific spatial format is an Ambisonics type audio signal, which can include at least one of FOA (First Order Ambisonics), HOA (Higher Order Ambisonics), and MOA (Mixed-order Ambisonics).
4. The audio signal encoding device according to any one of claims 1 to 4, in, The metadata-related information associated with the audio signal includes at least one of metadata associated with the audio signal and an audio signal-related parameter obtained based on the metadata.
5. The audio signal encoding device according to claim 1, in, The spatial position information related to the audio signal includes distance information of a sound source of the audio signal, and determining the encoding parameter based on the metadata related information includes: A distance-dependent coding coefficient matrix is constructed based on the distance information of the sound source of the audio signal.
6. The audio signal encoding device according to claim 1, in, Determining the encoding parameters based on the metadata related information further includes: determining the encoding parameters based on a combination of the spatial position information of the sound source and the reference spatial position information.
7. The audio signal encoding device according to claim 6, in, The spatial position information of the sound source includes the distance of the sound source, the reference spatial position includes the reference distance, the combination of the spatial position information of the sound source and the reference spatial position includes the ratio relationship between the distance of the sound source and the reference distance, and, The method of determining the encoding parameters based on the combination of the spatial position information of the sound source and the specific reference spatial position further includes: A distance-dependent coding coefficient matrix is constructed based on the ratio of the distance of the sound source to the reference distance.
8. The audio signal encoding device according to claim 6, in, in, The reference spatial position is a spatial position of an audio rendering device, where the audio rendering device includes at least one of a speaker on an audio rendering side and a virtual speaker.
9. The audio signal encoding device according to claim 1, in, The metadata related information also includes directional information of the sound source of the audio signal, and The encoding parameters are determined based on the metadata related information, and further include: determining a coding matrix based on the directional information of the sound source of the audio signal, and The method of encoding the audio signal into an encoded audio signal in a specific spatial format based on the encoding parameter further includes: The audio signal is encoded by multiplying or convolving it with the encoding matrix.
10. The audio signal encoding device according to claim 1, in, Audio signal encoding is performed by solving the acoustic equations constructed for the audio signal, and The encoding parameters are determined based on the metadata related information, further comprising: The encoding parameter is determined by solving a sub-equation related to the distance information of the sound source which is included in or can be derived from the acoustic equation based on a specific function.
11. The audio signal encoding device according to claim 10, in, For solving the acoustic equation, the solution of the sub-equation related to the distance information of the sound source is expressed in a standing wave form or a traveling wave form based on a specific function.
12. The audio signal encoding device according to claim 10, in, The specific function includes a spherical harmonic function, and the spherical harmonic function includes at least one of a Neumann function, a Hankel function, and a Bessel function.
13. The audio signal encoding device according to claim 1, in, Audio signal encoding is based on solving acoustic equations constructed for audio signals, and The method further comprises: applying a specific filter to the solution of the acoustic equation, The filtering includes at least one of HIR filtering and FIR filtering, Among them, the HIR filter is constructed by performing spectrum fitting based on frequency information in the digital domain by using the least square method; the FIR filter is obtained by performing time-frequency transformation to obtain zero and pole characteristics and then using bilinear transformation.
14. The audio signal encoding device according to claim 2, in, Determining encoding parameters based on the metadata related information further includes: A corresponding distance-dependent coding coefficient matrix is determined for channels of a coded audio signal in a specific spatial format.
15. The audio signal encoding device according to claim 14, in, The audio signal is a time-domain audio signal, which will be encoded into an Ambisonics signal, and different distance-related encoding coefficient matrices are determined based on Ambisonics channels of different orders.
16. An audio signal decoding device for audio rendering, in, The audio signal decoding device is configured to: Acquire a coded audio signal, wherein the coded audio signal comprises a coded audio signal obtained by the audio signal coding device according to any one of claims 1 to 15, and The encoded audio signal is spatially decoded to obtain a decoded audio signal for audio rendering.
17. The audio signal decoding device according to claim 16, in, Spatial decoding of the encoded audio signal comprises: Construct the decoding matrix, and A decoding matrix is applied to the encoded audio signal to spatially decode the encoded audio signal.
18. The audio signal decoding device according to claim 17, in, The decoding matrix comprises a decoding matrix predefined based on attribute information of the sound rendering device, wherein the attribute information comprises at least one of spatial orientation information or type information of the sound rendering device, and / or In the case of orthogonal type coding, the decoding matrix and the encoding matrix are transposed to each other.
19. The audio signal decoding device according to claim 16, in, The audio signal decoding device is further configured to: The encoded audio signal is preprocessed based on the listener position information, and then the preprocessed encoded audio signal is spatially decoded.
20. The audio signal decoding device according to claim 19, in, The position information of the listener includes the head direction of the listener, and The preprocessing includes rotating and / or adjusting the amplitude of the encoded signal based on the head direction of the listener.
21. An audio rendering system, include: The audio signal encoding device according to any one of claims 1 to 15, and An audio signal decoding device according to any one of claims 16 to 20.
22. A method for encoding an audio signal for audio rendering, include: Acquire metadata related information associated with the audio signal, where the metadata related information includes spatial position information associated with the audio signal; as well as Based on metadata related information associated with the audio signal, the audio signal is spatially encoded to obtain an encoded audio signal.
23. The audio signal encoding method according to claim 22, in, The method of spatially encoding an audio signal to obtain an encoded audio signal comprises: Determining encoding parameters based on the metadata related information; and The audio signal is encoded into an encoded audio signal of a specific spatial format based on the encoding parameters.
24. The audio signal encoding method according to claim 23, in, The spatial position information related to the audio signal includes distance information of a sound source of the audio signal, and determining the encoding parameter based on the metadata related information includes: A distance-dependent coding coefficient matrix is constructed based on the distance information of the sound source of the audio signal.
25. The audio signal encoding method according to claim 23, in, Determining the encoding parameters based on the metadata related information further includes: determining the encoding parameters based on a combination of the spatial position information of the sound source and the reference spatial position information.
26. The audio signal encoding method according to claim 25, in, The spatial position information of the sound source includes the distance of the sound source, the reference spatial position includes the reference distance, the combination of the spatial position information of the sound source and the reference spatial position includes the ratio relationship between the distance of the sound source and the reference distance, and, The method of determining the encoding parameters based on the combination of the spatial position information of the sound source and the specific reference spatial position further includes: A distance-dependent coding coefficient matrix is constructed based on the ratio of the distance of the sound source to the reference distance.
27. The audio signal encoding method according to claim 23, in, The metadata related information also includes directional information of the sound source of the audio signal, and The encoding parameters are determined based on the metadata related information, and further include: determining a coding matrix based on the directional information of the sound source of the audio signal, and The method of encoding the audio signal into an encoded audio signal in a specific spatial format based on the encoding parameter further includes: The audio signal is encoded by multiplying or convolving it with the encoding matrix.
28. The audio signal encoding method according to claim 23, in, Audio signal encoding is performed by solving the acoustic equations constructed for the audio signal, and The encoding parameters are determined based on the metadata related information, further comprising: The encoding parameter is determined by solving a sub-equation related to the distance information of the sound source which is included in or can be derived from the acoustic equation based on a specific function.
29. The audio signal encoding method according to claim 22, in, Audio signal encoding is based on solving acoustic equations constructed for audio signals, and The method further comprises: applying a specific filter to the solution of the acoustic equation, The filtering includes at least one of HIR filtering and FIR filtering, Among them, the HIR filter is constructed by performing spectrum fitting based on frequency information in the digital domain by using the least square method; the FIR filter is obtained by performing time-frequency transformation to obtain zero and pole characteristics and then using bilinear transformation.
30. The audio signal encoding method according to claim 23, in, Determining encoding parameters based on the metadata related information further includes: A corresponding distance-dependent coding coefficient matrix is determined for channels of a coded audio signal in a specific spatial format.
31. A method for decoding an audio signal for audio rendering, include: Acquire a coded audio signal, wherein the coded audio signal comprises a coded audio signal obtained by the audio signal coding device according to any one of claims 1 to 15, and The encoded audio signal is spatially decoded to obtain a decoded audio signal for audio rendering.
32. The audio signal decoding method according to claim 31, in, Spatial decoding of the encoded audio signal comprises: Construct the decoding matrix, and A decoding matrix is applied to the encoded audio signal to spatially decode the encoded audio signal.
33. The audio signal decoding method according to claim 31, in, The method further comprises: The encoded audio signal is preprocessed based on the listener position information, and then the preprocessed encoded audio signal is spatially decoded.
34. A chip, include: At least one processor and an interface, wherein the interface is used to provide computer-executable instructions to the at least one processor, and the at least one processor is used to execute the computer-executable instructions to implement the method according to any one of claims 22-33.
35. An electronic device, include: Memory; and A processor coupled to the memory, the processor being configured to execute the method according to any one of claims 22-33 based on instructions stored in the memory device.
36. A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 22 to 33.
37. A computer program product comprising instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 22-33.
Citation Information
Patent Citations
Apparatus and method for encoding a spatial audio representation or apparatus and method for decoding an encoded audio signal using transport metadata and related computer programs
CN113490980A
Audio renderer gain calculation method and device, equipment and storage medium
CN115348530A
Binaural audio rendering of spatial audio
GB202218103D0
Audio rendering system and method, and electronic device
WO2022262750A1
Audio rendering system and method and electronic device
WO2022262758A1
