Method and apparatus for determining a virtual speaker set - Patents.com

Abstract

The present application provides a method and apparatus for determining a virtual speaker set. The method for determining a virtual speaker set includes: determining a target virtual speaker from F preset virtual speakers based on an audio signal to be processed, where each of the F virtual speakers corresponds to S virtual speakers, F is a positive integer, and S is a positive integer greater than 1; and obtaining position information of each of the S virtual speakers corresponding to the target virtual speaker from a preset virtual speaker distribution table, where the virtual speaker distribution table includes position information of K virtual speakers, the position information includes an elevation scale and an azimuth scale, K is a positive integer greater than 1, F≦K, and F×S≧K. The present application can improve the audio signal reproduction effect.

Description

[Technical field]

[0001] The present application relates to the field of audio technology, and in particular to a method and apparatus for determining a virtual speaker set. [Background technology]

[0002] This application claims priority to Chinese Patent Application No. 202110247466.1, entitled "METHOD AND APPARATUS FOR DETERMINING VIRTUAL SPEAKER SET," filed with the State Intellectual Property Office of the People's Republic of China on March 5, 2021, which is hereby incorporated by reference in its entirety.

[0003] Three-dimensional audio technology is an audio technology in which sound events in the real world and three-dimensional sound field information are acquired, processed, transmitted, rendered, and played back through a computer by signal processing and the like. Three-dimensional audio technology gives sound a strong sense of space, surroundings, and immersion, giving people a "virtual face-to-face" acoustic experience. At present, the mainstream three-dimensional audio technology is higher order ambisonics (HOA) technology. Due to the characteristics of HOA technology not being related to speaker placement during the playback stage in recording and encoding, and the characteristics of data rotatability of the HOA format, HOA technology has a high degree of freedom in three-dimensional audio playback, and therefore has attracted more attention and been studied more widely.

[0004] In HOA technology, the HOA signal is converted into a virtual speaker signal, and then the binaural signal for playback can be obtained by mapping. In the above process, the uniform distribution of the virtual speakers can achieve the best sampling effect. For example, the virtual speakers are distributed on the vertices of a regular tetrahedron. However, in three-dimensional space, there are only five kinds of regular polyhedrons, namely, a regular tetrahedron, a regular cube, a regular octahedron, a regular dodecahedron, and a regular icosahedron. Therefore, the number of virtual speakers that can be arranged is limited, which is not applicable to the distribution of a larger amount of virtual speakers. Summary of the Invention

[0005] The present application provides a method and apparatus for determining a virtual speaker set to improve the audio signal reproduction effect.

[0006] According to a first aspect, the present application provides a method for determining a virtual speaker set, the method including: determining a target virtual speaker from F preset virtual speakers based on an audio signal to be processed, where each of the F virtual speakers corresponds to S virtual speakers, where F is a positive integer and S is a positive integer greater than 1; and obtaining position information of each of the S virtual speakers corresponding to the target virtual speaker from a preset virtual speaker distribution table, where the virtual speaker distribution table includes position information of K virtual speakers, the position information includes an elevation scale and an azimuth scale, K is a positive integer greater than 1, F≦K, and F×S≧K.

[0007] In the present application, a virtual speaker distribution table is preset, so that by deploying the virtual speakers according to the distribution table, a high average value of the signal-to-noise ratio (SNR) of the HOA restored signal can be obtained, and based on such distribution, S virtual speakers that have the highest correlation with the HOA coefficients of the audio signal to be processed are selected, thereby achieving the optimal sampling effect and improving the audio signal reproduction effect.

[0008] In a possible implementation, determining a target virtual speaker from the F preset virtual speakers based on the audio signal to be processed includes obtaining higher order Ambisonics HOA coefficients of the audio signal, obtaining F groups of HOA coefficients corresponding to the F virtual speakers, where the F virtual speakers are in one-to-one correspondence with the F groups of HOA coefficients, and determining as the target virtual speaker a virtual speaker corresponding to the group of HOA coefficients that has a maximum correlation with the HOA coefficients of the audio signal and is among the F groups of HOA coefficients.

[0009] A coding analysis is performed on the audio signal to be processed. For example, the sound field distribution of the audio signal to be processed is analyzed, including the characteristics such as the number, directivity, and dispersion of the sound source of the audio signal, to obtain the HOA coefficient of the audio signal, and the HOA coefficient of the audio signal is used as one of the decision conditions for determining how to select the target virtual speaker. A virtual speaker that matches the audio signal to be processed can be selected based on the HOA coefficient of the audio signal to be processed and the HOA coefficient of the candidate virtual speakers (i.e., the above-mentioned F virtual speakers). In this application, the virtual speaker is called a target virtual speaker. A dot product may be performed between the HOA coefficients of the F virtual speakers and the HOA coefficient of the audio signal separately, and the virtual speaker with the maximum absolute value of the dot product is selected as the target virtual speaker. It should be noted that the target virtual speaker may alternatively be determined using another method, which is not particularly limited in this application.

[0010] In a possible implementation, the S virtual speakers corresponding to the target virtual speaker satisfy the following condition: the S virtual speakers include the target virtual speaker and (S-1) virtual speakers placed around the target virtual speaker, and any one of the (S-1) correlations between the (S-1) virtual speakers and the target virtual speaker is greater than each of the (K-S) correlations between the target virtual speaker and (K-S) virtual speakers other than the S virtual speakers among the K virtual speakers.

[0011] When the target virtual speaker is determined, the target virtual speaker is the center virtual speaker that is most highly correlated with the HOA coefficient of the audio signal to be processed. The S virtual speakers corresponding to each center virtual speaker are the S virtual speakers that are most highly correlated with the HOA coefficient of the center virtual speaker. Therefore, the S virtual speakers corresponding to the target virtual speaker are also the S virtual speakers that are most highly correlated with the HOA coefficient of the audio signal to be processed.

[0012] In a possible implementation, the K virtual speakers satisfy the following condition: the K virtual speakers are distributed on a preset sphere, the preset sphere includes L latitudinal regions, where L>1, and the m-th latitudinal region among the L latitudinal regions is T m It contains latitude circles, is in K virtual speakers, and is i The difference in azimuth angle between adjacent virtual speakers distributed on the th latitude circle is α m , 1≦m≦L, and T m is a positive integer, 1≦m i ≦T m and T m >1, the elevation angle difference between any two adjacent latitude circles in the m-th latitude region is α m is satisfied.

[0013] In a possible implementation, the nth latitude region of the L latitude regions is T n It contains latitude circles, is in K virtual speakers, and has n iThe difference in azimuth angle between adjacent virtual speakers distributed on the th latitude circle is α n where 1≦n≦L and T n is a positive integer, 1≦n i ≦T n and T n >1, the elevation angle difference between any two adjacent latitude circles in the nth latitude region is α n where α n = α m or α n ≠α m and n ≠ m.

[0014] In a possible implementation, the cth latitude region among the L latitude regions is T c Contains latitude circles, T c One of the latitude circles is an equatorial latitude circle, which is in the K virtual speakers and is located at c i The difference in azimuth angle between adjacent virtual speakers distributed on the th latitude circle is α c , 1≦c≦L, and T c is a positive integer, 1≦c i ≦T c and T c >1, the elevation angle difference between any two adjacent latitude circles in the cth latitude region is α c where α c <α m and c ≠ m.

[0015] In a possible implementation, F virtual speakers are arranged such that m i The azimuth angle difference α between adjacent virtual speakers distributed on the latitudinal circle and among the F virtual speakers mi is α m Greater than.

[0016] In a possible embodiment, α mi =q×α m where q is a positive integer greater than 1.

[0017] In a possible implementation, the correlation R between the kth virtual speaker of the K virtual speakers and the target virtual speaker isfk satisfies the following formula: R fk =B f (θ,φ) B k (θ,φ), where θ represents the azimuth angle of the target virtual speaker, φ represents the elevation angle of the target virtual speaker, and B f (θ,φ) represents the HOA coefficients of the target virtual speaker, and B k (θ,φ) represents the HOA coefficient of the k-th virtual speaker among the K virtual speakers.

[0018] According to a second aspect, the present application provides an apparatus for determining a virtual speaker set, the apparatus including: a determination module configured to determine a target virtual speaker from F preset virtual speakers based on an audio signal to be processed, each of the F virtual speakers corresponding to S virtual speakers, F being a positive integer and S being a positive integer greater than 1; and an acquisition module configured to acquire position information of each of the S virtual speakers corresponding to the target virtual speaker from a preset virtual speaker distribution table, the virtual speaker distribution table including position information of the K virtual speakers, the position information including an elevation scale and an azimuth scale, K being a positive integer greater than 1, F≦K, and F×S≧K.

[0019] In a possible implementation, the determination module is particularly configured to obtain higher-order Ambisonics HOA coefficients of the audio signal and to obtain F groups of HOA coefficients corresponding to F virtual speakers, the F virtual speakers being in one-to-one correspondence with the F groups of HOA coefficients, and the determination module is further configured to determine as a target virtual speaker a virtual speaker corresponding to the group of HOA coefficients that has a maximum correlation with the HOA coefficients of the audio signal and that is among the F groups of HOA coefficients.

[0020] In a possible implementation, the S virtual speakers corresponding to the target virtual speaker satisfy the following condition: the S virtual speakers include the target virtual speaker and (S-1) virtual speakers placed around the target virtual speaker, and any one of the (S-1) correlations between the (S-1) virtual speakers and the target virtual speaker is greater than each of the (K-S) correlations between the target virtual speaker and (K-S) virtual speakers other than the S virtual speakers among the K virtual speakers.

[0021] In a possible implementation, the K virtual speakers satisfy the following condition: the K virtual speakers are distributed on a preset sphere, the preset sphere includes L latitudinal regions, where L>1, and the m-th latitudinal region among the L latitudinal regions is T m It contains latitude circles, is in K virtual speakers, and is i The difference in azimuth angle between adjacent virtual speakers distributed on the th latitude circle is α m , 1≦m≦L, and T m is a positive integer, 1≦m i ≦T m where T m >1, the elevation angle difference between any two adjacent latitude circles in the m-th latitude region is α m is satisfied.

[0022] In a possible implementation, the nth latitude region of the L latitude regions is T n It contains latitude circles, is in K virtual speakers, and has n i The difference in azimuth angle between adjacent virtual speakers distributed on the th latitude circle is α n where 1≦n≦L and T n is a positive integer, 1≦n i ≦T n and T n >1, the elevation angle difference between any two adjacent latitude circles in the nth latitude region is α n where α n = α m or α n ≠α mand n ≠ m.

[0023] In a possible implementation, the cth latitude region among the L latitude regions is T c Contains latitude circles, T c One of the latitude circles is an equatorial latitude circle, which is in the K virtual speakers and is located at c i The difference in azimuth angle between adjacent virtual speakers distributed on the th latitude circle is α c , 1≦c≦L, and T c is a positive integer, 1≦c i ≦T c where T c >1, the elevation angle difference between any two adjacent latitude circles in the cth latitude region is α c where α c <α m and c ≠ m.

[0024] The F virtual speakers are arranged as follows: i The azimuth angle difference α between adjacent virtual speakers distributed on the latitudinal circle and among the F virtual speakers mi is α m Greater than, satisfies.

[0025] In a possible implementation, α mi =q×α m where q is a positive integer greater than 1.

[0026] In a possible implementation, the correlation R between the kth virtual speaker of the K virtual speakers and the target virtual speaker is fk satisfies the following formula: R fk =B f (θ,φ) B k (θ,φ), where θ represents the azimuth angle of the target virtual speaker, φ represents the elevation angle of the target virtual speaker, and B f (θ,φ) represents the HOA coefficients of the target virtual speaker, and B k (θ,φ) represents the HOA coefficient of the k-th virtual speaker among the K virtual speakers.

[0027] According to a third aspect, the present application provides an audio processing device including one or more processors and a memory configured to store one or more programs, the one or more programs being executed by the one or more processors to cause the one or more processors to implement a method according to any possible implementation of the first aspect.

[0028] According to a fourth aspect, the present application provides a computer-readable storage medium comprising a computer program, which, when executed on a computer, enables the computer to carry out a method according to any possible implementation of the first aspect. [Brief description of the drawings]

[0029] [Figure 1] 1 is an exemplary diagram of the structure of an audio playback system according to the present application; [Diagram 2] 1 is an exemplary diagram of the structure of an audio decoding system 10 according to the present application; [Diagram 3] FIG. 2 is an exemplary diagram of the structure of an HOA encoding device according to the present application. [Figure 4a] FIG. 2 is an exemplary schematic diagram of a preset sphere according to the present application. [Figure 4b] FIG. 2 is an exemplary schematic diagram of elevation and azimuth angles according to the present application. [Figure 5a] FIG. 2 is an exemplary distribution diagram of K virtual speakers. [Figure 5b] FIG. 2 is an exemplary distribution diagram of K virtual speakers. [Figure 6a] FIG. 2 is an exemplary distribution diagram of K virtual speakers. [Figure 6b] FIG. 2 is an exemplary distribution diagram of K virtual speakers. [Figure 7] 1 is an exemplary flowchart of a method for determining a virtual speaker set according to the present application. [Figure 8]1 is an exemplary diagram of the structure of an apparatus for determining a virtual speaker set according to the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution in the present application will be described clearly and completely below with reference to the accompanying drawings of the present application. It is clear that the described embodiments are only a part, not all, of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present application.

[0031] In the specification, embodiments, claims, and accompanying drawings of this application, the terms "first", "second", etc. are merely for distinction and explanation, and should not be understood as an indication or implication of relative importance, or an indication or implication of sequence. In addition, the terms "comprise", "have", and any variations thereof are intended to cover non-exclusive inclusion, for example, including a series of steps or units. A method, system, product, or device is not necessarily limited to the literally recited steps or units, and may include other steps or units that are not literally recited or that are inherent to such a process, method, product, or device.

[0032] In this application, it is to be understood that "at least one (item)" refers to one or more, and "multiple" refers to two or more. The term "and / or" is used to describe an association relationship between related entities, and represents that three relationships may exist. For example, "A and / or B" represents three cases: only A is present, only B is present, and both A and B are present, where A and B may be singular or plural. The character " / " generally indicates an "or" relationship between related entities. "At least one of the following items" or similar expressions indicates any combination of items, including any combination of single items or multiple items. For example, at least one of a, b, c indicates a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c may be singular or plural. Two values ​​connected by the character ~ generally indicate a range of values. A range of values ​​includes the two values ​​connected by the character ~.

[0033] The relevant terms in this application are explained as follows:

[0034] Audio Frame: Audio data is in stream format. In practical applications, to facilitate audio processing and transmission, the amount of audio data in one period is usually selected as one frame of audio. The period is called "sampling period", and the value of the period can be determined based on the requirements of the codec and the requirements of a specific application. For example, the period ranges from 2.5ms to 60ms, where ms is milliseconds.

[0035] Audio Signal: An audio signal is a frequency and amplitude variation information carrier of regular sound waves, including voice, music, and sound effects. Audio is a continuously varying analog signal, which can be represented by a continuous curve and called a sound wave. A digital signal generated from audio by analog-to-digital conversion or by a computer is an audio signal. Sound waves have three important parameters, namely frequency, amplitude, and phase, which determine the characteristics of an audio signal.

[0036] Below is the system architecture to which this application applies.

[0037] Figure 1 is an exemplary diagram of the configuration of an audio playback system according to the present application. As shown in Figure 1, the audio playback system includes an audio transmitting device and an audio receiving device. The audio transmitting device includes a device that can perform audio encoding and transmit an audio bitstream, such as a mobile phone, a computer (such as a notebook computer, a desktop computer, etc.), a tablet (a portable tablet or a car-mounted tablet). The audio receiving device includes a device that can receive, decode, and play an audio bitstream, such as a true wireless stereo (TWS) earphone, a general wireless earphone, a sound box, a smart watch, or a smart glass.

[0038] A Bluetooth connection can be established between the audio transmitting device and the audio receiving device, and voice and music transmission can be supported between the audio transmitting device and the audio receiving device. Examples of widely applied audio transmitting devices and audio receiving devices are mobile phones and TWS earphones, wireless head-mounted headsets or wireless neck-ring headsets, or mobile phones and other terminal devices (such as smart sound boxes, smart watches, smart glasses, or car-mounted sound boxes). Optionally, examples of audio transmitting devices and audio receiving devices can alternatively be tablet computers, notebook computers, or desktop computers and TWS earphones, wireless head-mounted headsets, wireless neck-ring headsets, or other terminal devices (such as smart sound boxes, smart watches, smart glasses, or car-mounted sound boxes).

[0039] It should be noted that in addition to a Bluetooth connection, the audio transmitting device and the audio receiving device may be connected by another communication technique, such as a Wi-Fi connection, a wired connection, or another wireless connection, which is not specifically limited in this application.

[0040] FIG. 2 is an exemplary diagram of a structure of an audio decoding system 10 according to the present application. As shown in FIG. 2, the audio decoding system 10 may include a source device 12 and a destination device 14. The source device 12 may be the audio transmitting device of FIG. 1, and the destination device 14 may be the audio receiving device of FIG. 1. The source device 12 generates encoded bitstream information. Thus, the source device 12 may also be referred to as an audio encoding device. The destination device 14 may decode the encoded bitstream information generated by the source device 12. Thus, the destination device 14 may also be referred to as an audio decoding device. In the present application, the source device 12 and the audio encoding device may be collectively referred to as an audio transmitting device, and the destination device 14 and the audio decoding device may be collectively referred to as an audio receiving device.

[0041] Source device 12 includes an encoder 20 and may optionally include an audio source 16, an audio preprocessor 18, and a communications interface 22.

[0042] Audio source 16 may be or include any type of audio capture device, e.g., capturing real-world sounds, and / or any type of audio generation device, e.g., computer audio processor, or any type of device configured to acquire and / or provide real-world audio or computer animation audio (such as screen content or virtual reality (VR) audio), and / or any combination thereof (e.g., augmented reality (AR) audio, mixed reality (MR) audio, and / or extended reality (XR) audio). Audio source 16 may be a microphone for capturing audio or a memory for storing audio. Audio source 16 may further include any type of interface (internal or external) for storing previously captured or generated audio and / or for acquiring or receiving audio. If audio source 16 is a microphone, audio source 16 may be, for example, a local audio collector or an audio collector integrated into the source device. If the audio source 16 is a memory, it may be, for example, a local memory or a memory integrated into the source device. If the audio source 16 includes an interface, it may be, for example, an external interface for receiving audio from an external audio source. The external audio source may be, for example, an external audio capture device such as a microphone, an external memory, or an external audio generation device. The external audio generation device may be, for example, an external computer audio processor, a computer, or a server. The interface may be any type of interface, for example, a wired or wireless interface, or an optical interface following any proprietary or standardized interface protocol.

[0043] In this application, the audio source 16 acquires a current scenario audio signal. The current scenario audio signal is an audio signal acquired by collecting a sound field at a microphone position in a space, and the current scenario audio signal is also called an original scenario audio signal. For example, the current scenario audio signal may be an audio signal acquired by higher order ambisonics (HOA) technology. The audio source 16 acquires a HOA signal to be encoded, for example, may acquire a HOA signal using a real acquisition device, or may synthesize a HOA signal using an artificial audio object. Optionally, the HOA signal to be encoded may be a time domain HOA signal or a frequency domain HOA signal.

[0044] The audio pre-processor 18 is configured to receive an original audio signal and perform pre-processing on the original audio signal to obtain a pre-processed audio signal. For example, the pre-processing performed by the audio pre-processor 18 may include trimming or noise removal.

[0045] Encoder 20 is configured to receive the preprocessed audio signal and to process the preprocessed audio signal to provide encoded bitstream information.

[0046] Communications interface 22 of source device 12 may be configured to receive bitstream information and transmit the bitstream to destination device 14 over communications channel 13. Communications channel 13 may be, for example, a direct wired or wireless connection and any type of network, for example, a wired or wireless network, or any combination thereof, or any type of private and public network, or any combination thereof.

[0047] The destination device 14 includes a decoder 30 and may optionally include a communications interface 28, an audio post-processor 32, and a playback device 34.

[0048] Communications interface 28 of destination device 14 is configured to receive bitstream information directly from source device 12 and provide the bitstream information to decoder 30. Communications interface 22 and communications interface 28 may be configured to transmit or receive bitstream information over communications channel 13 between source device 12 and destination device 14.

[0049] Each of communication interface 22 and communication interface 28 may be configured as a one-way communication interface, indicated by an arrow going from source device 12 to destination device 14 and corresponding to communication channel 13 in FIG. 2, or a two-way communication interface, and may be configured to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or the transmission of data, such as encoded audio data.

[0050] The decoder 30 is configured to receive the bitstream information and to decode the bitstream information to obtain decoded audio data.

[0051] The audio post-processor 32 is configured to perform post-processing on the decoded audio data to obtain post-processed audio data. The post-processing performed by the audio post-processor 32 may include, for example, cropping or resampling.

[0052] The playback device 34 is configured to receive the post-processed audio data and play the audio to a user or listener. The playback device 34 may be or include any type of playback device configured to play the restored audio, for example, an integrated or external speaker. For example, the speaker may include a horn, a sound box, etc.

[0053] Fig. 3 is an exemplary diagram of the structure of an HOA encoding device according to the present application. As shown in Fig. 3, the HOA encoding device can be used in the encoder 20 of the above-mentioned audio decoding system 10. The HOA encoding device includes a virtual speaker setting unit, a coding analysis unit, a virtual speaker set generation unit, a virtual speaker selection unit, a virtual speaker signal generation unit, and a core encoder processing unit.

[0054] The virtual speaker setting unit is configured to configure the virtual speakers based on the encoder setting information to obtain virtual speaker setting parameters. The encoder setting information includes, but is not limited to, HOA order, encoding bit rate, user-defined information, etc. The virtual speaker setting parameters include, but are not limited to, the quantity of the virtual speakers, the HOA order of the virtual speakers, etc.

[0055] The virtual speaker setting parameters output by the virtual speaker setting unit are used as input of the virtual speaker set generation unit.

[0056] The coding analysis unit is configured to perform coding analysis of the HOA signal to be coded, for example, to analyze the sound field distribution of the HOA signal to be coded, including characteristics such as the quantity of sound sources, directivity, and variance of the HOA signal to be coded, in order to obtain one of the decision conditions for determining how to select a target virtual speaker.

[0057] In the present application, the HOA encoding device may alternatively not include a coding analysis unit, in other words, the HOA encoding device may not analyze the input signal. This is not a limitation. In this case, a default setting is used to determine how to select the target virtual speaker.

[0058] The HOA encoding device obtains a HOA signal to be encoded. For example, a HOA signal recorded by a real sound pickup device or a HOA signal synthesized using an artificial sound object may be used as the input of the encoder, and the HOA signal to be encoded input to the encoder may be a time domain HOA signal or a frequency domain HOA signal.

[0059] The virtual speaker set generation unit is configured to generate a virtual speaker set, which may include multiple virtual speakers, where the virtual speakers in the virtual speaker set may be referred to as "candidate virtual speakers".

[0060] The virtual speaker set generation unit generates HOA coefficients of the designated candidate virtual speakers. The coordinates (i.e., position information) of the candidate virtual speakers and the HOA orders of the candidate virtual speakers are provided by the virtual speaker setting unit and are used to generate HOA coefficients of the candidate virtual speakers. The method of determining the coordinates of the candidate virtual speakers includes, but is not limited to, generating K virtual speakers according to an equidistance rule and generating K candidate virtual speakers that are not uniformly distributed according to the auditory principle. The coordinates of the uniformly distributed virtual speakers are generated based on the quantity of the candidate virtual speakers.

[0061] Next, the HOA coefficients for the virtual speakers are generated.

[0062] A sound wave is transmitted through an ideal medium. The phase velocity of the sound wave is k=w / c and the angular frequency is w=2πf, where f is the sound wave frequency and c is the sound speed. Therefore, the sound pressure p satisfies the following equation (1). ▽2 p+k 2 p=0 (1), where ▽ 2 is the Laplacian operator.

[0063] The following equation (2) can be obtained for the sound pressure p by solving equation (1) in spherical coordinates:

[0064]

number

[0065] Where: where r is the radius of the sphere, θ is the azimuth angle (although the azimuth angle is sometimes called the azimuth angle), φ is the elevation angle, k is the phase velocity, s is the amplitude of an ideal plane wave, and m is the order number of the HOA order.

[0066]

number

[0067] represents the spherical Bessel functions, also known as radial basis functions, where the first J is the imaginary unit,

[0068]

number

[0069] does not change with angle,

[0070]

number

[0071] are the spherical harmonics corresponding to θ and φ,

[0072]

number

[0073] is a spherical harmonic function in the sound source direction. The Ambisonics coefficients are:

[0074]

number

[0075] It is.

[0076] Therefore, the general expansion form (4) of the sound pressure p can be obtained as follows:

[0077]

number

[0078] The above equation (3) may indicate that the sound field may be expanded on a sphere based on spherical harmonics, and may also be represented based on Ambisonics coefficients.

[0079] Accordingly, if the Ambisonics coefficients are known, the sound field can be restored. By using the Ambisonics coefficients as an approximate description of the sound field, if equation (3) is truncated to the Nth term, the Ambisonics coefficients are called the Nth order HOA coefficients, which are also called Ambisonics coefficients. The Nth order Ambisonics coefficients are (N+1) in total. 2 The HOA signal has channels. Optionally, the HOA order may range from 2nd to 10th order. When spherical harmonics are superimposed based on the coefficients corresponding to the sampling points of the HOA signal, the spatial sound field at the moment corresponding to the sampling points can be restored. The HOA coefficients of the virtual loudspeaker can be generated according to this principle. s , φ sare set to the azimuth angle and elevation angle, i.e., the position information of the virtual speaker, and the HOA coefficients, also called Ambisonics coefficients, of the virtual speaker can be obtained according to Equation (3). For example, for a third-order HOA signal, if s=1, the HOA coefficients corresponding to the 16-channel, third-order HOA signal can be obtained by the spherical harmonic function

[0080]

number

[0081] The formula for calculating the HOA coefficients corresponding to the 16-channel and third-order HOA signals is specifically shown in Table 1.

[0082] [Table 1]

[0083] In Table 1, θ represents the azimuth angle as the position information of the virtual speaker on the preset sphere, φ represents the elevation angle as the position information of the virtual speaker on the preset sphere, l represents the HOA order, where l = 0, 1, ..., N, and m represents the direction parameter of each order, where m = -1, ..., 1. According to the polar coordinate formulas in Table 1, the HOA coefficients corresponding to the 16-channel and third-order HOA signals of the virtual speaker can be obtained based on the position information of the virtual speaker.

[0084] The HOA coefficients of the candidate virtual speakers output by the virtual speaker set generation unit are used as the input of the virtual speaker selection unit.

[0085] The virtual speaker selection unit is configured to select a target virtual speaker from a plurality of candidate virtual speakers in the virtual speaker set based on the HOA signal to be encoded, and this target virtual speaker may also be referred to as a "virtual speaker matching the HOA signal to be encoded" or for short as a matching virtual speaker.

[0086] The virtual speaker selection unit selects a designated suitable virtual speaker based on the HOA signal to be encoded and the HOA coefficients of the candidate virtual speakers output by the virtual speaker set generation unit.

[0087] In the following, an example is used to explain how to select a suitable virtual speaker. In a possible implementation, an inner product is performed between the HOA coefficient adaptation of the candidate virtual speaker and the HOA coefficient of the HOA signal to be encoded, and the candidate virtual speaker with the maximum absolute value of the inner product is selected as the target virtual speaker; that is, the suitable virtual speaker (projection) of the HOA signal to be encoded to the candidate virtual speaker is superimposed on the linear combination of the HOA coefficient of the candidate virtual speaker, and then the projection vector is subtracted from the HOA signal to be encoded to obtain a difference. The above process is repeated on this difference to implement iterative calculation. A suitable virtual speaker is generated in each iteration, and the coordinates of the suitable virtual speaker and the HOA coefficient of the suitable virtual speaker are output. It can be understood that multiple suitable virtual speakers are selected, and one suitable virtual speaker is generated in each iteration. (In addition, other implementation methods are not limited).

[0088] The coordinates of the target virtual speaker and the HOA coefficients of the target virtual speaker output from the virtual speaker selection unit are used as inputs of the virtual speaker signal generation unit.

[0089] The virtual speaker signal generating unit is configured to generate a virtual speaker signal based on the HOA signal to be encoded and attribute information of the target virtual speaker. If the attribute information is position information, the HOA coefficient of the target virtual speaker is determined based on the position information of the target virtual speaker. If the attribute information includes an HOA coefficient, the HOA coefficient of the target virtual speaker is obtained from the attribute information.

[0090] The virtual speaker signal generation unit calculates a virtual speaker signal based on the HOA signal to be encoded and the HOA coefficients of a target virtual speaker.

[0091] The HOA coefficients of the virtual speaker are represented by a matrix A, and the HOA signal to be encoded can be obtained by linear combination using the matrix A. Furthermore, the theoretical optimal solution w, i.e., the virtual speaker signal, can be obtained using the least squares method. For example, the following calculation formula can be used: w=A -1 X

[0092] A -1 represents the inverse matrix of matrix A, the size of matrix A is (M × C), C is the number of target virtual speakers, M is the number of channels of N-th order HOA coefficients, and M = (N + 1) 2 where a represents the HOA coefficient of the target virtual speaker. For example,

[0093]

number

[0094] X represents the HOA signal to be encoded, the size of matrix X is (M×L), M is the number of channels of N-th order HOA coefficients, L is the number of time-domain or frequency-domain sampling points, and x represents the coefficients of the HOA signal to be encoded. For example,

[0095]

number

[0096] The virtual speaker signals output by the virtual speaker signal generation unit are used as inputs of the core encoder processing unit.

[0097] The core encoder processing unit is configured to perform core encoder processing on the virtual speaker signals to obtain a transmission bitstream.

[0098] The core encoder processing includes, but is not limited to, transforms, quantization, psychoacoustic models, bitstream generation, etc., and may process frequency domain or time domain transmission channels, which is not limited herein.

[0099] Based on the description of the above embodiments, the present application provides a method for determining a virtual speaker set. The method for determining a virtual speaker set is based on the following presets:

[0100] 1. Virtual speaker distribution table

[0101] The virtual speaker distribution table includes position information of K virtual speakers, the position information includes an elevation scale and an azimuth scale, and K is a positive integer greater than 1. The K virtual speakers are set to be distributed on a preset sphere. The preset sphere may include X latitude circles and Y longitude circles. X and Y may be the same or different. Both X and Y are positive integers. For example, X is 512, 768, 1024, etc., and Y is 512, 768, 1024, etc. The virtual speakers are placed at the intersections of the X latitude circles and the Y longitude circles. Larger values ​​of X and Y indicate more candidate selection positions of the virtual speakers and better reproduction effects of the sound field formed by the finally selected virtual speakers.

[0102] 4a is an exemplary schematic diagram of a preset sphere according to the present application. As shown in FIG. 4a, the preset sphere includes L (L>1) latitude regions, and the mth latitude region is T m It contains latitude circles and has m of K virtual speakers. i The azimuth angle difference between adjacent virtual speakers distributed on the th latitude circle is α m , 1≦m≦L, and T m is a positive integer, 1≦m i ≦T m It is. m > 1, the elevation angle difference between any two adjacent latitude circles in the m-th latitude region is α mFIG. 4b is a schematic diagram of an example of an elevation angle and an azimuth angle according to the present application. As shown in FIG. 4b, the tip angle between the connecting line between the position of the virtual speaker and the center of the sphere and a preset horizontal plane (for example, the plane on which the equatorial circle is placed, the plane on which the South Pole is placed, or the plane on which the North Pole is placed, where the plane on which the South Pole is placed is perpendicular to the connecting line between the South Pole and the North Pole, and the plane on which the North Pole is placed is perpendicular to the connecting line between the South Pole and the North Pole) is the elevation angle of the virtual speaker. The tip angle between the projection of the connecting line between the position of the virtual speaker and the center of the sphere on the horizontal plane and the set initial direction is the azimuth angle of the virtual speaker.

[0103] It should be understood that the K virtual speakers are distributed on one or more latitudinal circles in each latitudinal region, and the distance between adjacent virtual speakers located on the same latitudinal circle is expressed using an azimuth angle difference, and the azimuth angle difference between all adjacent virtual speakers on the same latitudinal circle is equal. For example, m i The azimuth angle difference between any two adjacent virtual speakers on the th latitude circle is α m For virtual speakers placed in the same latitudinal region, if the latitudinal region includes multiple latitudinal circles, the azimuth angle difference between adjacent virtual speakers is the same on every latitudinal circle in the latitudinal region. For example, in the m-th latitudinal region, i The azimuth angle difference between adjacent virtual speakers on the th latitude circle and the mth i+1 The difference in azimuth angle between adjacent virtual speakers on the th latitude circle is α m In addition, when a latitudinal region includes multiple latitudinal circles, the distance between the latitudinal circles in the latitudinal region is expressed as an elevation angle difference, and the elevation angle difference between any two adjacent latitudinal circles is equal to the azimuth angle difference between adjacent virtual speakers in the latitudinal region.

[0104] In a possible implementation, α n = α m or α n ≠α m where α n is the azimuth angle difference between adjacent virtual speakers among K virtual speakers and distributed on any latitude circle in the nth latitude region, where n ≠ m.

[0105] In other words, for virtual speakers located in different latitude regions, the azimuth angle difference between adjacent virtual speakers may be equal, in which case α n = α m or may not be equal, in which case α n ≠α m It should be understood that in the present application, the azimuth angle differences between adjacent virtual speakers in the L latitudinal regions may all be equal, or the azimuth angle differences between adjacent virtual speakers in the L latitudinal regions may not all be equal, and further, the azimuth angle differences between adjacent virtual speakers in some of the L latitudinal regions may be equal, and such azimuth angle differences and the azimuth angle differences between adjacent virtual speakers in other latitudinal regions may not be equal. These are not limited.

[0106] In a possible implementation, α c <α m and α c is in K virtual speakers and c is the azimuth angle difference between adjacent virtual speakers distributed on the th latitude circle, and m c The th latitude circle is any latitude circle that is in the L latitude regions and that is in the latitude region that includes the equatorial latitude circle.

[0107] Specifically, among the L latitudinal regions, the azimuth angle difference between adjacent virtual speakers in the latitudinal region that includes the equatorial latitude circle is the smallest; in other words, among the L latitudinal regions, the virtual speakers in the latitudinal region that includes the equatorial latitude circle are most densely distributed.

[0108] Optionally, the positions of the K virtual speakers in the virtual speaker distribution table may be represented by scales, and the scales may include elevation scales and azimuth scales. For example, on any latitude circle, the azimuth angle of one of the virtual speakers distributed on the latitude circle is set to 0, and then the corresponding azimuth scale is obtained by conversion according to a preset conversion formula between azimuth angle and azimuth scale. Since the azimuth angle difference between any adjacent virtual speakers on the latitude circle is equal, the azimuth angles of other virtual speakers on the latitude circle can be obtained to obtain the azimuth scales of other virtual speakers according to the aforementioned conversion formula. It should be noted that the specific virtual speakers on the latitude circle whose azimuth angles are set to 0 are not particularly limited in this application. Similarly, since the elevation angle difference between adjacent virtual speakers in the longitude circle direction satisfies the above-mentioned requirements, the elevation angles of other virtual speakers may be obtained after the virtual speaker with an elevation angle of 0 is set, and the elevation angle scales of all virtual speakers on the longitude circle may be obtained according to the conversion formula between the preset elevation angle and the elevation angle scale. It should be noted that in the present application, the virtual speaker on the longitude circle with the elevation angle set to 0 is not particularly limited. For example, the virtual speaker may be a virtual speaker placed on the equatorial circle, a virtual speaker placed at the South Pole, or a virtual speaker placed at the North Pole.

[0109] Optionally, an elevation angle φ of the kth virtual speaker among the K virtual speakers k and elevation scale φ k ' satisfies the following formula (i.e., the conversion formula between elevation angle and elevation scale):

[0110]

number

[0111] r k represents the radius of the longitude circle on which the kth virtual speaker is placed, and round() represents rounding.

[0112] Azimuth angle θ of the kth virtual speaker among K virtual speakers k and azimuth scale θ k' satisfies the following formula (i.e., the conversion formula between azimuth angle and azimuth angle scale):

[0113]

number

[0114] r k represents the radius of the latitude circle on which the kth virtual speaker is placed, and round() represents rounding.

[0115] 5a and 5b are exemplary distribution diagrams of K virtual speakers. As shown in FIG. 5a, the azimuth angle difference between adjacent virtual speakers in a latitude region including the equatorial latitude circle is smaller than the azimuth angle difference between adjacent virtual speakers in other latitude regions, and α c <α m As shown in Fig. 5b, the K virtual speakers are randomly and approximately uniformly distributed on the preset sphere.

[0116] table 2 Figure 5 shows a comparison between the distribution diagrams shown in Figure 5a and Figure 5b. With K=1669, it can be seen that the average value of the signal-to-noise ratio (SNR) of the HOA restored signal obtained by the distribution method in Figure 5a is higher than that of the HOA restored signal obtained by the distribution method in Figure 5b.

[0117] [Table 2]

[0118] table 2 As shown in FIG. 1, different types of test audios are used in this embodiment, and the file names from 1 to 12 are single-source speech signal, single-source instrument signal, two-source speech signal, two-source instrument signal, three-source speech and instrument mixed signal, four-source speech and instrument mixed signal, two-source noise signal 1, two-source noise signal 2, two-source noise signal 3, two-source noise signal 4, two-source Ambisonics signal 1, and two-source Ambisonics signal 2, respectively.

[0119] 6a and 6b are exemplary distribution diagrams of K virtual speakers. As shown in FIG. 6a, the azimuth angle difference between adjacent virtual speakers in L latitudinal regions is equal, and α n = α m As shown in Fig. 6b, the K virtual speakers are randomly and approximately uniformly distributed on the preset sphere.

[0120] table 3 Figure 6 shows a comparison between the distribution diagrams shown in Figure 6a and Figure 6b. With K=1669, it can be seen that the average value of the signal-to-noise ratio (SNR) of the HOA restored signal obtained by the distribution method in Figure 6a is higher than that of the HOA restored signal obtained by the distribution method in Figure 6b.

[0121] [Table 3]

[0122] table 3 As shown in FIG. 1, different types of test audios are used in this embodiment, and the file names from 1 to 12 are single-source speech signal, single-source instrument signal, two-source speech signal, two-source instrument signal, three-source speech and instrument mixed signal, four-source speech and instrument mixed signal, two-source noise signal 1, two-source noise signal 2, two-source noise signal 3, two-source noise signal 4, two-source Ambisonics signal 1, and two-source Ambisonics signal 2, respectively.

[0123] For example, the table 4 is an example of a virtual speaker distribution table. In this example, K is 530. 4 describes the specific distribution of 530 virtual speakers whose ordinal numbers range from 0 to 529. "Position" represents the azimuth and elevation scales of the virtual speaker with the corresponding ordinal number. In the "Position" column of the table, the number before the "," is the azimuth scale, and the number after the "," is the elevation scale.

[0124] [Table 4-1]

[0125] [Table 4-2]

[0126] [Table 4-3]

[0127] table 4 It should be noted that the sphere on which the virtual speakers are distributed includes 1024 longitude circles and 1024 latitude circles (the South Pole and the North Pole each correspond to one latitude circle), and the 1024 longitude circles and 1024 latitude circles correspond to 1024×1022+2=1046530 intersections, and the 1046530 intersections have their own elevation angles and azimuth angles. Accordingly, the 1046530 intersections have their own elevation angle scale and azimuth angle scale, and the table 4 The positions of the 530 virtual speakers are 530 of the 1046530 intersections. 4 The elevation scales are calculated based on the fact that the elevation angle at the equator is 0. Specifically, all elevation angles corresponding to the elevation scales other than the equatorial elevation scale are elevation angles relative to the plane on which the equator lies.

[0128] 2.F preset virtual speakers

[0129] The F virtual speakers satisfy the following conditions: i The difference in azimuth angle α between adjacent virtual speakers distributed on the th latitude circle mi is α m Larger than m i The th latitude circle is one of the latitude circles in the mth latitude region.

[0130] For ease of explanation, a virtual speaker among the K virtual speakers is called a candidate virtual speaker, and any virtual speaker among the F virtual speakers is called a center virtual speaker (sometimes called a first-round virtual speaker). Specifically, for any latitude circle on the preset sphere, one or more virtual speakers may be selected as a center virtual speaker from the multiple candidate virtual speakers distributed on that latitude circle, and the center virtual speaker is added to the F virtual speakers. When multiple virtual speakers are selected, the azimuth angle difference α between adjacent center virtual speakers is mi is the azimuth angle difference α between adjacent candidate virtual speakers. m is larger than α mi >α m That is, multiple candidate virtual speakers are distributed on a specific latitude circle. The central virtual speaker is selected from the multiple candidate virtual speakers, and the density is low. For example, when the azimuth angle difference α m is equal to 5°, and the azimuth angle difference α between adjacent central virtual speakers mi is equal to 8°.

[0131] In a possible implementation, α mi =q×α m where q is a positive integer greater than 1. The azimuth angle difference between adjacent central virtual speakers and the azimuth angle difference between adjacent candidate virtual speakers can be considered to be in a multiple relationship. For example, the azimuth angle difference α m is equal to 5°, and the azimuth angle difference α between adjacent central virtual speakers mi is equal to 10°.

[0132] 3. Each of the F virtual speakers corresponds to S virtual speakers.

[0133] For ease of explanation, a virtual speaker among the S virtual speakers is called a target virtual speaker. Specifically, the S virtual speakers corresponding to an arbitrary center virtual speaker satisfy the following condition: The S virtual speakers include an arbitrary center virtual speaker and (S-1) virtual speakers arranged around the arbitrary center virtual speaker, and any one of (S-1) correlations between the arbitrary center virtual speaker and the (S-1) virtual speakers is greater than each of (KS) correlations between (KS) virtual speakers among K virtual speakers other than the S virtual speakers and the arbitrary center virtual speaker.

[0134] That is, S R fk are K R fk The maximum R of S in fk K R fk When sorted in descending order, the first S R fk is the maximum S R fk It is.

[0135] R fk represents the correlation between an arbitrary center virtual speaker and the kth virtual speaker among the K virtual speakers, and R fk satisfies the following formula: R fk =B f (θ,φ) B k (θ,φ)

[0136] θ represents the azimuth angle of any virtual speaker, φ represents the elevation angle of any virtual speaker, and B f (θ,φ) represents the HOA coefficients of any virtual speaker, and B k (θ,φ) represents the HOA coefficient of the k-th virtual speaker among the K virtual speakers.

[0137] S target virtual speakers may be determined for each center virtual speaker according to the above-mentioned method. It should be understood that in the present application, F virtual speakers are preset from K virtual speakers. Therefore, the position of each center virtual speaker may also be represented by an elevation scale and an azimuth scale. Furthermore, each center virtual speaker corresponds to S virtual speakers, and the S virtual speakers are also from the K virtual speakers. Therefore, the position of each target virtual speaker may also be represented by an elevation scale and an azimuth scale.

[0138] FIG. 7 is an exemplary flowchart of a method for determining a virtual speaker set according to the present application. The process 700 may be implemented by the encoder 20 or the decoder 30 in the above-mentioned embodiment. That is, the encoder 20 of the audio sending device implements audio encoding and then transmits bitstream information to the audio receiving device. The decoder 30 of the audio receiving device decodes the bitstream information to obtain a target audio frame, and then performs rendering based on the target audio frame to obtain a sound field audio signal corresponding to one or more virtual speakers. The process 700 is described as a series of steps or operations. It should be understood that the process 700 may be performed in various orders and / or simultaneously, and is not limited to the execution order shown in FIG. 7. As shown in FIG. 7, the method includes the following steps:

[0139] Step 701: Determine a target virtual speaker from F preset virtual speakers based on the audio signal to be processed.

[0140] As described above, a coding analysis is performed on the audio signal to be processed. For example, the sound field distribution of the audio signal to be processed is analyzed, including the characteristics such as the number, directivity, and dispersion of the sound source of the audio signal, to obtain the HOA coefficient of the audio signal, and the HOA coefficient is used as one of the decision conditions for determining how to select the target virtual speaker. A virtual speaker that matches the audio signal to be processed can be selected based on the HOA coefficient of the audio signal to be processed and the HOA coefficients of the candidate virtual speakers (i.e., the above-mentioned F virtual speakers). In this application, the virtual speaker is called a target virtual speaker.

[0141] In a possible implementation, the HOA coefficients of the audio signal may be first obtained, then F groups of HOA coefficients corresponding to F virtual speakers are obtained, and the F virtual speakers are in one-to-one correspondence with the F groups of HOA coefficients, and then the virtual speaker corresponding to the group of HOA coefficients that has the maximum correlation with the HOA coefficients of the audio signal and is in the F groups of HOA coefficients is determined as the target virtual speaker.

[0142] In the present application, a dot product may be performed between the HOA coefficients of the F virtual speakers and the HOA coefficients of the audio signal separately, and the virtual speaker with the largest absolute value of the dot product is selected as the target virtual speaker. Specifically, each of the F groups of HOA coefficients is selected as the (N+1) 2 coefficients, and the HOA coefficients of the audio signal are (N+1) 2The HOA coefficients of the audio signal include N coefficients, where N represents the order of the audio signal. Thus, the HOA coefficients of the audio signal have a one-to-one correspondence with each of the F groups of HOA coefficients. Based on this correspondence, an inner product is performed between the HOA coefficients of the audio signal and each of the F groups of HOA coefficients to obtain a correlation between the HOA coefficients of the audio signal and each of the F groups of HOA coefficients. It should be noted that the target virtual speaker may alternatively be determined using another method, which is not particularly limited in this application.

[0143] Step 702: Obtain position information of each of the S virtual speakers corresponding to the target virtual speaker from a preset virtual speaker distribution table, where the position information includes an elevation angle scale and an azimuth angle scale.

[0144] Based on the above-mentioned preset in this application, when a target virtual speaker (i.e., a center virtual speaker) is determined, S virtual speakers corresponding to the target virtual speaker can be obtained. The position information of the S virtual speakers can be obtained based on the earliest set virtual speaker distribution table. The same expression is used for the K virtual speakers, and the position information of the S virtual speakers is respectively expressed by an elevation scale and an azimuth scale.

[0145] Once the target virtual speaker is determined, the target virtual speaker can be considered to be the center virtual speaker that is most highly correlated with the HOA coefficients of the audio signal to be processed. The S virtual speakers corresponding to each center virtual speaker are the S virtual speakers that are most highly correlated with the HOA coefficients of the center virtual speaker. Therefore, the S virtual speakers corresponding to the target virtual speaker are also the S virtual speakers that are most highly correlated with the HOA coefficients of the audio signal to be processed.

[0146] In the present application, a virtual speaker distribution table is preset, so that by deploying the virtual speakers according to the distribution table, a high average value of the signal-to-noise ratio (SNR) of the HOA restored signal can be obtained, and based on such distribution, S virtual speakers that have the highest correlation with the HOA coefficients of the audio signal to be processed are selected, thereby achieving the optimal sampling effect and improving the audio signal reproduction effect.

[0147] 8 is an exemplary diagram of the configuration of an apparatus for determining a virtual speaker set according to the present application. As shown in FIG. 8, the apparatus can be used in the encoder 20 or the decoder 30 in the above-mentioned embodiment. The apparatus for determining a virtual speaker set in this embodiment may include a determination module 801 and an acquisition module 802. The determination module 801 is configured to determine a target virtual speaker from F preset virtual speakers based on an audio signal to be processed, each of the F virtual speakers corresponds to S virtual speakers, F is a positive integer, and S is a positive integer greater than 1. The acquisition module 802 is configured to acquire position information of each of the S virtual speakers corresponding to the target virtual speaker from a preset virtual speaker distribution table, the virtual speaker distribution table includes position information of K virtual speakers, the position information includes an elevation scale and an azimuth scale, K is a positive integer greater than 1, F≦K, and F×S≧K.

[0148] In a possible implementation, the determination module 801 is particularly configured to obtain higher-order Ambisonics HOA coefficients of the audio signal and to obtain F groups of HOA coefficients corresponding to F virtual speakers, the F virtual speakers being in one-to-one correspondence with the F groups of HOA coefficients, and the determination module is further configured to determine as a target virtual speaker a virtual speaker corresponding to the group of HOA coefficients that has a maximum correlation with the HOA coefficients of the audio signal and is among the F groups of HOA coefficients.

[0149] In a possible implementation, the S virtual speakers corresponding to the target virtual speaker satisfy the following condition: The S virtual speakers include the target virtual speaker and (S-1) virtual speakers placed around the target virtual speaker, and any one of the (S-1) correlations between the (S-1) virtual speakers and the target virtual speaker is greater than each of the (KS) correlations between the target virtual speaker and (KS) virtual speakers other than the S virtual speakers among the K virtual speakers.

[0150] In a possible implementation, the K virtual speakers satisfy the following condition: The K virtual speakers are distributed on a preset sphere, and the preset sphere includes L latitudinal regions, where L>1, and the m-th latitudinal region among the L latitudinal regions is T m It contains latitude circles, is in K virtual speakers, and is i The difference in azimuth angle between adjacent virtual speakers distributed on the th latitude circle is α m , 1≦m≦L, and T m is a positive integer, 1≦m i ≦T m and T m >1, the elevation angle difference between any two adjacent latitude circles in the m-th latitude region is α m It is.

[0151] In a possible implementation, the nth latitude region of the L latitude regions is T n It contains latitude circles, is in K virtual speakers, and has n i The difference in azimuth angle between adjacent virtual speakers distributed on the th latitude circle is α n where 1≦n≦L and T n is a positive integer, 1≦n i ≦T n and T n >1, the elevation angle difference between any two adjacent latitude circles in the nth latitude region is α n where α n = α m or α n ≠α m and n ≠ m.

[0152] In a possible implementation, the cth latitude region among the L latitude regions is T c Contains latitude circles, T c One of the latitude circles is an equatorial latitude circle, which is in the K virtual speakers and is located at c i The difference in azimuth angle between adjacent virtual speakers distributed on the th latitude circle is α c , 1≦c≦L, and T c is a positive integer, 1≦c i ≦T c and T c >1, the elevation angle difference between any two adjacent latitude circles in the cth latitude region is α c where α c <α m and c ≠ m.

[0153] In a possible implementation, the F virtual speakers satisfy the following conditions: i The azimuth angle difference α between adjacent virtual speakers among F virtual speakers distributed on the latitudinal circle mi is α m Greater than.

[0154] In a possible implementation, α mi =q×α m where q is a positive integer greater than 1.

[0155] In a possible implementation, the correlation R between the kth virtual speaker of the K virtual speakers and the target virtual speaker is fk satisfies the following formula: R fk =B f (θ,φ) B k (θ,φ), where θ represents the azimuth angle of the target virtual speaker, φ represents the elevation angle of the target virtual speaker, and B f (θ,φ) represents the HOA coefficients of the target virtual speaker, and B k (θ,φ) represents the HOA coefficient of the k-th virtual speaker among the K virtual speakers.

[0156] The apparatus in this embodiment may be used to implement the technical solution of the method embodiment shown in FIG. 7, and the implementation principle and technical effect of the apparatus are similar and will not be described again in this specification.

[0157] In the implementation process, the steps in the above-mentioned method embodiments can be implemented using hardware integrated logic circuits in a processor or using instructions in the form of software. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The steps of the method disclosed in this application may be directly performed by a hardware coding processor, or may be performed by a combination of hardware and software modules as a coding processor. The software modules may be located in a mature storage medium in the art, for example, a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, or a register. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above-mentioned method in combination with the hardware of the processor.

[0158] The memory in the above embodiments may be volatile or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM) used as an external cache. By way of example and not of limitation, many forms of RAM may be used, such as static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synchlink DRAM, SLDRAM), and direct rambus random access memory (direct rambus RAM, DR RAM). It should be noted that the memory of the systems and methods described herein includes, but is not limited to, these and any other suitable types of memory.

[0159] Those skilled in the art may recognize that the units and algorithm steps may be implemented by electronic hardware, or a combination of computer software and electronic hardware, in combination with the examples described in the embodiments disclosed herein. Whether a function is executed by hardware or software depends on the design constraints of a specific application and technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementations are not considered to go beyond the scope of this application.

[0160] For the sake of convenience and concise description, the detailed work processes of the above-mentioned systems, devices and units may be clearly understood by those skilled in the art by referring to the corresponding processes in the above-mentioned method embodiments, and the details will not be described again in this specification.

[0161] In some embodiments provided in the present application, it should be understood that the disclosed system, device, and method may be implemented in other manners. For example, the described device embodiment is merely an example. For example, the division into units is merely a logical functional division, and may be other divisions in actual implementation. For example, multiple units or components may be combined or integrated to become another system, or some features may be ignored or not implemented. In addition, the shown or discussed mutual couplings or direct couplings or communication connections may be implemented using some interfaces. Indirect couplings or communication connections between devices or units may be implemented in electronic, mechanical, or other forms.

[0162] The units described as separate parts may or may not be physically separated, and the parts shown as units may or may not be physical units, and may be located in one place or distributed among multiple network units. Some or all of the units may be selected based on the actual requirements to achieve the objectives of the solutions of the embodiments.

[0163] In addition, the functional units in the embodiments of the present application may be integrated into one processing unit, each of the units may exist physically alone, or two or more units may be integrated into one unit.

[0164] When a function is implemented in the form of a software functional unit and sold or used as an independent product, the function may be stored in a computer-readable storage medium. Based on such understanding, the essential technical solution of the present application, or a part that contributes to the prior art, or a part of the technical solution, may be implemented in the form of a software product. The computer software product is stored in a storage medium and includes some instructions for instructing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes any medium that can store program code, such as a USB flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0165] The above description is merely a specific implementation of the present application, and does not limit the scope of protection of the present application. Any variations or replacements that can be easily conceived by those skilled in the art within the technical scope disclosed in the present application shall fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the scope of protection of the claims.

Claims

1. 1. A method for determining a virtual speaker set, comprising: determining a target virtual speaker from F preset virtual speakers based on the audio signal to be processed, each of the F virtual speakers corresponding to S virtual speakers, F being a positive integer and S being a positive integer greater than 1; obtaining position information of each of S virtual speakers corresponding to the target virtual speaker from a preset virtual speaker distribution table, the preset virtual speaker distribution table including position information of K virtual speakers, the position information including an elevation angle scale and an azimuth angle scale, K is a positive integer greater than 1, F≦K, and F×S≧K; Including, said step of determining a target virtual speaker from F preset virtual speakers based on an audio signal to be processed further comprising: obtaining higher order Ambisonics HOA coefficients of the audio signal; obtaining F groups of HOA coefficients corresponding to the F virtual speakers, the F virtual speakers having a one-to-one correspondence with the F groups of HOA coefficients; determining as the target virtual speaker a virtual speaker corresponding to a group of HOA coefficients that has a maximum correlation with the HOA coefficients of the audio signal and is among the F groups of HOA coefficients; A method for determining a virtual speaker set comprising:

2. The S virtual speakers corresponding to the target virtual speaker satisfy the following conditions: The S virtual speakers include the target virtual speaker and (S-1) virtual speakers placed around the target virtual speaker, and any one of the (S-1) correlations between the (S-1) virtual speakers and the target virtual speaker is greater than each of the (K-S) correlations between the target virtual speaker and (K-S) virtual speakers other than the S virtual speakers among the K virtual speakers. The method according to claim 1 , wherein

3. The K virtual speakers satisfy the following conditions: The K virtual speakers are distributed on a preset sphere, the preset sphere including L latitudinal regions, where L>1; The m-th latitudinal region among the L latitudinal regions is T m The latitude circles are included in the K virtual speakers and m i The difference in azimuth angle between adjacent virtual speakers distributed on the th latitude circle is α m , 1≦m≦L, and T m is a positive integer, 1≦m i ≦T m where: T m >1, the elevation angle difference between any two adjacent latitude circles in the m-th latitude region is α m That is, The method according to claim 1 or 2, which satisfies the above.

4. The n-th latitudinal region among the L latitudinal regions is T n The K virtual speakers include n latitude circles. i The difference in azimuth angle between adjacent virtual speakers distributed on the th latitude circle is α n where 1≦n≦L and T n is a positive integer, 1≦n i ≦T n and T n >1, the elevation angle difference between any two adjacent latitude circles in the nth latitude region is α n where α n = α m or α n ≠α m 4. The method of claim 3, wherein n≠m.

5. The c-th latitude region among the L latitude regions is T c The T c One of the latitude circles is an equatorial latitude circle, which is in the K virtual speakers and is c i The difference in azimuth angle between adjacent virtual speakers distributed on the th latitude circle is α c where 1≦c≦L and T c is a positive integer, 1≦c i ≦T c where T c >1, the elevation angle difference between any two adjacent latitude circles in the c-th latitude region is α c where α c <α m 4. The method of claim 3, wherein c≠m.

6. The F virtual speakers satisfy the following conditions: Said m i The azimuth angle difference α between adjacent virtual speakers distributed on the latitudinal circle and among the F virtual speakers mi is α m Greater than The method according to any one of claims 3 to 5, wherein

7. α mi = q × α m 7. The method of claim 6, wherein q is a positive integer greater than 1.

8. The correlation R between the k-th virtual speaker of the K virtual speakers and the target virtual speaker fk satisfies the following formula, R fk =B f (θ,φ)・B k (θ,φ), here θ represents the azimuth angle of the target virtual speaker, φ represents the elevation angle of the target virtual speaker, and B f (θ,φ) represents the HOA coefficients of the target virtual speaker, k The method of claim 2 , wherein (θ, φ) represents the HOA coefficient of the kth virtual speaker.

9. An apparatus for determining a virtual speaker set, comprising: a determination module configured to determine a target virtual speaker from F preset virtual speakers based on an audio signal to be processed, each of the F virtual speakers corresponding to S virtual speakers, F being a positive integer and S being a positive integer greater than 1; an acquisition module configured to acquire position information of each of S virtual speakers corresponding to the target virtual speaker from a preset virtual speaker distribution table, the preset virtual speaker distribution table including position information of K virtual speakers, the position information including an elevation angle scale and an azimuth angle scale, K is a positive integer greater than 1, F≦K, and F×S≧K; Including, The apparatus for determining a virtual speaker set, wherein the determination module is particularly configured to obtain higher-order Ambisonics HOA coefficients of the audio signal and to obtain F groups of HOA coefficients corresponding to the F virtual speakers, the F virtual speakers being in one-to-one correspondence with the F groups of HOA coefficients, and the determination module is further configured to determine as the target virtual speaker a virtual speaker corresponding to a group of HOA coefficients that has a maximum correlation with the HOA coefficients of the audio signal and is in the F groups of HOA coefficients.

10. The S virtual speakers corresponding to the target virtual speaker satisfy the following conditions: The S virtual speakers include the target virtual speaker and (S-1) virtual speakers placed around the target virtual speaker, and any one of the (S-1) correlations between the (S-1) virtual speakers and the target virtual speaker is greater than each of the (K-S) correlations between the target virtual speaker and (K-S) virtual speakers other than the S virtual speakers among the K virtual speakers. The apparatus according to claim 9, wherein

11. The K virtual speakers satisfy the following conditions: The K virtual speakers are distributed on a preset sphere, the preset sphere including L latitudinal regions, where L>1; The m-th latitudinal region among the L latitudinal regions is T m The latitude circles are included in the K virtual speakers and m i The difference in azimuth angle between adjacent virtual speakers distributed on the th latitude circle is α m , 1≦m≦L, and T m is a positive integer, 1≦m i ≦T m where: T m >1, the elevation angle difference between any two adjacent latitude circles in the m-th latitude region is α m That is, The apparatus according to claim 9 or 10, which satisfies the above.

12. The n-th latitudinal region among the L latitudinal regions is T n The K virtual speakers include n latitude circles. i The difference in azimuth angle between adjacent virtual speakers distributed on the th latitude circle is α n where 1≦n≦L and T n is a positive integer, 1≦n i ≦T n and T n >1, the elevation angle difference between any two adjacent latitude circles in the nth latitude region is α n where α n = α m or α n ≠α m 12. The apparatus of claim 11, wherein n ≠ m.

13. The c-th latitude region among the L latitude regions is T c The T c One of the latitude circles is an equatorial latitude circle, which is in the K virtual speakers and is c i The difference in azimuth angle between adjacent virtual speakers distributed on the th latitude circle is α c where 1≦c≦L and T c is a positive integer, 1≦c i ≦T c where T c >1, the elevation angle difference between any two adjacent latitude circles in the c-th latitude region is α c where α c <α m 12. The apparatus of claim 11, wherein c≠m.

14. The F virtual speakers satisfy the following conditions: Said m i The azimuth angle difference α between adjacent virtual speakers distributed on the latitudinal circle and among the F virtual speakers mi is α m Greater than The device according to any one of claims 11 to 13, wherein

15. α mi = q × α m 15. The apparatus of claim 14, wherein q is a positive integer greater than 1.

16. The correlation R between the k-th virtual speaker of the K virtual speakers and the target virtual speaker fk satisfies the following formula, R fk =B f (θ,φ)・B k (θ,φ) where θ represents the azimuth angle of the target virtual speaker, φ represents the elevation angle of the target virtual speaker, and B f (θ,φ) represents the HOA coefficients of the target virtual speaker, k The apparatus of claim 10 , wherein (θ, φ) represents the HOA coefficient of the kth virtual speaker.

17. 1. An audio processing device, comprising: one or more processors; and a memory configured to store one or more programs. An audio processing device, wherein the one or more programs, when executed by the one or more processors, enable the one or more processors to implement a method according to any one of claims 1 to 8.

18. A computer readable storage medium comprising a computer program, the computer program being capable of executing a method according to any one of claims 1 to 8 when executed on a computer.

Images