Acoustic synthesis system

Abstract

The present disclosure is a system comprising: an up-sampling unit for upsampling sampling data; and a down-sampling unit for downsampling upsampled data upsampled by the up-sampling unit. The up-sampling unit and the down-sampling unit are connected over a communication network.

Description

Sound Synthesis System

[0001] The present disclosure relates to audio synthesis techniques.

[0002] In recent years, with the trend toward remote social interactions, there has been a demand for remote collaboration using sounds other than human speech, such as remote ensembles. In ensembles, separate sounds sampled at different locations must be appropriately synthesized at each location and fed back to the performers. In this sound synthesis, deviations in the sampling clocks of the separately sampled sounds cause degradation in the sound quality after synthesis.

[0003] Crystal oscillators are commonly used as clock sources for electronic devices, with an accuracy of approximately ±20 to ±100 ppm. For example, even if sampling is set to 48 kHz, if a crystal oscillator with an accuracy of approximately ±100 ppm is used, the sampling frequency will accurately range from a minimum of 47.9952 kHz to a maximum of 48.0048 kHz. This means that a deviation of up to approximately 9.6 samples occurs per second. In this case, approximately 9.6 samples of data will be left over or missing during audio synthesis, causing degradation of sound quality. There are two known approaches to addressing this deviation in the sampling clock.

[0004] The first is sampling clock synchronization. Audio equipment that requires high sound quality has a word clock terminal and a clock synchronization mechanism that enables distribution of its own clock or sampling dependent on a received clock. The network audio Dante uses a time synchronization protocol called PTP to synchronize sampling and playback clocks in remote locations (see, for example, Non-Patent Document 1). This method requires dedicated equipment for clock synchronization, and therefore requires limited specialized knowledge and is difficult to use in limited environments.

[0005] The second is resampling. Resampling is a technique for changing the sampling frequency to a different frequency, performing calculations to interpolate intermediate data not present in the pre-conversion sampling data, and then resampling at the converted frequency. Correcting even slight frequency deviations requires a large amount of calculation for the interpolated data, and the interpolation inherently involves a large processing delay, so resampling has not been used for sound synthesis, such as in real-time remote ensembles.

[0006] The synchronization method cannot be adopted when handling audio signals accompanying video signals such as HDMI (registered trademark), because a synchronization system cannot be added. In other words, although video signals that do not have a synchronization mechanism between multiple sources, such as HDMI, are widely used, it is not possible to synthesize or generate high-quality audio signals through synchronization in such an environment.

[0007] Without a synchronization mechanism, the resampling described above is necessary, but this requires a large amount of calculation and is not applicable to sound synthesis requiring low latency, such as in remote ensembles. Specifically, methods such as upsampling, bandpass filtering, and downsampling are known for resampling. Accurate resampling requires upsampling to the least common multiple of the two frequencies before and after conversion, which significantly increases the amount of data. Furthermore, this increased data is typically interpolated by insert sampling using an FIR (finite impulse response) filter. This filter operation requires a multiply-and-accumulate operation equal to the number of filter stages. Resampled data is then downsampled.

[0008] To estimate the computational complexity, we estimate the number of multiplications. Considering resampling from (48k + d1) Hz to (48k + d2) Hz, upsampling is performed at (48k + d1) * (48k + d2) Hz. Because d1 and d2 are significantly smaller than 48k, the upsampling is approximately 48k * 48k = 2.3 GHz. Furthermore, since the FIR filter must be wider than the audio period, assuming 100 periods, the pre-conversion samples are 200 samples, and the upsampling results in 200 * 48k samples. In this case, the FIR filter has 9.6M stages, requiring 9.6M multiplications per sample. Since the sampling frequency is 2.3 GHz, we can see that 2.3G * 9.6M = approximately 22E (exa: 10^18) multiplications per second are required per channel. Furthermore, since audio synthesis involves synthesizing several to several tens of channels, calculations are required for each channel. This cannot be calculated in real time using current standalone computers.

[0009] Therefore, for high-quality remote collaboration such as remote ensemble performances, it is necessary to reduce the amount of computation required to resample audio signals sampled at multiple different clocks in remote locations, and then synthesize and generate the audio signals.

[0010] Sheu, Jia-Shing, Ho-Nien Shou, and Wei-Jun Lin. , “Realization of an Ethernet-based synchronous audio playback system.”, Multimedia Tools and Applications, 75.16 (2016): 9797-9818.

[0011] The present disclosure aims to reduce the amount of calculation required for audio signals sampled at remote locations using multiple different clocks.

[0012] In the resampling method of the present disclosure, an upsampling unit upsamples sampling data, and a downsampling unit connected to the upsampling unit via a communication network downsamples the upsampling data upsampled by the upsampling unit.

[0013] In the audio synthesis system according to the present disclosure, the upsampling unit and the downsampling unit are connected via a communication network. In this manner, the present disclosure generates an audio signal by performing upsampling and its interpolation process at the sampling point, and performing only downsampling according to a clock at the audio synthesis point.

[0014] The upsampling and interpolation processes are performed on the transmitting side of the audio signal. By adopting this configuration, the amount of data circulating over the network increases, but the computationally intensive processes previously performed in the synthesis process become unnecessary. Therefore, the present disclosure can resample audio signals sampled at multiple different clocks in remote locations with reduced computational complexity to generate audio signals.

[0015] An acoustic synthesis system according to the present disclosure includes: a sampling unit that samples an acoustic signal; an upsampling unit that upsamples the sampled data sampled by the sampling unit; an interpolation unit that calculates a signal level of the upsampling data upsampled by the upsampling unit; a data transmission unit that transmits the upsampling data having the value calculated by the interpolation unit to the communication network; a data reception unit that receives the upsampling data transmitted from the data transmission unit; a downsampling unit that downsamples the upsampling data received by the data reception unit; and a synthesis unit that synthesizes the downsampling data downsampled by the downsampling unit. In the present disclosure, the data transmitted from the data transmission unit may be referred to as "oversampling data."

[0016] The sound synthesis system according to the present disclosure may further include an intermediate sound synthesis unit connected to the communication network, which synthesizes upsampled data upsampled by the upsampling unit. Since the sound synthesis system according to this embodiment includes the intermediate sound synthesis unit, the amount of data circulated over the network can be reduced. This allows the sound synthesis system according to the present disclosure to reduce the amount of calculations performed by the downsampling unit.

[0017] The sound synthesis system according to the present disclosure may include a sampling unit that samples sound signals and a data transmission unit that transmits the sampled data sampled by the sampling unit, and the sampling unit and the data transmission unit may be connected to the upsampling unit via the communication network. In the sound synthesis system of this embodiment, since the upsampling unit is connected to the communication network, the upsampling function of the sampling unit can be omitted.

[0018] The above disclosures can be combined as much as possible.

[0019] According to the present disclosure, it is possible to reduce the amount of calculation required for audio signals sampled at multiple different clocks in remote locations. This allows audio signals sampled at multiple different clocks to be resampled with reduced calculations, and audio signals to be synthesized and generated. Therefore, the present disclosure enables remote collaboration, such as a high-quality remote ensemble performance.

[0020] 1 shows an embodiment of the audio synthesis system of the present disclosure. 2 shows an embodiment of the audio synthesis system of the present disclosure. 3 shows an example configuration of a sampling / audio synthesis unit. 4 is an explanatory diagram showing a specific example of resampling. 5 shows an embodiment of the audio synthesis system of the present disclosure. 6 shows an example configuration of an intermediate audio synthesis unit. 7 shows an example configuration of an intermediate audio synthesis unit. 8 shows an embodiment of the audio synthesis system of the present disclosure. 9 shows an example configuration of an oversampling unit. 10 shows an example configuration of a sampling / audio synthesis unit.

[0021] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings indicate the same components.

[0022] Generally, resampling processes integrate upsampling, interpolation, and downsampling. In particular, the increase in the amount of calculations in this resampling process is due to multiplication processes using FIR filters in the interpolation section. In other words, in a typical synthesis process, the amount of calculations increases with the number n of distant acoustic signal sources.

[0023] 1 shows an embodiment of a sound synthesis system according to the present disclosure. The sound synthesis system of this embodiment is a system in which a transmitting device 81 and a receiving device 82 are connected via a network 92. The network 92 includes not only a wide-area computer network, but also an internal or local network within a device or between devices.

[0024] The transmitting device 81 includes an upsampling unit 12, and the receiving device 82 includes a downsampling unit 22. The audio synthesis system of the present disclosure executes the resampling method of the present disclosure. Specifically, in the resampling method of the present disclosure, the upsampling unit 12 upsamples the sampled data of the audio signal, and the downsampling unit 22 downsamples the upsampled data upsampled by the upsampling unit 12.

[0025] The transmitting device 81 may include an interpolation unit 13. In this case, the resampling method of the present disclosure includes the following steps: an upsampling unit 12 upsamples the sampling data of the acoustic signal; an interpolation unit 13 interpolates the upsampled data; and a downsampling unit 22 downsamples the upsampled data interpolated by the interpolation unit 13.

[0026] As described above, in the present disclosure, the resampling process is separated between the transmitting device 81 and the receiving device 82, and the interpolation process is performed in the transmitting device 81, so that the resampling process before the synthesis process in the receiving device 82 is only downsampling. This makes it possible to significantly reduce the amount of calculation in the receiving device 82. As a result, as a practical system, it is possible to perform resampling with a reduced amount of calculation on acoustic signals sampled at multiple different clocks in remote locations, and to synthesize and generate acoustic signals.

[0027] Second Embodiment An embodiment of the audio synthesis system of the present disclosure is shown in Fig. 2. In the audio synthesis system of this embodiment, a plurality of sampling / audio synthesis units 91 are connected via a network 92.

[0028] The sampling / acoustic synthesis units 91 are connected to one another via a network 92, and transmit oversampled data obtained by sampling each acoustic signal to other sampling / acoustic synthesis units 91. Furthermore, each sampling / acoustic synthesis unit 91 receives oversampled data from another sampling / acoustic synthesis unit 91, downsamples and synthesizes the data, and outputs an acoustic signal.

[0029] Furthermore, the sampling / audio synthesis unit 91 absorbs the difference between the sampling frequency of another sampling / audio synthesis unit 91 and the acoustic synthesis sampling frequency of the sampling / audio synthesis unit 91 by resampling, thereby synthesizing and generating a high-quality acoustic signal.

[0030] Furthermore, in order to reduce the amount of calculation, the sampling / audio synthesis unit 91 may perform upsampling and its interpolation processing at the sampling point among the resampling processing and transmit oversampled data. On the other hand, the sampling / audio synthesis unit 91 at the audio synthesis point may perform only downsampling according to the clock to synthesize and generate an audio signal.

[0031] 3 shows an example of the configuration of the sampling / audio synthesis unit 91. The sampling / audio synthesis unit 91 includes a sampling unit 10 and an audio synthesis unit 20. Although the figure shows an integrated system, they may also be configured as separate systems.

[0032] The sampling unit 10 includes an input / sampling unit 11, an upsampling unit 12, an interpolation unit 13, and a data transmission unit 14.

[0033] The input / sampling unit 11 samples the input acoustic signal using the sampling clock of the sampling unit 10. As a result, sampling data sampled using the sampling clock of the sampling unit 10 is generated. In this embodiment, the frequency of the sampling clock is f ｋ Here is an example where

[0034] If the input acoustic signal is an analog signal, it may be directly oversampled using a sampling clock equivalent to upsampling, in which case the subsequent upsampling unit 12 and interpolation unit 13 can be omitted.

[0035] The upsampling unit 12 upsamples the sampling data from the input / sampling unit 11. For example, the sampling data S11 to S13 shown in FIG. 4(a) are upsampled by a factor of four, as indicated by the circles in FIG. 4(b). The values ​​of the upsampling data interpolated by upsampling can usually be inserted as 0. The upsampling unit may also perform gain control, delay insertion, reverb, and other processes for controlling sound quality. Furthermore, to prevent an increase in the amount of data, the upsampling ratio may be set to several times.

[0036] The interpolation unit 13 calculates the value of the interpolated up-sampling data. The up-sampling data value can be an appropriate intermediate data value according to the input acoustic signal. For example, the calculation of the intermediate data value is performed using f ｋ A low pass filter (LPF) that cuts off frequencies equal to or greater than 1 / 2 can be used. This LPF can be configured as an FIR filter.

[0037] The data transmission unit 14 converts the data into an appropriate data format and physical format for handling on the network 92. For example, the upsampling data can be packetized using protocols such as IP (Internet Protocol), UDP (User Datagram Protocol), or RTP (Real-time Transport Protocol) and transmitted to the network 92. Sampling frequency information may also be added as accompanying information. Furthermore, in order to acquire accurate sampling frequency information, the data transmission unit 14 may have a time and clock synchronization and acquisition function such as PTP (Precision Time Protocol).

[0038] The audio synthesis unit 20 includes a data receiving unit 21 , a plurality of downsampling units 22 , a synthesis unit 23 , and an output unit 24 .

[0039] The data receiving unit 21 receives the oversampled data, extracts acoustic data from the received data, and outputs the data. ｋ It is also possible to extract and output any information that can be a clock source, such as: In addition, the received data includes acoustic data transmitted from a plurality of sampling / acoustic synthesis units 91, and each of these can be extracted and output as different acoustic data according to the accompanying information.

[0040] The following are examples of methods for obtaining information that can serve as a clock source: - Regenerating the original clock from the time, for example by stamping time information on the audio or additional packets themselves. - Regenerating the original clock from increases or decreases in the received buffer volume of audio packets by monitoring this volume. For example, if the buffer volume increases, it is known that the remote sampling clock is fast, and if the buffer volume decreases, it is known that the remote sampling clock is slow. Therefore, it is possible to regenerate the sampling clock according to increases or decreases in the buffer volume. - Regenerating the clock from other information that uses the same clock as the audio. For example, in the case of an audio signal accompanying a video signal, it is possible to regenerate the clock based on the video signal. - Measure the frequency of the remote clock in advance using a different method, and send the frequency value information to regenerate it.

[0041] The downsampling unit 22 performs downsampling using the sampling clock processed by the synthesis unit 23. For example, the downsampling unit 22 selects sampling data S21, S24, and S27 that match the sampling clock from among the upsampling data shown as S11 to S13 and S21 to S27 in FIG.

[0042] When downsampling, the downsampling points may be determined based on the original sampling clock information, or may be determined using a different method, such as from the buffer amount of packet data. If the number of original data samples is insufficient, the estimated sampling data may be used as the downsampling data by linearly interpolating the previous and next sampling data.

[0043] The synthesis unit 23 synthesizes the downsampled data obtained by the multiple downsampling units 22. This generates an output audio signal. Before or after the synthesis, gain control, delay insertion, reverb, and other processing for controlling sound quality may be performed. The sampling clock of the synthesis unit 23 may be input from an external source.

[0044] The output unit 24 outputs the output acoustic signal from the synthesis unit 23. Here, the output acoustic signal may be a digital signal, or may be converted into an analog signal and output.

[0045] Third Embodiment Fig. 5 shows an example of an audio synthesis system according to this embodiment. In this embodiment, an intermediate audio synthesis unit 93 is connected to a network 92. The intermediate audio synthesis unit 93 has a function of synthesizing audio signals upsampled by the upsampling unit 12. In the example of the second embodiment, it has a function of synthesizing oversampled data. This makes it possible to reduce the amount of oversampled data circulated over the network 92. This also makes it possible to reduce the amount of calculations in the sampling / audio synthesis unit 91.

[0046] FIG. 6 shows an example configuration of the intermediate acoustic synthesis unit 93. The intermediate acoustic synthesis unit 93 includes a data receiving unit 31, a synthesis unit 32, and a data transmitting unit 33. The data receiving unit 31 receives oversampled data transmitted from each sampling / acoustic synthesis unit 91, extracts acoustic data from the received data, and outputs the extracted acoustic data. This allows the intermediate acoustic synthesis unit 93 to acquire the upsampled data transmitted from each sampling / acoustic synthesis unit 91. The synthesis unit 23 synthesizes the multiple oversampled data received by the data receiving unit 31. If the oversampled data transmitted from each sampling / acoustic synthesis unit 91 contains clock deviation, interpolation, such as downsampling and upsampling, including insertion and removal of sampling data, may be performed before synthesis in the synthesis unit 23. The data transmitting unit 33 transmits the upsampled data synthesized by the synthesis unit 23.

[0047] The intermediate audio synthesis unit 93 may also have the function of the sampling / audio synthesis unit 91. For example, the intermediate audio synthesis unit 93 may have an upsampling function including an upsampling unit 35 and an interpolation unit 36, as shown in FIG. 7 . The functions of the upsampling unit 35 and the interpolation unit 36 ​​are similar to the upsampling unit 12 and the interpolation unit 13 provided in the sampling unit 10. By providing the intermediate audio synthesis unit 93 with the upsampling function, the oversampled data transmitted from each sampling / audio synthesis unit 91 can be appropriately synthesized and transmitted to the network 92.

[0048] 7, the intermediate audio synthesis unit 93 may have a downsampling function including a downsampling unit 34. The function of the downsampling unit 34 is similar to that of the downsampling unit 22 provided in the audio synthesis unit 20. By providing the intermediate audio synthesis unit 93 with the downsampling function, the oversampled data transmitted from each sampling / audio synthesis unit 91 can be appropriately synthesized in the synthesis unit 32.

[0049] When the intermediate audio synthesis unit 93 has at least one of an upsampling function and a downsampling function, the synthesis unit 23 may synthesize the upsampling data transmitted from each sampling / audio synthesis unit 91 to have the same frequency. This allows each sampling / audio synthesis unit 91 to receive oversampling data of the same frequency. Note that, although an example in which both an upsampling function and a downsampling function are provided is shown in FIG. 7, only one of these functions may be provided.

[0050] Furthermore, this embodiment may include a plurality of intermediate audio synthesis units 93. Also, a plurality of intermediate audio synthesis units 93 may be provided for one source sampling / audio synthesis unit 91. In other words, an n:m network configuration with n sources and m intermediate audio synthesis units 93 is also possible.

[0051] 8 shows an example of an audio synthesis system according to this embodiment. In this embodiment, an oversampling unit 94 is connected to a network 92, and a sampling / audio synthesis unit 91 can use the oversampling unit 94.

[0052] 9 shows an example of the configuration of the oversampling unit 94. The oversampling unit 94 includes a data receiving unit 41, an upsampling unit 42, an interpolation unit 43, and a data transmitting unit 44. The upsampling unit 42 and the interpolation unit 43 have the same functions as the upsampling unit 12 and the interpolation unit 13 included in the sampling unit 10.

[0053] 10 shows an example of the configuration of the sampling / audio synthesis unit 91 of this embodiment. The audio synthesis system according to this embodiment includes an oversampling unit 94, which makes it possible to omit the upsampling unit 12 and the interpolation unit 13 in the sampling / audio synthesis unit 91.

[0054] In this embodiment, the sampling unit 11 and the data transmission unit 14 are connected to the upsampling unit 12 via a communication network 92. A data receiving unit 41 of the oversampling unit 94 receives the sampling data transmitted from the sampling / audio synthesis unit 91. Then, a data transmission unit 44 of the oversampling unit 94 transmits the upsampling data generated by the upsampling unit 42 and the interpolation unit 43. Therefore, the audio synthesis system according to this embodiment can reduce the amount of calculation in the sampling / audio synthesis unit 91, which is the transmission source of the input audio signal.

[0055] According to this embodiment, the sampling / audio synthesis unit 91 is not limited to the configuration shown in Fig. 3, and a configuration without the upsampling unit 12 and the interpolation unit 13 as shown in Fig. 10 may also be adopted. Therefore, this embodiment allows the system of the present disclosure to be applied to a variety of sound sources. In particular, by arranging the oversampling unit 94 in the network close to the reception point, the amount of communication on the network 92 can be reduced.

[0056] Furthermore, this embodiment may include multiple oversampling units 94. Also, multiple oversampling units 94 may be provided for one source sampling / audio synthesis unit 91. In other words, it is also possible to have an n:m network configuration with n sources and m oversampling units 94.

[0057] (Other Embodiments) The transmitting device 81, the receiving device 82, the sampling / audio synthesis unit 91, the intermediate audio synthesis unit 93, and the oversampling unit 94 in the present disclosure can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network. Furthermore, the system of the present disclosure is not limited to acoustic signals and can be applied to any signal that undergoes resampling.

[0058] 10: Sampling unit 11: Input / sampling unit 12, 35, 42: Upsampling unit 13, 36, 43: Interpolation unit 14, 33, 44: Data transmission unit 20: Sound synthesis unit 21, 31, 41: Data reception unit 22, 34, 53: Downsampling unit 23, 32: Synthesis unit 24: Output unit 81: 82: Receiving device 91: Sampling / sound synthesis unit 92: Network 93: Intermediate sound synthesis unit 94: Oversampling unit

Claims

1. An upsampling unit that performs upsampling of sampling data, a downsampling unit that performs downsampling of the upsampling data upsampled by the upsampling unit, and a system in which the upsampling unit and the downsampling unit are connected by a communication network.

2. The system according to claim 1, wherein an intermediate acoustic synthesis unit that synthesizes the upsampling data upsampled by the upsampling unit is connected to the communication network.

3. A sampling unit that samples an acoustic signal, a data transmission unit that transmits the sampling data sampled by the sampling unit, and the system according to claim 1, wherein the sampling unit and the data transmission unit are connected to the upsampling unit and the communication network.

4. A resampling method in which an upsampling unit performs upsampling of sampling data, and a downsampling unit connected to the upsampling unit by a communication network performs downsampling of the upsampling data upsampled by the upsampling unit.

Images