Information processing device, information processing program, and information processing method

Abstract

This invention provides an information processing device that can mitigate problems such as sound quality degradation that occur when different clock frequencies are used between audio devices, using a simple mechanism. [Solution] The present invention is an information processing device connected to an audio processing unit, comprising: a plurality of buffers that hold audio data input and output via a plurality of audio devices; means for calculating the difference in the amount of audio data stored in each buffer; means for calculating the trend of the current difference in the amount of data based on the calculated difference in the amount of data stored in the past; means for comparing whether the trend of the current difference in the amount of data stored is within the expected value or within the range of the expected value, and adjusting the number of samples to be obtained from each buffer according to the result; and means for converting the obtained samples into a predetermined number of samples when samples are obtained by increasing or decreasing the predetermined number of samples supplied to the audio processing unit.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing program, and an information processing method, and can be applied to, for example, an information processing apparatus that eliminates delays caused by differences in clock frequencies occurring between voice devices.

Background Art

[0002] Conventionally, when the clock frequencies are different between voice devices such as microphones and speakers, there has been a problem that delays occur in voice playback and recording due to this.

[0003] FIG. 7 is an explanatory diagram showing problems in voice processing due to differences in clock frequencies between conventional voice devices. The voice processing apparatus 100 shown in FIG. 7 includes microphones M1 and M2 having different clock frequencies, voice buffers B1 and B2 that store the voices picked up by microphones M1 and M2, and a voice processing application A that periodically acquires voice samples from voice buffers B1 and B2 and performs predetermined voice processing.

[0004] As shown in FIG. 7, the clock frequency of microphone M1 is 16000 Hz, and the clock frequency of microphone M2 is 16001 Hz. As shown in FIG. 7(A), when voice processing is started, the voice processing apparatus 100 sequentially acquires voices from microphones M1 and M2. The acquired voices are temporarily held in voice buffers B1 and B2. At this time, more voices are acquired on the side of the device with the higher clock frequency (microphone M2).

[0005] The voice processing application A periodically acquires a fixed number of voices (voice data) from voice buffers B1 and B2 respectively. As a result, more voice data remains in the voice buffer B2 of microphone M2, which has the higher clock frequency. Thus, when the clock frequencies of the microphones are different, even if processing is attempted at the same sampling rate of 16 kHz, differences occur between the devices.

[0006] Next, as time passes from Figure 7(A) to the state shown in Figure 7(B), more audio data accumulates in the audio buffer B2 of microphone M2, causing a delay in audio processing by audio processing application A. Then, at some point, an overflow occurs, and audio processing application A recovers the accumulated data all at once (discards the delayed audio all at once).

[0007] Furthermore, if the clock frequency of microphone M2 is less than 16kHz, the opposite phenomenon (the amount of audio data accumulated in audio buffer B2 gradually decreases) may occur, and at some point, audio processing application A may become unable to acquire a fixed number of audio data samples (an underrun may occur).

[0008] As a countermeasure to the above, hardware synchronization, a method of synchronizing all audio devices to a common clock, has conventionally been used (Patent Document 1, etc.). Another possible countermeasure is to use a fixed-size buffer for storing audio data and transfer the data in a batch. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Japanese Patent Publication No. 2001-258100 [Non-patent literature]

[0010] [Non-Patent Document 1] JT-G711 PCM Encoding Method for Voice Frequency Band Signals, Appendix 1: High-Quality, Low-Computation Algorithm for Packet Loss Compensation for Standard JT-G711 (P10~) [Accessed September 12, 2024], [Online], INTERNET,<https: / / www.ttc.or.jp / application / files / 9315 / 5425 / 2192 / JT-G711v6.pdf> [Overview of the project] [Problems that the invention aims to solve]

[0011] However, the aforementioned hardware synchronization is difficult to implement, especially between heterogeneous devices, due to its complex and costly control. Furthermore, the fixed buffer method lacks flexibility and presents problems with real-time delay adjustment.

[0012] In other words, conventional technology lacks a mechanism to dynamically adapt to fluctuations in clock frequency and differences between devices. As a result, delays accumulate during long audio playback or recording sessions, leading to degradation of sound quality and synchronization issues. Additionally, audio interruptions occur during buffer overflow / underrun.

[0013] Therefore, there is a need for an information processing device, an information processing program, and an information processing method that can mitigate problems such as sound quality degradation and synchronization errors that occur when different clock frequencies are used between audio devices, using a simple mechanism. [Means for solving the problem]

[0014] The first aspect of the present invention is an information processing apparatus connected to an audio processing unit that periodically performs predetermined audio processing, comprising: a plurality of buffers that hold audio data input and output via a plurality of audio devices; a storage amount calculation means for calculating the difference in the amount of audio data stored in each buffer; a buffer storage amount estimation means for calculating the trend of the current storage amount difference based on the difference in the storage amount calculated by the storage amount calculation means and past data on the difference in the storage amount; an audio acquisition determination means for comparing whether the trend of the current storage amount difference is within the expected value or within the range of the expected value, and adjusting the number of samples to be acquired from each buffer according to the comparison result; and a sample number conversion means for converting the acquired samples to the predetermined number of samples when the audio acquisition determination means acquires samples by increasing or decreasing the predetermined number of samples supplied to the audio processing unit.

[0015] The second information processing program of the present invention is characterized in that a computer mounted on an information processing device connected to an audio processing unit that periodically performs predetermined audio processing functions as follows: a plurality of buffers that hold audio data input and output via a plurality of audio devices; a storage amount calculation means that calculates the difference in the amount of audio data stored in each buffer; a buffer storage amount estimation means that calculates the trend of the current storage amount difference based on the difference in the storage amount calculated by the storage amount calculation means and past data on the difference in the storage amount; an audio acquisition determination means that compares whether the trend of the current storage amount difference is within the expected value or within the range of the expected value, and adjusts the number of samples to be acquired from each buffer according to the comparison result; and a sample number conversion means that converts the acquired samples to the predetermined number of samples when the audio acquisition determination means acquires samples by increasing or decreasing the predetermined number of samples supplied to the audio processing unit.

[0016] The third aspect of the present invention relates to an information processing device used in an information processing device connected to an audio processing unit that periodically performs predetermined audio processing, wherein the information processing device comprises a plurality of buffers, a storage amount calculation means, a buffer storage amount estimation means, an audio acquisition determination means, and a sample number conversion means, each of the buffers holds audio data input and output via a plurality of audio devices, the storage amount calculation means calculates the difference in the storage amount of the audio data stored in each of the buffers, the buffer storage amount estimation means calculates the trend of the current storage amount difference based on the storage amount difference calculated by the storage amount calculation means and past data on the storage amount difference, the audio acquisition determination means compares whether the trend of the current storage amount difference is within the expected value or within the range of the expected value, and adjusts the number of samples to be acquired from each of the buffers according to the comparison result, and the sample number conversion means converts the acquired samples to the predetermined number of samples when samples are acquired by increasing or decreasing the predetermined number of samples supplied to the audio processing unit by the audio acquisition determination means. [Effects of the Invention]

[0017] According to the present invention, problems such as deterioration of sound quality and synchronization deviation that occur when the clock frequencies are different between audio devices can be alleviated with a simple mechanism.

Brief Description of the Drawings

[0018] [Figure 1] It is a block diagram showing the functional configuration of the audio processing apparatus according to the embodiment. [Figure 2] It is a flowchart showing the overall operation of the audio acquisition unit according to the embodiment. [Figure 3] It is a flowchart showing the operation of the buffer difference estimation unit according to the embodiment. [Figure 4] It is a block diagram showing the functional configuration of the audio processing apparatus according to the modification. [Figure 5] It is a flowchart showing the overall operation of the audio acquisition unit according to the modification. [Figure 6] It is a flowchart showing the operation of the buffer difference estimation unit according to the modification. [Figure 7] It is an explanatory diagram showing problems in audio processing due to differences in clock frequencies between conventional audio devices.

Modes for Carrying Out the Invention

[0019] (A) Main Embodiment Hereinafter, an embodiment of an information processing apparatus, an information processing program, and an information processing method according to the present invention will be described in detail with reference to the drawings.

[0020] (A-1) Configuration of the Embodiment FIG. 1 is a block diagram showing the functional configuration of the audio processing apparatus according to the embodiment.

[0021] The audio processing device 1 may be composed entirely of hardware (e.g., a dedicated chip), or it may be composed entirely of software (a program). The audio processing device 1 may also be composed by installing a program (including an information processing program of the embodiment) on a computer having a processor and memory, in addition to input devices (microphone M1, microphone M2, etc.).

[0022] As shown in Figure 1, the audio processing device 1 is a device comprising the above-mentioned microphone M (M1, M2), the above-mentioned audio buffer B (B1, B2), the audio acquisition control unit 10, and the audio processing unit 20. For example, the audio processing device 1 may be configured as a device that inputs and outputs information using only audio.

[0023] Microphones M1 and M2 are connected to the main unit of the device via a communication cable or wireless network.

[0024] Audio buffer B1 stores the audio picked up by microphone M1 (including ambient noise and environmental sounds), and audio buffer B2 stores the audio picked up by microphone M2. The capacities of audio buffers B1 and B2 are not particularly limited, but for example, they may be the maximum size that can be specified by the audio engine of the OS (Operating System) that manages the buffers (for example, a size that can store 300ms of audio).

[0025] The audio acquisition control unit 10 controls the audio (samples) extracted from microphones M1 and M2 periodically (for example, every frame: 16ms, etc.) or in an event-driven manner (for example, when audio data of 16ms or more accumulates in audio buffer B1 or audio buffer B2).

[0026] The audio acquisition control unit 10 includes a buffer storage amount difference calculation unit 11, a buffer difference estimation unit 12, an audio acquisition determination unit 13, and a sample number conversion unit 14.

[0027] The buffer storage amount difference calculation unit 11 periodically calculates the difference in the amount of audio stored in audio buffer B1 and audio buffer B2.

[0028] In theory, if microphones M1 and M2 were operating at the same clock frequency, there would be no difference in the amount of data stored in audio buffers B1 and B2. However, even if the clock frequencies of microphones M1 and M2 are the same according to the catalog specifications, individual differences in microphone components (such as crystal oscillators) and the surrounding environment may actually cause the clock frequencies of microphones M1 and M2 to differ. For example, even if microphone M1 operates at approximately 16000Hz as per the catalog specifications, microphone M2 may not operate as per the catalog specifications and may operate at 16001 (or 15999)Hz. In that case, as mentioned above, there will be a difference in the amount of data stored in audio buffers B1 and B2.

[0029] The buffer difference estimation unit 12 performs processing to absorb the difference in the amount of audio stored in audio buffer B1 and audio buffer B2, which was calculated by the buffer storage amount difference calculation unit 11.

[0030] In other words, since the amount of buffers accumulated fluctuates up and down when viewed on a frame-by-frame basis, it is not advisable to simply determine the amount of each buffer's accumulated amount based on the value calculated by the buffer accumulation difference calculation unit 11 (buffer difference on a frame-by-frame basis). Therefore, the buffer difference estimation unit 12 calculates the trend of the difference in the current accumulated amounts of both buffers using statistical processing to address this fluctuation in the buffer difference.

[0031] The audio acquisition determination unit 13 determines whether the value calculated by the buffer difference estimation unit 12 (the adopted value described later) is as expected, and adjusts the amount of audio acquired from audio buffer B1 and audio buffer B2 based on the determination result. That is, if there is no difference between audio buffer B1 and audio buffer B2, the synchronization is in place and there is no problem. On the other hand, if it is not as expected, a synchronization discrepancy has occurred, and there is a possibility that an overflow or underrun will occur at some point, so the audio is increased or decreased (in real time) each time it is determined that it is not as expected.

[0032] The sample count conversion unit 14 converts the sample count of the acquired samples when the audio acquisition determination unit 13 increases or decreases the amount of audio and acquires samples (audio) from the audio buffer B1 (or audio buffer B2). The sample count conversion unit 14 provides the converted samples to the audio processing unit 20.

[0033] The detailed processing of the buffer storage difference calculation unit 11, buffer difference estimation unit 12, audio acquisition determination unit 13, and sample number conversion unit 14 described above will be explained in the operation section.

[0034] The audio processing unit 20 performs predetermined audio processing on the audio picked up by microphones M1 and M2. In this embodiment, the audio processing performed by the audio processing unit 20 is not particularly limited.

[0035] (A-2) Operation of the embodiment Next, the operation of the voice processing device 1 of this embodiment, which has the configuration described above, will be explained. This embodiment is characterized by the operation of the voice acquisition control unit 10, so the following explanation will focus on the operation of the voice acquisition control unit 10.

[0036] Figure 2 is a flowchart showing the overall operation of the audio acquisition unit according to the embodiment.

[0037] When the acoustic signals (speech) picked up by microphones M1 and M2 are stored in audio buffers B1 and B2, respectively, the processing shown in Figure 2 begins.

[0038] <s101> The buffer storage amount difference calculation unit 11 periodically calculates the difference in the amount of audio stored in audio buffer B1 and audio buffer B2. For example, if the audio processing unit 20 processes at a sampling rate of 16KHz, it calculates the difference in the amount of audio stored in audio buffers B1 and B2 at each frame period of a predetermined length (e.g., 16ms).

[0039] <s102> The buffer difference estimation unit 12 calculates the trend of the difference in the amount of audio stored in audio buffers B1 and B2.

[0040] Figure 3 is a flowchart showing the operation of the buffer difference estimation unit according to the embodiment.

[0041] <s102-1> The buffer difference estimation unit 12 calculates the median of the buffer differences over the past few frames. For example, the buffer difference estimation unit 12 calculates the median of the difference in the amount of audio stored in audio buffers B1 and B2 for 7 frames (the value calculated in step S101 above).

[0042] <s102-2> Furthermore, the buffer difference estimation unit 12 calculates the average of the median buffer differences of frames within a certain period in the past. For example, the buffer difference estimation unit 12 calculates the average of the median values ​​(the median values ​​calculated in step S102-1 above) for the past 25 frames.

[0043] <s102-3> The buffer difference estimation unit 12 determines whether the difference between the mean value calculated in step S102-2 and the median value calculated in step S102-1 is within a certain range (within a certain range). If the difference between the calculated mean value and median value is within a certain range, the buffer difference estimation unit 12 proceeds to the next step S102-4. If the difference between the calculated mean value and median value exceeds a certain range, the unit proceeds to step S102-5, which will be described later.

[0044] <s102-4> If the buffer difference estimation unit 12 determines in step S102-3 above that the difference between the mean and median is below a certain range, it estimates the median calculated in step S102-1 above as the difference between the current accumulated amounts of audio buffer B1 and audio buffer B2 (and adopts this value).

[0045] <s102-5> On the other hand, if the buffer difference estimation unit 12 determines in step S102-3 above that the difference between the mean and the median exceeds a certain range, it determines whether the median calculated in step S102-1 above has been rejected (in other words, determined to be NO in step S102-3 above) for a certain number of consecutive times or more.

[0046] If the buffer difference estimation unit 12 rejects the median calculated in step S102-1 for a certain number of consecutive times, it proceeds to step S102-4, treating the difference in the current accumulated amounts of audio buffer B1 and audio buffer B2 as an exception, and adopts the median calculated in step S102-1 as the value obtained by the current accumulated amounts of audio buffer B1 and audio buffer B2. On the other hand, if the buffer difference estimation unit 12 does not reject the median calculated in step S102-1 for a certain number of consecutive times, it proceeds to the next step S102-6.

[0047] <s102-6> If the buffer difference estimation unit 12 determines in step S102-5 above that the median value has not been rejected for a certain number of consecutive times, it will, as an exception, use the previously adopted value (1 frame (16ms) ago) as the difference in the accumulated amounts of audio buffer B1 and audio buffer B2.

[0048] <s103> The audio acquisition determination unit 13 compares the adopted value calculated by the buffer difference estimation unit 12 with the expected value. Here, the expected value may be a fixed value such as "0", or it may be within a certain range (-1 to +1), and there are no particular restrictions on the specific value. Note that an expected value of "0" would be set, for example, when it is necessary to perform audio processing that strictly requires the same amount of data stored in audio buffer B1 and audio buffer B2 (audio processing without synchronization discrepancies). In any case, whether to set a fixed value or a certain range as the expected value will depend on the content and environment of each audio processing.

[0049] If the adopted value calculated by the buffer difference estimation unit 12 is the expected value (or within the range of the expected value), the voice acquisition determination unit 13 proceeds to the next step S104. On the other hand, if the adopted value calculated by the buffer difference estimation unit 12 is not the expected value (or within the range of the expected value), it proceeds to step S105, which will be described later.

[0050] <s104> If the audio acquisition control unit 10 determines in step S103 that the adopted value is the expected value (or within the range of the expected value), it acquires a predetermined number of samples (for example, 256 samples) from audio buffer B1 and audio buffer B2 respectively and outputs them directly to the audio processing unit 20.

[0051] <s105> On the other hand, if the audio acquisition control unit 10 determines in step S103 that the adopted value is not the expected value (or within the range of the expected value), it acquires samples by increasing or decreasing a predetermined number of samples from audio buffer B1 and audio buffer B2, respectively, and outputs them to the sample number conversion unit 14.

[0052] For example, if audio buffer B2 has more samples than audio buffer B1 (leading to an overflow), one extra sample (257 in total) is taken from audio buffer B2 than the usual 256. On the other hand, if audio buffer B2 has fewer samples than audio buffer B1 (leading to an underrun), one less sample (255 in total) is taken from audio buffer B2 than the usual 256.

[0053] <s106> The sample count conversion unit 14 performs a sample count conversion on the audio (samples) acquired by increasing or decreasing the sample count in step S105 described above. The specific processing for sample count conversion is not particularly limited, but for example, triangular weighting (for example, the method described in Non-Patent Document 1) can be used.

[0054] For example, in the overflow case described above, if 257 samples are obtained from audio buffer B2, they are converted to 256 samples according to equation (1) below. y(i) = (257-i) / 256 * x(i) + (i+1) / 256 * x(i+1) …(1)

[0055] In equation (1) above, y represents the transformed sample and x represents the original sample. By substituting values ​​from "0" to "255" for i, we can obtain transformed samples from y(0) to y(255) (a total of 256 samples).

[0056] Furthermore, for example, in the underrun case described above, if 255 samples are obtained from audio buffer B2, they are converted to 256 samples according to equations (2) to (4) below. y(i): (When it is sample number 0) y(0) = x(0) …(2) y(i): (For samples 1 to 254) y(i) = (255-i) / 255 * x(i) + i / 255 * x(i-1)…(3) y(i): (When sample number 255 is used) y(255) = x(254) …(4)

[0057] Similarly to the above, y represents the transformed sample and x represents the sample before transformation. Using equations (2) to (4) above, we can obtain the transformed samples y(0) to y(255) (a total of 256 samples).

[0058] The above example shows a sampling conversion when processing at 16000kHz sampling × 16ms frames (=256 samples / frame). However, if processing at, for example, 48000kHz sampling × 16ms frames (=768 samples / frame), the above method can be applied after changing the values ​​(fixed values) in each of the above formulas.

[0059] The sample conversion unit 14 outputs the converted samples to the audio processing unit 20, and the series of processes ends.

[0060] (A-3) Effects of the Embodiment This embodiment provides the following effects.

[0061] The audio processing device 1 periodically monitors the difference in the amount of data stored in audio buffer B1 and audio buffer B2 (after each frame is processed). If the difference between the two buffers does not meet expectations, the device adjusts the number of samples taken from the buffers so that the amounts of data stored in audio buffer B1 and audio buffer B2 become equal. This prevents synchronization errors.

[0062] At the same time, overflows and underruns are prevented from occurring, and users no longer perceive the audio as suddenly cutting out. In this embodiment, the increased or decreased sample count is handled by triangular weighting, resulting in, for example, instantaneous output of audio at 257 / 256 times the normal speed and 255 / 256 times the normal speed. However, this is far more convenient for users than having the audio cut out (it has virtually no effect on human hearing).

[0063] (B) Other embodiments The present invention is not limited to the embodiments described above, and modified embodiments such as those exemplified below can also be cited.

[0064] (B-1) In the above embodiment, an example of application to two microphones was shown, but the present invention can also be applied to an audio processing device with three or more microphones. In the case of three or more microphones, the difference in the amount of audio buffer stored in each microphone device can be determined using the same method as described above.

[0065] (B-2) Structure of the modified example Figure 4 is a block diagram showing the functional configuration of the modified audio processing device. Note that the following description of the configuration of the modified (B-2) is based on the configuration of the (A-1) embodiment, so configurations that are the same as those of the (A-1) embodiment will be omitted.

[0066] The voice processing device 101 has a configuration based on the voice processing device 10 (Figure 1).

[0067] The audio processing device 101 may be configured, for example, by installing a program (including an information processing program according to the embodiment) on a computer having a processor and memory, in addition to input devices (microphone M1, microphone M2, microphone M3, etc.).

[0068] As shown in Figure 4, the audio processing device 1 is a device that includes the above-mentioned microphone M (M1, M2, M3), the above-mentioned audio buffer B (B1, B2, B3), an audio acquisition control unit 1010, and an audio processing unit 1020.

[0069] Microphones M1, M2, and M3 are connected to the main unit of the device via a communication cable or wireless network.

[0070] Furthermore, the audio buffer B3 stores the audio picked up by microphone M3.

[0071] The voice acquisition control unit 1010 is configured based on the voice acquisition control unit 10 (Figure 1).

[0072] The audio acquisition control unit 1010 controls the audio (samples) extracted from microphones M1, M2, and M3 periodically (for example, every frame: 16ms, etc.) or in an event-driven manner (for example, when audio data of 16ms or more accumulates in audio buffer B1, audio buffer B2, or audio buffer B3).

[0073] The voice acquisition control unit 1010 includes a buffer storage amount calculation unit 1011, a buffer difference estimation unit 1012, a voice acquisition determination unit 1013, and a sample number conversion unit 141014.

[0074] The buffer storage amount calculation unit 1011 is configured based on the buffer storage amount difference calculation unit 11 (Figure 1).

[0075] The buffer storage amount calculation unit 1011 periodically calculates the amount of audio stored in audio buffer B1, audio buffer B2, and audio buffer B3.

[0076] The buffer difference estimation unit 1012 has a configuration based on the buffer difference estimation unit 12 (Figure 1).

[0077] The buffer difference estimation unit 1012 performs a process to absorb the difference in the amount of audio stored in audio buffers B1, B2, and B3, which are calculated by the buffer storage amount calculation unit 1011.

[0078] The voice acquisition and determination unit 1013 has a configuration based on the voice acquisition and determination unit 13 (Figure 1).

[0079] The audio acquisition determination unit 1013 determines whether the values ​​calculated by the buffer difference estimation unit 1012 (the second adopted value, third adopted value, and fourth adopted value, which will be described later as adopted values) are as expected, and based on the determination result, it increases or decreases (adjusts) the audio acquired from audio buffers B1, B2, and B3.

[0080] The sample number conversion unit 1014 has a configuration based on the sample number conversion unit 14 (Figure 1).

[0081] The sample count conversion unit 1014 converts the sample count of the acquired samples when the audio acquisition determination unit 1013 increases or decreases the amount of audio and acquires samples (audio) from the audio buffer B1 (or audio buffers B2, audio buffers B3). The sample count conversion unit 1014 provides the converted samples to the audio processing unit 1020.

[0082] The detailed processing of the buffer storage amount calculation unit 1011, buffer difference estimation unit 1012, audio acquisition determination unit 1013, and sample number conversion unit 1014 described above will be explained in the operation section.

[0083] The voice processing unit 1020 is configured based on the voice processing unit 20 (Figure 1).

[0084] The audio processing unit 1020 performs predetermined audio processing on the audio picked up by microphones M1, M2, and M3.

[0085] (B-3) Actions of the torture Next, the operation of the modified audio processing device 101 having the above configuration will be described. This modified version is characterized by the operation of the audio acquisition control unit 1010, so the following description will focus on the operation of the audio acquisition control unit 1010. Note that the following description of the operation of the modified version (B-3) is based on the operation of the embodiment (A-2), so operations that are the same as those of the embodiment (A-2) will be omitted.

[0086] Figure 5 is a flowchart showing the overall operation of the audio acquisition unit in the modified example.

[0087] When the acoustic signals (speech) picked up by microphones M1, M2, and M3 are stored in audio buffers B1, B2, and B3, respectively, the processing shown in Figure 5 begins.

[0088] <s10101> The buffer storage amount calculation unit 1011 periodically calculates the amount of audio stored in audio buffers B1, B2, and B3. For example, if the audio processing unit 1020 processes at a sampling rate of 16 kHz, it calculates the amount of audio stored in audio buffers B1, B2, and B3 at intervals of a predetermined frame period (e.g., 16 ms).

[0089] <s10102> The buffer difference estimation unit 1012 calculates the trend of the amount of audio stored in audio buffers B1, B2, and B3.

[0090] Figure 6 is a flowchart showing the operation of the buffer difference estimation unit according to the embodiment.

[0091] <s10102-1> The buffer difference estimation unit 1012 calculates the median buffer size for the past few frames for each of the three devices.

[0092] <s10102-2> Furthermore, the buffer difference estimation unit 1012 calculates the average of the median buffer values ​​of frames within a certain period in the past.

[0093] <s10102-3> The buffer difference estimation unit 1012 determines whether the difference between the average value calculated in step S10102-2 and the median value calculated in step S10102-1 is below a certain range (within a certain range).

[0094] Here, the buffer difference estimation unit 1012 determines the difference between the median and the moving average of the median related to audio buffer B1, the difference between the median and the moving average of the median related to audio buffer B2, and the difference between the median and the moving average of the median related to audio buffer B3.

[0095] The buffer difference estimation unit 1012 proceeds to the next step S10102-4 if the difference between the calculated median and the moving average of the median is below a certain range, and proceeds to step S10102-5, which will be described later, if the difference between the calculated median and the moving average of the median exceeds a certain range.

[0096] <s10102-4> If the buffer difference estimation unit 1012 determines in step S10102-3 above that the difference between the median and the moving average of the median is below a certain range, it estimates the median calculated in step S10102-1 above as the current accumulated amount of audio buffer B1, audio buffer B2, and audio buffer B3 (this will be the first adopted value).

[0097] <s10102-5> On the other hand, if the buffer difference estimation unit 1012 determines in step S10102-3 above that the difference between the mean and the median exceeds a certain range, it determines whether the median calculated in step S10102-1 above has been rejected (in other words, determined to be NO in step S10102-3 above) for a certain number of consecutive times or more.

[0098] Here, the buffer difference estimation unit 1012 determines the difference between the median and the moving average of the median related to audio buffer B1, the difference between the median and the moving average of the median related to audio buffer B2, and the difference between the median and the moving average of the median related to audio buffer B3.

[0099] If the buffer difference estimation unit 1012 rejects the median calculated in step S10102-1 for a certain number of consecutive times, it proceeds to step S10102-4, assuming that the current accumulated amounts of audio buffers B1, B2, and B3 are no exception, and sets the median calculated in step S10102-1 as the first accepted value for the current accumulated amounts of audio buffers B1, B2, and B3. On the other hand, if the buffer difference estimation unit 1012 does not reject the median calculated in step S10102-1 for a certain number of consecutive times, it proceeds to the next step S10102-6.

[0100] <s10102-6> If the buffer difference estimation unit 1012 determines in step S10102-5 above that the median has not been rejected for a certain number of consecutive times, it makes an exception and uses the first accepted value from the previous time (1 frame (16ms) ago) as the accumulated amount of audio buffer B1, audio buffer B2, and audio buffer B3.

[0101] <s10103> Step S10103 consists of Step S10103-1 and Step S10103-2.

[0102] <s10103-1> The buffer difference estimation unit 1012 calculates the following three differences (referred to as the second adopted value, the third adopted value, and the fourth adopted value).

[0103] The buffer difference estimation unit 1012 calculates the difference between the adopted value of audio buffer B1 calculated by the buffer difference estimation unit 1012 and the average value of the adopted values ​​of the three audio buffers (audio buffer B1, audio buffer B2, and audio buffer B3) calculated by the buffer difference estimation unit 1012 (this is called the second adopted value).

[0104] The buffer difference estimation unit 1012 calculates the difference between the adopted value of audio buffer B2 calculated by the buffer difference estimation unit 1012 and the average value of the adopted values ​​of the three audio buffers (audio buffer B1, audio buffer B2, and audio buffer B3) calculated by the buffer difference estimation unit 1012 (this is called the third adopted value).

[0105] The buffer difference estimation unit 1012 calculates the difference between the adopted value of audio buffer B3 calculated by the buffer difference estimation unit 1012 and the average value of the adopted values ​​of the three audio buffers (audio buffer B1, audio buffer B2, and audio buffer B3) calculated by the buffer difference estimation unit 1012 (this is the fourth adopted value).

[0106] <s10103-2> The voice acquisition determination unit 1013 compares the following three accepted values ​​with the expected value.

[0107] The audio acquisition determination unit 1013 compares the second adopted value calculated by the buffer difference estimation unit 1012 with the expected value.

[0108] The audio acquisition determination unit 1013 compares the third adopted value calculated by the buffer difference estimation unit 1012 with the expected value.

[0109] The audio acquisition determination unit 1013 compares the fourth adopted value calculated by the buffer difference estimation unit 1012 with the expected value.

[0110] Here, the expected value can be a fixed value such as "0", or it can be within a certain range (-1 to +1), and there are no particular restrictions on the specific value. An expected value of "0" would be set, for example, when performing audio processing that strictly requires the same storage amounts in audio buffers B1, B2, and B3 (audio processing without synchronization issues). In any case, whether to set a fixed value or a certain range as the expected value depends on the content and environment of each audio processing task.

[0111] If the adopted value (second adopted value, third adopted value, or fourth adopted value) calculated by the buffer difference estimation unit 1012 is within the expected value range, the voice acquisition determination unit 1013 proceeds to the next step S10104. On the other hand, if the adopted value (second adopted value, third adopted value, or fourth adopted value) calculated by the buffer difference estimation unit 12 is not within the expected value range, the unit proceeds to step S10105, which will be described later.

[0112] <s10104> If the audio acquisition control unit 1010 determines in step S10103 that the selected value (second selected value, third selected value, or fourth selected value) is within the expected range (or within the range of the expected value), it acquires a predetermined number of samples (for example, 256 samples) from audio buffers B1, B2, and B3 respectively and outputs them directly to the audio processing unit 1020.

[0113] <s10105> On the other hand, if the audio acquisition control unit 1010 determines in step S10103 that the adopted value (second adopted value, third adopted value, or fourth adopted value) is not the expected value (or within the range of the expected value), it acquires samples by increasing or decreasing a predetermined number of samples from audio buffers B1, B2, and B3, respectively, and outputs them to the sample number conversion unit 1014.

[0114] For example, if audio buffer B2 has more samples than audio buffers B1 and B3 (leading to an overflow), one extra sample (257 in total) is taken from audio buffer B2 than the usual 256. On the other hand, if audio buffer B2 has fewer samples than audio buffers B1 and B3 (leading to an underrun), one less sample (255 in total) is taken from audio buffer B2 than the usual 256.

[0115] <s10106> The sample count conversion unit 1014 performs a sample count conversion on the audio (samples) acquired by increasing or decreasing the sample count in step S10105 described above. The specific processing for sample count conversion is not particularly limited, but for example, triangular weighting (for example, the method described in Non-Patent Document 1) can be used.

[0116] The sample conversion unit 1014 outputs the converted samples to the audio processing unit 1020, thus ending the series of processes.

[0117] (B-4) In the above embodiment, the flowchart in Figure 3 was used as an example to explain the process by which the buffer difference estimation unit 12 calculates the trend of the difference in the amount of audio stored in audio buffers B1 and B2. However, the contents of the process in the flowchart in Figure 3 may be partially changed, omitted, or added to. In addition, the buffer difference estimation unit 12 may calculate the trend of the difference in the amount of audio stored in audio buffers B1 and B2 using a method other than that shown in Figure 3. In any case, the process by which the buffer difference estimation unit 12 calculates the trend of the difference in the amount of audio stored in audio buffers B1 and B2 is not particularly limited.

[0118] (B-5) In the above embodiment, a microphone was used as an example of an audio device, but the present invention may be applied to an audio processing device that is not limited to a microphone, such as a speaker. [Explanation of Symbols]

[0119] 1...Speech processing unit, 10...Speech acquisition control unit, 11...Buffer storage amount difference calculation unit, 12...Buffer difference estimation unit, 13...Speech acquisition determination unit, 14...Sample number conversion unit, 20...Speech processing unit, B1, B2...Speech buffer, M1, M2...Microphone.

Claims

1. An information processing device connected to a voice processing unit that periodically performs predetermined voice processing, Multiple buffers that hold audio data input and output via multiple audio devices, A means for calculating the difference in the amount of audio data stored in each of the aforementioned buffers, A buffer storage amount estimation means calculates the trend of the current storage amount difference based on the difference in the storage amount calculated by the storage amount calculation means and past data on the difference in the storage amount, A voice acquisition determination means compares whether the trend of the difference in the current accumulated amount is within the expected value or within the expected value range, and adjusts the number of samples to be acquired from each buffer according to the comparison result. When the voice acquisition determination means acquires samples by increasing or decreasing the predetermined number of samples supplied to the voice processing unit, a sample number conversion means converts the acquired samples to the predetermined number of samples. An information processing device characterized by having the following features.

2. The information processing apparatus according to claim 1, characterized in that the audio processing is performed for each frame period obtained by dividing the audio data into frame periods of a predetermined length, and each component of the information processing apparatus is executed for each frame period.

3. A computer installed in an information processing device that connects to a voice processing unit that periodically performs predetermined voice processing, Multiple buffers that hold audio data input and output via multiple audio devices, A means for calculating the difference in the amount of audio data stored in each of the aforementioned buffers, A buffer storage amount estimation means calculates the trend of the current storage amount difference based on the difference in the storage amount calculated by the storage amount calculation means and past data on the difference in the storage amount, A voice acquisition determination means compares whether the trend of the difference in the current accumulated amount is within the expected value or within the expected value range, and adjusts the number of samples to be acquired from each buffer according to the comparison result. When the voice acquisition determination means acquires samples by increasing or decreasing the predetermined number of samples supplied to the voice processing unit, a sample number conversion means converts the acquired samples to the predetermined number of samples. An information processing program characterized by its ability to function in this way.

4. An information processing method used in an information processing device connected to an audio processing unit that periodically performs predetermined audio processing, The information processing device includes multiple buffers, a means for calculating the amount of data stored, a means for estimating the amount of data stored in the buffer, a means for determining whether data is acquired, and a means for converting the number of samples. Each of the aforementioned buffers holds audio data that is input and output via multiple audio devices, The storage amount calculation means calculates the difference in the amount of audio data stored in each buffer, The buffer storage amount estimation means calculates the trend of the current storage amount difference based on the difference in the storage amount calculated by the storage amount calculation means and past data on the difference in the storage amount. The voice acquisition determination means compares whether the trend of the difference in the current accumulated amount is within the expected value or within the expected value range, and adjusts the number of samples to be acquired from each buffer according to the comparison result. The sample number conversion means converts the acquired samples to the predetermined number of samples when the number of samples acquired by the voice acquisition determination means increases or decreases the number of samples supplied to the voice processing unit by a predetermined number of samples. An information processing method characterized by the following:

Images