Speech processing device, speech processing method, and speech processing program

Abstract

The interval detection unit (102) detects a mixed speech interval from the time-series audio data, which is a time range in which a mixed speech exists in which multiple voices are mixed. The audio separation unit (103) separates the mixed speech in the mixed speech interval into the multiple voices.

Description

Technical Field

[0001] This disclosure relates to voice processing.

Background Art

[0002] As a technology related to this disclosure, there is the technology of Patent Document 1. The technology of Patent Document 1 separates an audio signal in which voices of multiple speakers are mixed into audio signals of the voices of each speaker.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the time-series signal output by the voice separation technology represented by the technology of Patent Document 1, processing distortion occurs. When another process (hereinafter referred to as post-processing) is performed after voice separation, this processing distortion may have an adverse effect on the post-processing. For example, when voice recognition is performed as post-processing after voice separation, the voice recognition accuracy may deteriorate due to processing distortion.

[0005] The main object of this disclosure is to reduce processing distortion due to voice separation.

Means for Solving the Problems

[0006] The voice processing apparatus according to this disclosure includes an interval detection unit that detects a mixed voice interval, which is a time range in which a mixed voice in which multiple voices are mixed exists, from time-series voice data, and a voice separation unit that separates the mixed voice in the mixed voice interval into the multiple voices.

Effects of the Invention

[0007] According to this disclosure, processing distortion caused by audio separation can be reduced. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing an example of the functional configuration of the audio processing device according to Embodiment 1. [Figure 2] A diagram showing an example of the hardware configuration of the audio processing device according to Embodiment 1. [Figure 3] A diagram showing an example of operation of the audio processing device according to Embodiment 1. [Figure 4] A diagram showing an example of operation of the audio processing device according to Embodiment 1. [Figure 5] A diagram showing an example of operation of the audio processing device according to Embodiment 1. [Figure 6] A diagram showing an example of operation of the audio processing device according to Embodiment 1. [Figure 7] A diagram showing an example of operation of the audio processing device according to Embodiment 1. [Figure 8] A diagram showing an example of the functional configuration of the audio processing device according to Embodiment 2. [Figure 9] A diagram showing an example of the functional configuration of the audio processing device according to Embodiment 3. [Figure 10] A diagram showing an example of the functional configuration of the audio processing device according to Embodiment 4. [Figure 11] A diagram showing an example of the functional configuration of the audio processing device according to Embodiment 5. [Figure 12] A diagram showing an example of operation of the audio processing device according to Embodiment 5. [Figure 13] A diagram showing an example of the functional configuration of the audio processing device according to Embodiment 6. [Figure 14] A diagram showing an example configuration of the model generation unit according to Embodiment 6. [Modes for carrying out the invention]

[0009] The embodiments will be described below with reference to the drawings. In the following description of the embodiments and in the drawings, the same reference numerals indicate the same part or a corresponding part.

[0010] Embodiment 1. ***Description of the Configuration*** FIG. 1 shows a functional configuration example of the audio processing apparatus 100 according to the present embodiment. Also, FIG. 2 shows a hardware configuration example of the audio processing apparatus 100 according to the present embodiment. The audio processing apparatus 100 is a computer. The audio processing apparatus 100 reduces processing distortion due to audio separation by limiting the target range of audio separation. More specifically, the audio processing apparatus 100 reduces processing distortion due to audio separation by limiting the target range of audio separation to a time range in which there is a mixed audio in which a plurality of audios are mixed. Note that the operation procedure of the audio processing apparatus 100 corresponds to an audio processing method. Also, a program for realizing the operation of the audio processing apparatus 100 corresponds to an audio processing program.

[0011] First, a hardware configuration example of the audio processing apparatus 100 shown in FIG. 2 will be described.

[0012] As hardware, the audio processing apparatus 100 includes a processor 901, a main storage device 902, an auxiliary storage device 903, and a communication device 904. Also, as functional components, the audio processing apparatus 100 includes a feature amount extraction unit 101, an interval detection unit 102, an audio separation unit 103, an audio restoration unit 104, and an audio connection unit 105 shown in FIG. 1. The functional components in FIG. 1 are realized by, for example, a program. Programs for realizing these functions are stored in the auxiliary storage device 903. These programs are loaded from the auxiliary storage device 903 to the main storage device 902. Then, the processor 901 executes these programs to perform the operations of the functional components in FIG. 1. FIG. 2 schematically shows a state in which the processor 901 is executing a program for realizing the functions of the functional components in FIG. 1. Although not shown in FIG. 2, the audio processing apparatus 100 may have an input / output device. Input / output devices include, for example, a mouse, keyboard, recording medium reader, recording medium writer, and display.

[0013] Next, we will describe an example of the functional configuration of the audio processing device 100 shown in Figure 1.

[0014] The feature extraction unit 101 acquires the mixed audio signal sequence 200. The feature extraction unit 101 acquires the mixed audio signal sequence 200 from a communication network, for example, via a communication device 904. Alternatively, the feature extraction unit 101 may acquire the mixed audio signal sequence 200 from a recording medium reading device of an input / output device. The mixed audio signal sequence 200 is a sequence of audio signals in time series. The mixed speech signal train 200 includes a time range in which a single speech signal exists (hereinafter referred to as the single speech section) and a time range in which a mixed speech signal exists (hereinafter referred to as the mixed speech section). A single voice is the voice of a single speaker. A mixed voice is a mixture of multiple voices from multiple speakers.

[0015] The feature extraction unit 101 extracts features of the audio signals contained in the mixed audio signal sequence 200. For example, the feature extraction unit 101 extracts the amplitude spectrum obtained by the short-time Fourier transform as a feature. Alternatively, the feature extraction unit 101 may extract features obtained by a neural network such as a multilayer perceptron or Conv1-D. The feature extraction unit 101 then outputs feature data 110, which shows the extracted features in a time series, to the interval detection unit 102. In the feature data 110, the features of a single speech are shown in single speech intervals, and the features of a mixed speech are shown in mixed speech intervals. Furthermore, the feature extraction unit 101 outputs the mixed speech signal sequence 200 to the section detection unit 102.

[0016] The interval detection unit 102 acquires feature data 110 from the feature extraction unit 101 as time-series audio data. The interval detection unit 102 then analyzes the feature quantities and detects the applicable interval from the feature quantity data 110. The applicable interval consists of a mixed speech interval, the preceding interval, and the following interval. The mixed speech interval is the time range in which mixed speech exists in the feature data 110. The preceding interval is the time range in the time series of feature data 110 that is immediately preceding the mixed speech interval and contains a single speech element. The immediate post-interval is the time range in the time series of feature data 110 that is immediately after the mixed speech interval and contains a single speech element. The interval detection unit 102 outputs the applicable interval feature quantity 120 to the speech separation unit 103. The applicable interval feature quantity 120 is the feature quantity of the applicable interval. Furthermore, the section detection unit 102 outputs the mixed audio signal sequence 200 and the unmixed audio section definition information 1200 to the audio connection unit 105. The unmixed speech section definition information 1200 is information that defines the unmixed speech section in the mixed speech signal train 200. The unmixed speech section is the section in the mixed speech signal train 200 other than the mixed speech section. Specifically, the unmixed speech section includes a preceding section and a succeeding section. The preceding section is the time range located before the mixed speech section in the time series of the mixed speech signal train 200. The succeeding section is the time range located after the mixed speech section in the time series of the mixed speech signal train 200. The processing performed by the interval detection unit 102 corresponds to interval detection processing.

[0017] The audio separation unit 103 obtains the applicable interval feature quantity 120 from the interval detection unit 102. The speech separation unit 103 uses the applicable interval feature quantity 120 to separate the feature quantity of the mixed speech in the mixed speech interval into the feature quantities of multiple speeches included in the mixed speech. Thus, strictly speaking, the speech separation unit 103 deals with the feature quantity of speech, not the speech itself, but for the sake of simplicity, it is sometimes explained as if the speech separation unit 103 deals with the speech itself. In other words, the separation of the feature quantity of mixed speech into the feature quantities of multiple speeches by the speech separation unit 103 is sometimes explained as the separation of mixed speech into multiple speeches. In the following, the multiple audio samples obtained by the audio separation unit 103 will be referred to as "separated audio." In the processing of the audio separation unit 103, the term "separated audio" strictly refers to the feature quantities of the separated audio. Similarly, the terms "forward-connected separated audio," "backward-connected separated audio," "immediately preceding single audio," and "immediately following single audio," described later, also strictly refer to the feature quantities of the respective audio samples in the processing of the audio separation unit 103.

[0018] The speech separation unit 103 uses the applicable interval feature quantity 120 to extract a forward-connected separated speech from multiple separated speeches. The forward-connected separated speech is a separated speech that is estimated to be the same speech as the preceding single speech. The preceding single speech is a single speech that exists in the preceding interval. Furthermore, the audio separation unit 103 uses the applicable interval feature quantity 120 to extract a subsequent connected separated audio from multiple separated audios. The subsequent connected separated audio is a separated audio that is estimated to be the same audio as the immediately following single audio. The immediately following single audio is a single audio that exists in the immediately following interval. The audio separation unit 103 outputs the separated audio information 130 to the audio restoration unit 104. The separated audio information 130 includes the feature quantities of each separated audio, an identifier for the separated audio corresponding to the preceding connected separated audio, an identifier for the separated audio corresponding to the following connected separated audio, the feature quantities of the immediately preceding single audio, and the feature quantities of the immediately following single audio. The processing performed by the audio separation unit 103 corresponds to audio separation processing.

[0019] The audio restoration unit 104 acquires separated audio information 130 from the audio separation unit 103. The audio reconstruction unit 104 reconstructs the audio signal of each separated audio from the feature quantities of each separated audio contained in the separated audio information 130. The audio reconstruction unit 104 also reconstructs the audio signal of the preceding single audio from the feature quantities of the preceding single audio. Furthermore, the audio reconstruction unit 104 reconstructs the audio signal of the following single audio from the feature quantities of the following single audio. The audio restoration unit 104 then outputs the restored audio information 140 to the audio connection unit 105. The restored audio information 140 includes the audio signals of each restored separated audio, the audio signal of the single audio immediately preceding the restored audio, the audio signal of the single audio immediately following the restored audio, the identifier of the separated audio corresponding to the forward connected separated audio, and the identifier of the separated audio corresponding to the backward connected separated audio.

[0020] The audio connection unit 105 acquires the mixed audio signal sequence 200 and the unmixed audio section definition information 1200 from the section detection unit 102. Furthermore, the audio connection unit 105 acquires restored audio information 140 from the audio restoration unit 104. The audio connection unit 105 uses the unmixed audio section definition information 1200 and the restored audio information 140 to connect the audio signals of the separated audio to the audio signals of the single audio within the mixed audio signal sequence 200. More specifically, the audio connection unit 105 connects the audio signal of the forward-connected separated audio to the audio signal of the forward single audio. The forward single audio is a single audio that exists in the forward section. The audio connection unit 105 can identify the forward single audio by referring to the unmixed audio section definition information 1200. Furthermore, the audio connection unit 105 connects the audio signal of the rear-connected separated audio to the audio signal of the rear single audio. The rear single audio is a single audio that exists in the rear section. The audio connection unit 105 can identify the rear single audio by referring to the unmixed audio section definition information 1200. The audio connection unit 105 generates a separated audio signal sequence 300 consisting only of single audio signals as a result of connecting the audio signals. The audio connection unit 105 then outputs the separated audio signal sequence 300. For example, the audio connection unit 105 transmits the separated audio signal train 300 to the source of the mixed audio signal train 200 via the communication device 904. Alternatively, the audio connection unit 105 may write the separated audio signal train 300 to the recording medium via the recording medium writing device of the input / output device.

[0021] ***Explanation of operation*** Next, an example of the operation of the voice processing device 100 according to this embodiment will be described with reference to Figures 3 to 7.

[0022] Here, we assume that the feature data 110 shown in Figure 3(a) was generated by the feature extraction unit 101. The interval detection unit 102 acquires the feature data 110 shown in Figure 3(a). The waveform in Figure 3 shows the time evolution of the feature. Note that the dashed and solid waveforms in Figure 3(a) represent the characteristic features of speech from different speakers. In the example shown in Figure 3(a), as indicated by reference numeral 1101, the speaker with the solid waveform begins speaking while the speaker with the dashed waveform is speaking, and the two voices are temporarily mixed. Subsequently, the speaker with the dashed waveform stops speaking, and the speaker with the solid waveform continues speaking.

[0023] The interval detection unit 102 analyzes the features shown in the feature data 110 in Figure 3(a) and detects from the feature data 110 the time range in which audio exists as the audio interval 121. Here, the section detection unit 102 is assumed to have detected three audio sections 121, as shown in Figure 3(b). The interval detection unit 102 can detect the speech interval 121 by, for example, using the method shown in Reference 1 below. The interval detection unit 102 can detect the speech interval 121 by changing the configuration of the training data shown in Reference 1. Reference 1 uses a classifier that distinguishes between noise and speech. Instead of the classifier in Reference 1, a non-speech interval and speech By using a classifier that identifies voice segments, the segment detection unit 102 can detect the voice segment 121. Reference 1: International Publication WO2019162990A1

[0024] Next, the section detection unit 102 detects the applicable section 125 from the audio section 121. The applicable section 125 consists of the preceding section 122, the mixed audio section 123, and the following section 124. The mixed speech section 123 is the time range in which the mixed speech exists. The preceding section 122 is located immediately before the mixed speech section 123 and is a time range in which a single speech element exists. The immediate following section 124 is located immediately after the mixed speech section 123 and is a time range in which a single speech element exists. The section detection unit 102 detects the mixed voice section 123 in the manner described below. The section detection unit 102 then detects the range of a predetermined time immediately preceding the detected mixed voice section 123 as the preceding section 122. Furthermore, the section detection unit 102 detects the range of a predetermined time immediately following the detected mixed voice section 123 as the following section 124. This default time can also be dynamically set based on the positional relationship between the section detected by the section detection unit 102 and other sections. Specifically, when two mixed speech sections 123 are adjacent within a short time interval, the section detection unit 102 uses the section information to set the preceding section 122 and the following section 124 so that they do not overlap with another mixed speech section. By dynamically setting the predetermined time based on its positional relationship with other sections, it becomes impossible for the mixed audio from another mixed audio section to be mistakenly mixed into the preceding section 122 or the following section 124, thereby improving the detection accuracy of the section detection unit 102. The section detection unit 102 may also notify the user of the position and time interval length of the detected mixed speech section 123 using a display or the like (not shown). Furthermore, if the default time is changed, the section detection unit 102 may also notify the user of the position of the section where the default time has been changed and the changed default time using a display or the like (not shown). The user may be, for example, the speaker or the operator operating the speech processing device 100.

[0025] Method for detecting mixed speech section 123 (1) - Rule-based The section detection unit 102 detects, for example, the point in the audio section 121 where the power of the audio signal suddenly increases as the starting point of the mixed audio section 123. The section detection unit 102 also detects the point where the power of the audio signal suddenly decreases as the ending point of the mixed audio section 123.

[0026] Method for detecting mixed speech section 123 (2) - Rule-based The section detection unit 102 can also detect the mixed speech section 123 using zero intersection density. In the audio section 121, the zero-crossing density of the audio signal is high. Utilizing this property, the section detection unit 102 first detects the time range where the zero-crossing density is above a threshold as the audio section 121. Furthermore, it is expected that the zero-crossing density will be even higher in the mixed audio section 123. Therefore, the section detection unit 102 detects the time range in the audio section 121 where the zero-crossing density is even higher as the mixed audio section 123.

[0027] Method for detecting mixed speech sections 123 (3) - Statistical basis The interval detection unit 102 may detect the mixed speech interval 123 by training a GMM (Gaussian Mixture Model). For example, the interval detection unit 102 learns a noise GMM and a speech GMM, compares the likelihood of their respective learning results, and detects the speech interval 121. Furthermore, the interval detection unit 102 learns noise GMM, single-voice GMM, and multiple-voice GMM, compares the likelihood of each learning result, and detects the mixed-voice interval 123. In addition to using GMM, the interval detection unit 102 uses high-dimensional features such as skewness to determine the speech interval It is possible to detect 121 and the mixed speech section 123. Furthermore, the interval detection unit 102 may be configured as an HMM (Hidden Markov Model).

[0028] Method for detecting mixed speech section 123 (4) - Neural network based Alternatively, the section detection unit 102 may detect the mixed speech section 123 using a neural network-based method. For example, the segment detection unit 102 detects the speech segment 121 and the mixed speech segment 123 by learning using a multilayer perceptron, RNN (Recurrent Neural Network), LSTM (Long Short Term Memory), etc.

[0029] The section detection unit 102 can be used in combination with the above detection methods. For example, the interval detection unit 102 can combine detection method (3) and detection method (4). In this case, the interval detection unit 102 detects the speech interval 121 and the mixed speech interval 123 by adding the statistically based likelihood and the likelihood obtained from the neural network. Furthermore, the section detection unit 102 may exclude a detected section from subsequent processing if its length is shorter than a set threshold. The threshold is a pre-set time interval length, which is set to a value shorter than that of a normal audio waveform. If the section detection unit 102 detects a section shorter than this threshold, it may have mistakenly detected a non-audio signal such as noise. By excluding sections shorter than the threshold from subsequent processing, unnecessary audio separation processing is suppressed. As a result, the amount of processing is reduced, and waveform distortion caused by unnecessary audio separation processing can be reduced.

[0030] The section detection unit 102 outputs the applied section feature quantity 120, which is the feature quantity of the applied section 125 detected as described above, to the speech separation unit 103.

[0031] The speech separation unit 103 uses the applicable interval feature quantity 120 to separate the mixed speech in the mixed speech interval 123 into multiple separated speeches. The speech separation unit 103 can separate mixed speech into multiple separated speeches by, for example, using the method described in Patent Document 1. Alternatively, the speech separation unit 103 may separate mixed speech into multiple separated speeches by learning the speech separation model described in Patent Document 1. Furthermore, the audio separation unit 103 uses the applicable interval feature quantity 120 to extract forward-connected separated audio and backward-connected separated audio from multiple separated audio.

[0032] Figure 4(b) shows the state in which the separated speech information 130 separates the mixed speech (a mixture of dashed and solid lines) of the mixed speech section 123 shown in Figure 4(a) into separated speech. Separated speech 126 is the dashed line speech separated from the mixed speech in mixed speech section 123. Separated speech 127 is the solid line speech separated from the mixed speech. Furthermore, separated speech 126 is estimated to be the same speech as the preceding single speech 128, and therefore corresponds to a forward connected separated speech. The speech separation unit 103 estimates that separated speech 126 and the preceding single speech 128 are the same speech because, in terms of features, separated speech 126 is more similar to the preceding single speech 128 than to the following single speech 129. Furthermore, separated speech 127 is estimated to be the same speech as the immediately following single speech 129, and thus corresponds to a backward-connected separated speech. The speech separation unit 103 estimates that separated speech 127 and immediately following single speech 129 are the same speech because, in terms of features, separated speech 127 is more similar to the immediately following single speech 129 than to the immediately preceding single speech 128. The audio separation unit 103 outputs the separated audio information 130 to the audio restoration unit 104. The separated audio information 130 includes, as described above, the feature quantities of separated audio 126 and separated audio 127, the identifier of separated audio 126 corresponding to the forward connected separated audio, and the identifier of separated audio corresponding to the backward connected separated audio. This includes the identifier of the separated speech 127, the feature quantities of the preceding single speech 128, and the feature quantities of the following single speech 129.

[0033] The audio restoration unit 104 restores the audio signals of separated audio 126 and separated audio 127 based on the separated audio information 130. The audio restoration unit 104 also restores the audio signal of the preceding single audio 128 from the feature quantities of the preceding single audio 128. Furthermore, the audio restoration unit 104 restores the audio signal of the following single audio 129 from the feature quantities of the following single audio 129. The audio restoration unit 104 outputs the restored audio information 140 to the audio connection unit 105. As described above, the restored audio information 140 includes the audio signals of the restored separated audio 126 and separated audio 127, the audio signal of the restored immediate preceding single audio 128, the audio signal of the restored immediate succeeding single audio 129, the identifier of separated audio 126 corresponding to the forward connected separated audio, and the identifier of separated audio 127 corresponding to the backward connected separated audio.

[0034] Figure 4(a) shows the forward section 1201 and the backward section 1202. The forward section 1201 is the section containing a single voice located before the mixed voice section 123. The backward section 1202 is the section containing a single voice located after the mixed voice section 123. In the non-mixed voice section definition information 1200, the forward section 1201 and the backward section 1202 are defined as non-mixed voice sections. As shown in Figure 4(c), the audio connection unit 105 refers to the unmixed audio section definition information 1200 and identifies the forward single audio signal 201, which is the audio signal of the forward section 1201, and the rear single audio signal 202, which is the audio signal of the rear section 1202, in the mixed audio signal sequence 200. The audio connection unit 105 then connects the audio signal of the forward connection separated audio 126 to the end of the forward single audio signal 201 (the part corresponding to the immediately preceding single audio 128). The audio connection unit 105 also connects the audio signal of the rear connection separated audio 127 to the beginning of the rear single audio signal 202 (the part corresponding to the immediately following single audio 129).

[0035] The audio connection unit 105 may also determine whether the audio signal of the preceding single audio 128 is similar to the trailing portion of the forward single audio signal 201. In this case, if the audio signal of the preceding single audio 128 is similar to the trailing portion of the forward single audio signal 201, the audio connection unit 105 connects the audio signal of the forward connection separated audio 126 to the trailing portion of the preceding single audio signal 201. Similarly, the audio connection unit 105 may determine whether the audio signal of the immediately following single audio 129 is similar to the leading portion of the rear single audio signal 202. In this case, if the audio signal of the immediately following single audio 129 is similar to the leading portion of the rear single audio signal 202, the audio connection unit 105 connects the audio signal of the rear connection / separated audio 127 to the beginning of the rear single audio signal 202. The audio connection unit 105 determines similarity using methods such as correlation functions and mean squared error. If there are three or more speakers, the audio connection unit 105 identifies the separated voice that has the preceding single voice 128 most similar to the ending portion of the preceding single voice signal 201 as the preceding connected separated voice 126. The audio connection unit 105 then connects the audio signal of the preceding connected separated voice 126 to the end of the preceding single voice signal 201. Similarly, the audio connection unit 105 identifies the separated audio having the immediately following single audio 129 that is most similar to the leading portion of the rear single audio signal 202 as the rear connected separated audio 127. The audio connection unit 105 then connects the audio signal of the rear connected separated audio 127 to the beginning of the rear single audio signal 202.

[0036] In the example shown in Figure 4(c), the audio connection unit 105 connects the audio signal of the forward connection separated audio 126 to the forward single audio signal 201 and the audio signal of the rear connection separated audio 127 to the rear single audio signal 202 using different audio output channels. In this embodiment, it is not possible to determine which audio signal belongs to which speaker. Therefore, as shown in the lower part of Figure 4(c), the solid line audio signal and the dashed line audio signal 2020, which represent audio signals from different speakers, may be output on the same audio output channel.

[0037] Furthermore, as shown in Figure 5, there are cases where the voice of one speaker is temporarily superimposed on the voice of another speaker that is continuously being transmitted. In the example in Figure 5, the speaker with the dashed waveform begins speaking while the speaker with the solid waveform is speaking, and the two voices are temporarily mixed. However, the speaker with the solid waveform finishes speaking after a short time, and thereafter only the speaker with the dashed waveform continues speaking.

[0038] In this case as well, as shown in Figure 5(a), the section detection unit 102 detects the time range in which the solid waveform and the dashed waveform overlap as the mixed speech section 123. The section detection unit 102 also detects the time range of the single voice immediately preceding the mixed speech section 123 and the time range of the single voice immediately following it as the preceding section 122 and the following section 124. Furthermore, as shown in Figure 5(b), the audio separation unit 103 separates the mixed audio into separated audio 126 and separated audio 1270. In Figure 5(b), separated audio 126 corresponds to the forward connected separated audio and the backward connected separated audio. In other words, separated audio 126 is presumed to be the same audio as the preceding single audio 128, and also presumed to be the same audio as the following single audio 129. On the other hand, separated audio 1270 does not fall under either forward-connected separated audio or rear-connected separated audio.

[0039] In this case, as shown in Figure 5(c), the audio connection unit 105 connects the audio signal of the separated audio 126 to the end of the forward single audio signal 203 (the portion corresponding to the immediately preceding single audio 128). The audio connection unit 105 also connects the audio signal of the separated audio 126 to the beginning of the rear single audio signal 204 (the portion corresponding to the immediately following single audio 129). The forward single audio signal 203 is the audio signal of the forward section 1203. The rear single audio signal 204 is the audio signal of the rear section 1204. On the other hand, since the separated audio 1270 does not fall under either the front-connected separated audio or the rear-connected separated audio, the audio connection unit 105 does not connect the separated audio 1270 to any single audio in any time range. The audio connection unit 105 then sets the separated audio 1270 to an audio output channel (back channel) that is different from the audio output channels to which the audio signal of the separated audio 126 is connected to the front single audio signal 203 and the rear single audio signal 204.

[0040] Furthermore, as described above, the audio connection unit 105 may determine the similarity between the audio signal of the preceding single audio 128 and the end portion of the forward single audio signal 203, and the similarity between the audio signal of the following single audio 129 and the beginning portion of the backward single audio signal 204.

[0041] Furthermore, as shown in Figure 6, the audio connection unit 105 may connect the audio signal of the front connection separated audio 126 to the front single audio signal 201 and connect the audio signal of the rear connection separated audio 127 to the rear single audio signal 202 using the same audio output channel. Figures 6(a) and 6(b) are the same as Figures 4(a) and 4(b). In Figure 6(c), the audio connection unit 105 connects the audio signal of the front connection separated audio 126 to the front single audio signal 201 and the audio signal of the rear connection separated audio 127 to the rear single audio signal 202, both on the same audio output channel. More specifically, on the same audio output channel, the audio connection unit 105 sets the single audio signal obtained by connecting the audio signal of the rear connection separated audio 127 to the single audio signal obtained by connecting the audio signal of the rear connection separated audio 127 to the single audio signal obtained by connecting the audio signal of the front connection separated audio 126 to the front single audio signal 201, followed by the single audio signal obtained by connecting the audio signal of the rear connection separated audio 127 to the rear single audio signal 202. Here, the audio connection unit 105 shifts the single audio signal obtained by connecting the audio signal of the rear connection separated audio 127 to the rear single audio signal 202 in time.

[0042] Furthermore, even if the separated audio 1270 shown in Figure 5 is present, the audio connection unit 105 can set the audio signal of the separated audio 1270 to the audio output channel where the audio signal of the separated audio 126 is connected to the front single audio signal 203 and the rear single audio signal 204, as shown in Figure 7. good. Figures 7(a) and 7(b) are the same as Figures 5(a) and 5(b). In Figure 7(c), the audio connection unit 105 sets the audio signal of separated audio 1270 to the audio output channel where the audio signal of separated audio 126 is connected to the forward single audio signal 203 and the rear single audio signal 204-1. More specifically, the audio connection unit 105 does not connect the audio signal of separated audio 1270 to any single audio in any time range, but sets the audio signal of separated audio 1270 after the audio signal of the single audio obtained by connecting the forward single audio signal 203, the audio signal of separated audio 126, and the audio signal of rear single audio signal 204-1. In the example of Figure 7(c), the rear single audio signal 204 is divided into rear single audio signal 204-1 and rear single audio signal 204-2. Also, as in the case of Figure 6, the audio connection unit 105 shifts the audio signal of separated audio 1270 and the rear single audio signal 204-2 backward in time.

[0043] ***Explanation of the effects of the embodiment*** In this embodiment, only the mixed speech sections requiring speech separation are detected as targets for speech separation. Conventional techniques perform audio separation on the entire input audio. As a result, processing distortion occurs across the entire audio, and this distortion negatively impacts post-processing. In this embodiment, by limiting the scope of audio separation, the adverse effects (processing distortion) caused by audio separation can be reduced.

[0044] Furthermore, even if there is a processing delay in the audio separation process, this embodiment can reduce the processing delay by limiting the scope of the audio separation.

[0045] Furthermore, even when the computational cost of voice separation is high, this embodiment can reduce the computational cost by limiting the scope of voice separation.

[0046] Embodiment 2. In this embodiment, the adverse effects of processing distortion in audio restoration are mitigated. This embodiment will primarily describe the differences from Embodiment 1. Matters not described below are the same as in Embodiment 1.

[0047] Figure 8 shows an example of the functional configuration of the audio processing device 100 according to this embodiment. In comparison with Figure 1, Figure 8 shows that the likelihood 1231 of the mixed speech section 123 is output from the section detection unit 102 to the speech reconstruction unit 104.

[0048] In this embodiment, the section detection unit 102 calculates the likelihood 1231 of the mixed speech section 123 in addition to the operation described in Embodiment 1. The section detection unit 102 then outputs the calculated likelihood 1231 to the audio restoration unit 104.

[0049] The audio restoration unit 104 restores the audio signal of each separated audio using the likelihood 1231. Specifically, the speech reconstruction unit 104 first calculates weighted feature quantities W from the following equation 1, and then uses these weighted feature quantities W as input to reconstruct the speech signals of each separated speech. In the following equation 1, alpha is the likelihood 1231. Also, W_before is the feature quantity of the mixed speech before separation. W_after is the feature quantity of the speech after separation (separated speech 126 and separated speech 127 in the example in Figure 4). W = alpha × W_after + (1 - alpha) × W_before Equation 1 However, in this embodiment, W, W_after, and W_before may be the audio signals themselves. In this case, the audio of the separated audio restored by the audio restoration unit 104 is defined as W_after, and the audio before separation is defined as W_before, and the weighted sum of Equation 1 is taken. The resulting audio W is used as the output of the audio restoration unit 104. For example, Reference 2 below shows that calculating a weighted sum of the audio before and after separation is a process to mitigate the adverse effects of processing distortion. Reference 2: Patent No. 7345702

[0050] ***Explanation of the effects of the embodiment*** In this embodiment, processing distortion can be reduced by calculating a weighted sum of the pre- and post-separation audio using the likelihood of the mixed audio section. As a result, this embodiment reduces the adverse effects of processing distortion on post-processing.

[0051] Embodiment 3. This embodiment describes an example of dividing the feature data 110 into predetermined sizes. This embodiment will primarily describe the differences from Embodiment 1. Matters not described below are the same as in Embodiment 1.

[0052] Figure 9 shows an example of the functional configuration of the audio processing device 100 according to this embodiment. Compared to Figure 1, Figure 9 shows the addition of a first division section 131 and a second division section 132. Furthermore, the feature extraction unit 101 outputs the feature data 110 and the mixed speech signal sequence 200 to the first splitting unit 131, and outputs the feature data 110 to the second splitting unit 132.

[0053] The functions of the first division unit 131 and the second division unit 132 are also implemented by a program, similar to the feature extraction unit 101. The program that implements the functions of the first division unit 131 and the second division unit 132 is executed by the processor 901.

[0054] The first division unit 131 divides the feature data 110 into predetermined sizes. The first division unit 131 divides the feature data 110 into sizes suitable for the interval detection unit 102 to detect the mixed speech interval 123. The first splitting unit 131 then outputs the split feature data 1131 obtained by splitting the feature data 110, along with the mixed speech signal sequence 200, to the interval detection unit 102. The interval detection unit 102 acquires segmented feature data 1131 instead of feature data 110, and uses the segmented feature data 1131 to detect the mixed speech interval 123. The detection process of the mixed speech interval 123 in the interval detection unit 102 is the same as that described in Embodiment 1, so the explanation is omitted. In this embodiment, the interval detection unit 102 outputs the applicable interval definition information 1230 to the speech separation unit 103 instead of the applicable interval feature quantity 120. The applicable interval definition information 1230 is information that defines the applicable interval 125.

[0055] The second division unit 132 divides the feature data 110 into predetermined sizes. The second division unit 132 divides the feature data 110 into sizes suitable for the speech separation unit 103 to perform speech separation. The second division unit 132 may divide the feature data 110 into the same sizes as the division size in the first division unit 131, or it may divide the feature data 110 into different sizes. The second splitting unit 132 then outputs the split feature data 1132 obtained by splitting the feature data 110 to the speech separation unit 103. The speech separation unit 103 obtains segmented feature data 1132 from the second segmentation unit 132. The speech separation unit 103 also obtains application interval definition information 1230 from the application interval feature 120. The speech separation unit 103 identifies the application interval 125 using the application interval definition information 1230 and performs speech separation on the identified application interval 125 using the segmented feature data 1132. The speech separation process itself in the speech separation unit 103 is the same as that described in Embodiment 1, so the explanation is omitted.

[0056] In this embodiment, the interval detection unit 102 operates using the segmented feature data 1131, and the speech separation unit 103 operates using the segmented feature data 1132. Therefore, the interval detection unit 102 and the speech separation unit 103 can operate in parallel. Furthermore, the first division section 131 may divide the feature data 110 by providing overlapping portions between preceding and succeeding divided feature data 1131. Similarly, the second division section 132 may also divide the feature data 110 by providing overlapping portions between preceding and succeeding divided feature data 1132.

[0057] ***Explanation of the effects of the embodiment*** In this embodiment, mixed speech intervals are detected using segmented feature data. Therefore, in this embodiment, the processing delay in mixed speech interval detection can be reduced to the size of the segmented feature data. Similarly, in this embodiment, speech separation is performed using segmented feature data. Therefore, in this embodiment, the processing delay in speech separation can be reduced to the size of the segmented feature data.

[0058] Furthermore, in this embodiment, the feature data is divided into sizes suitable for detecting mixed speech segments. Therefore, according to this embodiment, the degradation of accuracy in detecting mixed speech segments due to the division of feature data can be reduced. Similarly, in this embodiment, the feature data is divided into sizes suitable for speech separation. Therefore, according to this embodiment, the degradation of accuracy in speech separation due to the division of feature data can be reduced.

[0059] Embodiment 4. This embodiment describes an example of extracting features suitable for detecting mixed speech sections and features suitable for speech separation. This embodiment will primarily describe the differences from Embodiment 1. Matters not described below are the same as in Embodiment 1.

[0060] Figure 10 shows an example of the functional configuration of the audio processing device 100 according to this embodiment. In comparison with Figure 1, Figure 10 shows that the feature extraction unit 101 is replaced by a first feature extraction unit 141 and a second feature extraction unit 142.

[0061] The functions of the first feature extraction unit 141 and the second feature extraction unit 142 are also implemented by a program, similar to the feature extraction unit 101. The program that implements the functions of the first feature extraction unit 141 and the second feature extraction unit 142 is executed by the processor 901.

[0062] The first feature extraction unit 141 acquires the mixed audio signal train 200. The first feature extraction unit 141 acquires the mixed audio signal train 200 from a communication network, for example, via a communication device 904. Alternatively, the first feature extraction unit 141 may acquire the mixed audio signal train 200 from a recording medium reading device of an input / output device. The first feature extraction unit 141 then extracts features from the mixed speech signal sequence 200 that are suitable for the interval detection unit 102 to detect the mixed speech interval 123. The first feature extraction unit 141 then outputs the first feature data 1141, which shows the extracted features, and the mixed speech signal sequence 200 to the interval detection unit 102. The first feature extraction unit 141 also outputs the mixed speech signal sequence 200 to the second feature extraction unit 142.

[0063] The interval detection unit 102 takes the first feature data 1141 instead of the feature data 110. The first feature data 1141 is used to detect the mixed speech section 123. The detection process of the mixed speech section 123 in the section detection unit 102 is the same as that described in Embodiment 1, so the explanation is omitted. In this embodiment, the interval detection unit 102 outputs the applicable interval definition information 1240 to the speech separation unit 103 instead of the applicable interval feature quantity 120. The applicable interval definition information 1240 is information that defines the applicable interval 125.

[0064] The second feature extraction unit 142 extracts features from the mixed speech signal sequence 200 that are suitable for the speech separation unit 103 to perform speech separation. The second feature extraction unit 142 may extract the same type of features as the first feature extraction unit 141, or it may extract different types of features. The second feature extraction unit 142 then outputs the second feature data 1142, which shows the extracted features, to the speech separation unit 103.

[0065] The speech separation unit 103 obtains second feature data 1142 from the second feature extraction unit 142. The speech separation unit 103 also obtains application interval definition information 1240 from the application interval feature 120. The speech separation unit 103 identifies the application interval 125 using the application interval definition information 1240 and performs speech separation on the identified application interval 125 using the second feature data 1142. The speech separation process in the speech separation unit 103 is the same as that described in Embodiment 1, so the explanation is omitted.

[0066] In this embodiment, the interval detection unit 102 operates using the first feature data 1141, and the speech separation unit 103 operates using the second feature data 1142. Therefore, the interval detection unit 102 and the speech separation unit 103 can operate in parallel.

[0067] ***Explanation of the effects of the embodiment*** In this embodiment, features suitable for detecting mixed speech segments are extracted. Therefore, according to this embodiment, the accuracy of detecting mixed speech segments can be improved. Furthermore, a model suitable for detecting mixed speech segments can be used. Similarly, in this embodiment, features suitable for speech separation are extracted. Therefore, according to this embodiment, the accuracy of speech separation can be improved. In addition, a model suitable for speech separation can be used.

[0068] Embodiment 5. This embodiment describes an example of how to perform voice enhancement. This embodiment will primarily describe the differences from Embodiment 1. Matters not described below are the same as in Embodiment 1.

[0069] Figure 11 shows an example of the functional configuration of the audio processing device 100 according to this embodiment. Compared to Figure 1, Figure 11 shows the addition of a voice enhancement unit 106.

[0070] The functions of the speech enhancement unit 106 are also implemented by a program, similar to the feature extraction unit 101. The program that implements the functions of the speech enhancement unit 106 is executed by the processor 901.

[0071] The audio enhancement unit 106 performs audio enhancement to separate noise from speech. The voice enhancement unit 106 uses, for example, the voice enhancement technology disclosed in Reference 3 below. Reference 3: https: / / www.merl.com / publications / docs / TR2016-113.pdf

[0072] In this embodiment, in addition to the processing described in Embodiment 1, the interval detection unit 102 determines the noise level in the feature data 110. Then, the interval detection unit 102 determines the time range (hereinafter referred to as the "audio enhancement interval") in which the audio enhancement unit 106 will perform audio enhancement, according to the noise level. The section detection unit 102, for example as shown in Figure 12, designates a time range shorter than the audio section 121 and longer than the application section 125 as the audio enhancement section 1255. The section detection unit 102 detects the voice enhancement section 1255 by the method shown below. The interval detection unit 102 then outputs the speech enhancement interval feature quantity 1251 and the applicable interval definition information 1252 to the speech enhancement unit 106. The speech enhancement interval feature quantity 1251 is the feature quantity of the speech enhancement interval 1255. The applicable interval definition information 1252 is the information that defines the applicable interval 125.

[0073] Method for detecting speech enhancement section 1255 (1) - Rule-based The section detection unit 102 detects the starting point of the voice enhancement section 1255 if the power of the voice signal at a point slightly before the starting point of the voice section 121 (hereinafter referred to as the starting candidate point) is higher than the power of the voice signal at a point a certain time before the starting candidate point (i.e., the power of the voice signal at the starting candidate point is higher than that of the non-voice section before the starting candidate point). The section detection unit 102 also detects the ending point of the voice enhancement section 1255 if the power of the voice signal at a point slightly after the ending point of the voice section 121 (hereinafter referred to as the ending candidate point) is higher than the power of the voice signal at a point a certain time after the ending candidate point.

[0074] Method for detecting speech enhancement interval 1255 (2) - Statistical basis The section detection unit 102 may learn the GMM and detect the speech enhancement section 1255. For example, the interval detection unit 102 learns a noise GMM and a speech GMM, compares the likelihood of their respective learning results, and detects the speech interval 121. In detecting the speech enhancement section 1255, the section detection unit 102 learns the noise GMM using high-power noise or special noise to which speech enhancement should be applied. Furthermore, the interval detection unit 102 may constitute an HMM.

[0075] Method for detecting speech enhancement interval 1255 (3) - Neural network based Furthermore, the section detection unit 102 detects the speech section 121 and the speech enhancement section 1255 by learning using a multilayer perceptron, RNN, LSTM, etc.

[0076] The speech enhancement unit 106 performs speech enhancement on the speech enhancement interval feature quantity 1251. The speech enhancement unit 106 then outputs the speech enhancement application section feature quantity 160 to the speech separation unit 103. The speech enhancement application section feature quantity 160 is the feature quantity of the application section 125 after speech enhancement. The speech enhancement unit 106 outputs the feature quantities corresponding to the application interval 125 defined in the application interval definition information 1252 from the speech enhancement interval feature quantities 1251 after speech enhancement as the speech enhancement application interval feature quantity 160.

[0077] The speech separation unit 103 performs speech separation using the speech enhancement application interval feature quantity 160 instead of the application interval feature quantity 120. The speech separation process itself in the speech separation unit 103 is the same as that described in Embodiment 1, so the explanation is omitted.

[0078] Alternatively, instead of the feature extraction unit 101 shown in Figure 10, a configuration with a first feature extraction unit 141 and a second feature extraction unit 142 may be provided, to which the speech enhancement unit 106 may be added. In this case, a third feature extraction unit may be provided to extract a third feature suitable for speech enhancement. In this case, the third feature is input from the third feature extraction unit to the speech enhancement unit 106.

[0079] ***Explanation of the effects of the embodiment*** According to this embodiment, noise can be removed by sound enhancement. Furthermore, in this embodiment, since the range of sound enhancement is limited according to the noise level, processing distortion associated with sound enhancement can be limited.

[0080] Furthermore, even if there is a processing delay in the voice enhancement, this embodiment can reduce the processing delay by limiting the range of voice enhancement. Furthermore, even when the computational cost of speech enhancement is high, this embodiment can reduce the computational cost by limiting the scope of speech enhancement.

[0081] Embodiment 6. This embodiment describes an example of generating a model for deriving the likelihood 1231 of the mixed speech section 123 described in Embodiment 2. This embodiment will primarily describe the differences from Embodiment 2. Matters not described below are the same as in Embodiment 2.

[0082] Figure 13 shows an example of the functional configuration of the audio processing device 100 according to this embodiment. Compared to Figure 8, Figure 13 shows the addition of a model generation unit 107 and a model storage unit 108.

[0083] The functions of the model generation unit 107 are also implemented by a program, similar to the feature extraction unit 101. The program that implements the functions of the model generation unit 107 is executed by the processor 901. The model memory unit 108 is implemented, for example, by the auxiliary storage device 903.

[0084] The model generation unit 107 generates a likelihood derivation model 170 through learning during the learning phase. The likelihood derivation model 170 is a model for deriving the likelihood 1231 of the mixed speech interval 123. The model generation unit 107 stores the generated likelihood derivation model 170 in the model storage unit 108. The model memory unit 108 stores the likelihood derivation model 170.

[0085] In this embodiment, the interval detection unit 102 derives the likelihood 1231 using the likelihood derivation model 170 during the inference phase. The interval detection unit 102 then outputs the derived likelihood 1231 to the audio restoration unit 104. As described in Embodiment 2, the audio restoration unit 104 restores the audio signal of each separated audio using the likelihood 1231.

[0086] Specifically, the model generation unit 107 generates the likelihood derivation model 170 as follows. Here, the speech separation unit 103 learns the speech separation model described in Patent Document 1 and separates the mixed speech into multiple separated speeches. The loss function of the speech separation model when the speech separation unit 103 learns the speech separation model is L_sep(W_true, W_after). Here, W_true is the true speech signal, and W_after is the separated speech signal. Then, the likelihood 1231 output by the interval detection unit 102 is defined as E_osd. In this embodiment, the loss function of the speech separation model when training the speech separation model is L_sep(W_true',W_after). W_true' can be expressed in the following equation 2. W_true'=E_osd×W_true+(1-E_osd)×W_before formula 2

[0087] The model generation unit 107 consists of, for example, training data 171, a speech mixing unit 172, a feature extraction unit 173, a teacher generation unit 174, and a model learning unit 175, as shown in Figure 14.

[0088] ***Explanation of the effects of the embodiment*** In this embodiment, a model is generated for deriving the likelihood of a mixed speech section. Therefore, according to this embodiment, the likelihood can be derived more accurately, and processing distortion can be effectively reduced.

[0089] Although embodiments 1 to 6 have been described above, two or more of these embodiments may be combined and implemented. Alternatively, one of these embodiments may be partially implemented. Alternatively, two or more of these embodiments may be partially combined and implemented. Furthermore, the configurations and procedures described in these embodiments may be modified as needed.

[0090] ***About Use Cases*** The following use cases are envisioned for the voice processing device 100 described in Embodiments 1 to 6. (1) When multiple speakers' voices overlap in a meeting, lecture, etc., the voices of each speaker are separated and speech recognition is performed for recording, transcription, etc. (2) In face-to-face sales, if the voices of the salesperson and the customer overlap, separate and record each speaker's voice, and keep each speaker's voice as evidence. (3) When a smartphone, smart speaker, etc. recognizes a user's instructions, questions, etc., to a voice assistant, other people's voices are removed. This allows the smartphone, smart speaker, etc., to accurately understand the user's instructions, questions, etc. (4) Develop hearing aids, headsets, etc., that separate the voices of multiple speakers and amplify the voice of the person being spoken to, making them easier to hear for people with hearing difficulties. (5) When evaluating the ability to communicate through spoken language in education, training, etc., the voices of multiple speakers are separated and the content of the speech and fluency are measured. (6) When listening to the voices of patients and users in medical and nursing care settings, remove the voices of other people. This can improve the quality of diagnosis and care services.

[0091] ***Supplementary explanation of hardware configuration*** Here, we will provide a supplementary explanation of the hardware configuration of the audio processing device 100. The processor 901 shown in Figure 2 is an integrated circuit (IC) that performs processing. Processor 901 includes components such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The main memory 902 shown in Figure 2 is RAM (Random Access Memory). The auxiliary storage device 903 shown in Figure 2 includes ROM (Read Only Memory), flash memory, HDD (Hard Disk Drive), etc. The communication device 904 shown in Figure 2 is an electronic circuit that performs data communication processing. The communication device 904 is, for example, a communication chip or a NIC (Network Interface Card).

[0092] Furthermore, the auxiliary storage device 903 also stores the OS (Operating System). Then, at least a portion of the OS is executed by processor 901. The processor 901 executes programs that realize the functions of the functional components shown in Figures 1, 8-11, and 13 (hereinafter referred to as Figure 1, etc.) while executing at least a part of the OS. Processor 901 executes the OS, handling task management, memory management, file management, communication control, and other functions. Furthermore, at least one of the information, data, signal values, and variable values ​​indicating the processing results of the functional components shown in Figure 1, etc., is stored in at least one of the main memory 902, auxiliary memory 903, registers in the processor 901, and cache memory. Furthermore, the programs that realize the functions of the functional components shown in Figure 1, etc., may be stored on portable recording media such as magnetic disks, flexible disks, optical disks, compact disks, Blu-ray® disks, DVDs, etc. And portable recording media containing the programs that realize the functions of the functional components shown in Figure 1, etc., may be distributed.

[0093] Furthermore, at least one of the "parts" of the functional components shown in Figure 1, etc., may be read as "circuit," "process," "procedure," "process," or "circuitry." Furthermore, the audio processing device 100 may be implemented by a processing circuit. Examples of processing circuits include logic ICs (Integrated Circuits), GAs (Gate Arrays), ASICs (Application Specific Integrated Circuits), and FPGAs (Field-Programmable Gate Arrays). In this case, the functional components shown in Figure 1, etc., are each implemented as part of the processing circuit. In this specification, the higher-level concept encompassing both the processor and the processing circuit is referred to as "processing circuitry." In other words, a processor and a processing circuit are specific examples of "processing circuits," respectively.

[0094] The various aspects of this disclosure are summarized below as an appendix. (Note 1) A section detection unit detects a mixed speech section, which is a time range in which a mixed speech exists where multiple voices are mixed, from time-series audio data. A speech processing device having a speech separation unit that separates the mixed speech in the mixed speech section into the plurality of speeches. (Note 2) The section detection unit, The audio processing device according to Appendix 1, which detects from the audio data a mixed audio section, a preceding section which is a time range in the time series of the audio data that is immediately before the mixed audio section and contains a single audio, and a succeeding section which is a time range in the time series of the audio data that is immediately after the mixed audio section and contains a single audio. (Note 3) The aforementioned audio processing device further, The audio processing device according to Appendix 1 or 2, having an audio connection unit that connects at least one of the multiple separated audios obtained by separation by the audio separation unit to any single audio. (Note 4) The aforementioned audio connection unit is One of the multiple separated audios is connected to a forward single audio which is a single audio that exists in the forward section, which is a time range located before the mixed audio section in the time series of the audio data. The audio processing device according to Appendix 3, which connects one of the plurality of separated audios to a single subsequent audio that exists in a subsequent section, which is a time range located after the mixed audio section in the time series of the audio data. (Note 5) The section detection unit, From the audio data, the mixed audio section, the preceding section which is the time range in the time series of the audio data that is immediately before the mixed audio section and contains a single audio, and the following section which is the time range in the time series of the audio data that is immediately after the mixed audio section and contains a single audio, are detected. The aforementioned audio separation unit is From the multiple separated audios, the separated audio that is estimated to be the same as the preceding single audio that exists in the preceding section is extracted as the forward connected separated audio. From the aforementioned multiple separated audios, a separated audio that is presumed to be the same as the single audio that appears immediately after the aforementioned section is extracted as a subsequent connected separated audio. The aforementioned audio connection unit is The aforementioned forward-connected separated audio is connected to the forward single audio, The audio processing device according to Appendix 4, which connects the rear-connected separated audio to the rear single audio. (Note 6) The aforementioned audio connection unit is The audio processing device according to Appendix 5, which connects the forward-connected separated audio to the forward single audio and connects the rear-connected separated audio to the rear single audio using different audio output channels. (Appendix 7) The aforementioned audio connection unit is The audio processing device according to Appendix 5, in the case where there is a separated audio among the plurality of separated audios that does not fall under either the forward-connected separated audio or the backward-connected separated audio, sets the separated audio to an audio output channel different from the audio output channel in which the forward-connected separated audio is connected to the forward single audio and the audio output channel in which the backward-connected separated audio is connected to the backward single audio, without connecting the separated audio to a single audio of any time range. (Note 8) The aforementioned audio connection unit is The audio processing device according to Appendix 5, which connects the forward-connected separated audio to the forward single audio and connects the rear-connected separated audio to the rear single audio on the same audio output channel. (Appendix 9) The aforementioned audio connection unit is The audio processing device according to Appendix 8, wherein, in the same audio output channel, a single audio obtained by connecting the forward single audio, the immediate preceding single audio, and the forward connection separation audio is set behind a single audio obtained by connecting the rear connection separation audio, the immediate following single audio, and the rear single audio. (Note 10) The aforementioned audio connection unit is The audio processing device according to Appendix 8, in the case where there is a separated audio among the plurality of separated audios that does not fall under either the forward-connected separated audio or the backward-connected separated audio, the separated audio is set to an audio output channel where the forward-connected separated audio is connected to the forward single audio and the backward-connected separated audio is connected to the backward single audio, without connecting the separated audio to a single audio of any time range. (Note 11) The section detection unit, In the single-speech interval, which is the time range in which a single speech exists, the features of the single speech are shown, and in the mixed-speech interval, the features of the mixed speech are shown. Time-series feature data is acquired as the speech data, and the features shown in the feature data are analyzed to obtain the features from the feature data. The mixed speech section is detected, The aforementioned audio separation unit is The feature quantities of the mixed speech in the mixed speech section are separated into the feature quantities of the multiple speeches. The aforementioned audio processing device further, The system includes a sound restoration unit that reconstructs multiple separated sounds from the multiple sound feature quantities obtained by the separation unit, The aforementioned audio connection unit is The audio processing device according to Appendix 3, which connects at least one of the multiple separated audios obtained by the restoration of the audio restoration unit to any single audio. (Note 12) The aforementioned audio restoration unit, The audio processing device according to Appendix 11, which restores the plurality of separated speeches using the likelihood of the mixed speech section. (Note 13) The aforementioned audio processing device further, A first division unit divides the audio data into sizes suitable for detection of the mixed audio section by the section detection unit, It has a second division unit that divides the audio data into sizes suitable for audio division by the aforementioned audio separation unit, The section detection unit, From the audio data after it has been divided by the first division unit, the mixed audio section is detected. The aforementioned audio separation unit is The audio processing device according to Appendix 1, which uses the audio data after it has been divided by the second division unit to separate the mixed audio in the mixed audio section into the plurality of audio. (Note 14) The aforementioned audio processing device further, A first feature extraction unit generates first feature data that shows feature quantities suitable for detecting the mixed speech section by the section detection unit, The system includes a second feature extraction unit that generates second feature data showing feature quantities suitable for speech segmentation by the aforementioned speech separation unit, The section detection unit, The first feature data is obtained as the audio data, and the mixed audio section is detected from the first feature data. The aforementioned audio separation unit is The speech processing device according to Appendix 1, which uses the second feature data to separate the mixed speech in the mixed speech section into the plurality of speeches. (Note 15) The aforementioned audio processing device further, It has a sound enhancement unit that performs sound enhancement to separate noise and speech, The section detection unit, The audio processing device according to Appendix 1, which determines a time range for causing the audio enhancement unit to perform audio enhancement according to the level of noise in the aforementioned audio data. (Note 16) The aforementioned audio processing device further, The system includes a model generation unit that generates a likelihood derivation model, which is a model for deriving the likelihood of the aforementioned mixed speech section. The aforementioned audio restoration unit, The speech processing device according to Appendix 12, which restores the plurality of separated speeches using the likelihood of the mixed speech interval derived using the likelihood derivation model. (Note 17) The computer detects a mixed speech interval, which is a time range in which mixed speech exists, from time-series audio data. A speech processing method in which the computer separates the mixed speech in the mixed speech section into the plurality of speeches. (Note 18) A section detection process that detects a mixed audio section, which is a time range in which a mixed audio exists where multiple voices are mixed, from time-series audio data, A speech processing program that causes a computer to perform a speech separation process that separates the mixed speech in the mixed speech section into the plurality of speeches. [Explanation of symbols]

[0095] 100 Speech processing unit, 101 Feature extraction unit, 102 Interval detection unit, 103 Speech separation unit, 104 Speech reconstruction unit, 105 Speech connection unit, 106 Speech enhancement unit, 107 Model generation unit, 108 Model storage unit, 110 Feature data, 120 Applicable interval features, 121 Speech interval, 122 Preceding interval, 123 Mixed speech interval, 124 Immediately following interval, 125 Applicable interval, 126 Separated speech, 127 Separated speech, 128 Preceding single speech, 129 Immediately following single speech, 130 Separated speech information, 131 First division unit, 132 Second division unit, 140 Reconstructed speech information, 141 First feature extraction unit, 142 Second feature extraction unit, 160 Speech enhancement applicable interval features, 170 Likelihood derivation model, 171 Training data, 172 Speech mixing unit, 173 Feature extraction unit, 174 Teacher generation unit, 175 Model learning unit, 200 Mixed speech signal sequence, 201 Forward single speech signal, 202 Backward single speech signal, 203 Forward single speech signal, 204 Backward single speech signal, 300 Separated speech signal sequence, 901 Processor, 902 Main memory, 903 Auxiliary memory, 904 Communication device, 1131 Split feature data, 1132 Split feature data, 1141 First feature data, 1142 Second feature data, 1200 Unmixed speech interval definition information, 1201 Forward interval, 1202 Backward interval, 1203 Forward interval, 1204 Backward interval, 1230 Application interval definition information, 1231 Likelihood, 1240 Application interval definition information, 1251 Speech enhancement interval features, 1252 Applicable interval definition information, 1255 speech enhancement interval, 1270 separated speech, 2020 speech signal.

Claims

1. A section detection unit detects, from time-series audio data, a mixed audio section which is a time range in which a mixed audio exists where multiple audios are mixed together, a preceding section which is a time range in the time series of the audio data that is located immediately before the mixed audio section and contains a single audio, and a succeeding section which is a time range in the time series of the audio data that is located immediately after the mixed audio section and contains a single audio. A speech processing device having a speech separation unit that separates the mixed speech in the mixed speech section into the plurality of speeches.

2. The aforementioned audio processing device further, The audio processing device according to claim 1, further comprising an audio connection unit that connects at least one of the multiple separated audios, which are the multiple audios obtained by separation by the audio separation unit, to any single audio.

3. The aforementioned audio connection unit is One of the multiple separated audios is connected to a forward single audio which is a single audio that exists in the forward section, which is a time range located before the mixed audio section in the time series of the audio data. The audio processing device according to claim 2, wherein one of the plurality of separated audios is connected to a single audio, which is a single audio, that exists in a later section, which is a time range located after the mixed audio section in the time series of the audio data.

4. The audio separation unit is From the multiple separated audios, the separated audio that is estimated to be the same as the preceding single audio that exists in the preceding section is extracted as the forward connected separated audio. From the aforementioned multiple separated audios, a separated audio that is presumed to be the same as the single audio that appears immediately after the aforementioned section is extracted as a subsequent connected separated audio. The aforementioned audio connection unit is The aforementioned forward-connected separated audio is connected to the forward single audio, The audio processing device according to claim 3, which connects the rear connected separated audio to the rear single audio.

5. The aforementioned audio connection unit is The audio processing device according to claim 4, wherein the connection of the forward-connected separated audio to the forward single audio and the connection of the rear-connected separated audio to the rear single audio are performed on different audio output channels.

6. The aforementioned audio connection unit is If, among the plurality of separated audio tracks, there is a separated audio track that does not fall under either the forward-connected separated audio track or the backward-connected separated audio track, then, without connecting that separated audio track to any single audio track of any time range, the audio output channel where the forward-connected separated audio track is connected to the forward single audio track and the audio output channel where the backward-connected separated audio track is connected to the backward single audio track are different. The audio processing device according to claim 4, wherein the separated audio is set to the audio output channel.

7. The aforementioned audio connection unit is The audio processing device according to claim 4, wherein the connection of the forward-connected separated audio to the forward single audio and the connection of the rear-connected separated audio to the rear single audio are performed on the same audio output channel.

8. The aforementioned audio connection unit is The audio processing device according to claim 7, wherein, in the same audio output channel, a single audio obtained by connecting the forward single audio, the immediate preceding single audio, and the forward connection separation audio is set behind a single audio obtained by connecting the rear connection separation audio, the immediate following single audio, and the rear single audio.

9. The aforementioned audio connection unit is The audio processing device according to claim 7 or 8, in the case where there is a separated audio among the plurality of separated audios that does not fall under either the forward-connected separated audio or the backward-connected separated audio, the separated audio is set to an audio output channel where the forward-connected separated audio is connected to the forward single audio and the backward-connected separated audio is connected to the backward single audio, without connecting the separated audio to any single audio of any time range.

10. The section detection unit, In the single-speech interval, which is the time range in which a single speech exists, the feature quantities of the single speech are shown, and in the mixed-speech interval, the feature quantities of the mixed speech are shown. Time-series feature quantity data is acquired as the speech data, and the feature quantities shown in the feature quantity data are analyzed to detect the mixed-speech interval from the feature quantity data. The aforementioned audio separation unit is The feature quantities of the mixed speech in the mixed speech section are separated into the feature quantities of the multiple speeches. The aforementioned audio processing device further, The system includes a sound restoration unit that reconstructs multiple separated sounds from the multiple sound feature quantities obtained by the separation unit, The aforementioned audio connection unit is The audio processing device according to claim 2, wherein at least one of the multiple separated audios obtained by the restoration of the audio restoration unit is connected to any single audio.

11. The aforementioned audio restoration unit, The audio processing device according to claim 10, which restores the plurality of separated voices using the likelihood of the mixed voice section.

12. The aforementioned audio processing device further, A first division unit divides the audio data into sizes suitable for detection of the mixed audio section by the section detection unit, It has a second division unit that divides the audio data into sizes suitable for audio division by the aforementioned audio separation unit, The section detection unit, From the audio data after it has been divided by the first division unit, the mixed audio section is detected. The aforementioned audio separation unit is The audio processing device according to claim 1, wherein the audio data after being divided by the second division unit is used to separate the mixed audio in the mixed audio section into the plurality of audio.

13. The aforementioned audio processing device further, A first feature extraction unit generates first feature data that shows feature quantities suitable for detecting the mixed speech section by the section detection unit, It includes a second feature extraction unit that generates second feature data showing feature quantities suitable for speech segmentation by the aforementioned speech separation unit, The section detection unit, The first feature data is obtained as the audio data, and the mixed audio section is detected from the first feature data. The aforementioned audio separation unit is The speech processing device according to claim 1, wherein the mixed speech in the mixed speech section is separated into the plurality of speeches using the second feature data.

14. The aforementioned audio processing device further, It has a sound enhancement unit that performs sound enhancement to separate noise and speech, The section detection unit, The audio processing apparatus according to claim 1, which determines a time range for causing the audio enhancement unit to perform audio enhancement according to the level of noise in the audio data.

15. The aforementioned audio processing device further, The system includes a model generation unit that generates a likelihood derivation model, which is a model for deriving the likelihood of the aforementioned mixed speech section. The aforementioned audio restoration unit, The speech processing device according to claim 11, wherein the plurality of separated speeches are restored using the likelihood of the mixed speech section derived using the likelihood derivation model.

16. The computer detects, from the time-series audio data, a mixed audio section which is a time range in which a mixed audio exists where multiple voices are mixed together, a preceding section which is a time range in the time series of the audio data that is immediately before the mixed audio section and contains a single voice, and a succeeding section which is a time range in the time series of the audio data that is immediately after the mixed audio section and contains a single voice. A speech processing method in which the computer separates the mixed speech in the mixed speech section into the plurality of speeches.

17. A section detection process that detects, from time-series audio data, a mixed audio section which is a time range in which a mixed audio exists where multiple audios are mixed together, a preceding section which is a time range in the time series of the audio data that is immediately before the mixed audio section and contains a single audio, and a succeeding section which is a time range in the time series of the audio data that is immediately after the mixed audio section and contains a single audio. A speech processing program that causes a computer to perform a speech separation process that separates the mixed speech in the mixed speech section into the plurality of speeches.

Images