Voice processing device, voice processing method, and voice processing program

Abstract

Provided is a section detection unit (102) that detects, from time-series voice data, a mixed voice section that is a time range in which a mixed voice in which a plurality of voices is mixed is present. Also provided is a voice separation unit (103) that separates the mixed voice in the mixed voice section into the plurality of voices.

Description

Audio processing device, audio processing method, and audio processing program

[0001] The present disclosure relates to audio processing.

[0002] A technique related to the present disclosure is disclosed in Patent Document 1. The technique in Patent Document 1 separates an audio signal containing a mixture of voices from multiple speakers into audio signals for each speaker.

[0003] Patent No. 6789455

[0004] Processing distortion occurs in a time-series signal output by a speech separation technique such as that disclosed in Patent Document 1. When another process (hereinafter referred to as post-processing) is performed after speech separation, this processing distortion may adversely affect the post-processing. For example, when speech recognition is performed as post-processing after speech separation, the processing distortion may degrade the speech recognition accuracy.

[0005] The main objective of this disclosure is to reduce processing distortion due to audio separation.

[0006] The audio processing device according to the present disclosure includes a section detection unit that detects a mixed audio section, which is a time range in which a mixed audio in which multiple audio segments are mixed, from time-series audio data, and an audio separation unit that separates the mixed audio in the mixed audio section into the multiple audio segments.

[0007] According to the present disclosure, processing distortion due to audio separation can be reduced.

[0008] FIG. 1 is a diagram showing an example of the functional configuration of a speech processing device according to embodiment 1. FIG. 2 is a diagram showing an example of the hardware configuration of a speech processing device according to embodiment 1. FIG. 3 is a diagram showing an example of the operation of a speech processing device according to embodiment 1. FIG. 4 is a diagram showing an example of the operation of a speech processing device according to embodiment 1. FIG. 5 is a diagram showing an example of the operation of a speech processing device according to embodiment 1. FIG. 6 is a diagram showing an example of the operation of a speech processing device according to embodiment 1. FIG. 7 is a diagram showing an example of the operation of a speech processing device according to embodiment 1. FIG. 8 is a diagram showing an example of the functional configuration of a speech processing device according to embodiment 2. FIG. 9 is a diagram showing an example of the functional configuration of a speech processing device according to embodiment 3. FIG. 10 is a diagram showing an example of the functional configuration of a speech processing device according to embodiment 4. FIG. 11 is a diagram showing an example of the functional configuration of a speech processing device according to embodiment 5. FIG. 12 is a diagram showing an example of the operation of a speech processing device according to embodiment 5. FIG. 13 is a diagram showing an example of the functional configuration of a speech processing device according to embodiment 6. FIG. 14 is a diagram showing an example of the configuration of a model generation unit according to embodiment 6.

[0009] Hereinafter, embodiments will be described with reference to the drawings. In the following description of the embodiments and the drawings, the same reference numerals denote the same or corresponding parts.

[0010] Embodiment 1. *** Description of Configuration *** Fig. 1 shows an example of the functional configuration of a voice processing device 100 according to this embodiment. Also, Fig. 2 shows an example of the hardware configuration of the voice processing device 100 according to this embodiment. The voice processing device 100 is a computer. The voice processing device 100 reduces processing distortion due to voice separation by limiting the target range of voice separation. More specifically, the voice processing device 100 reduces processing distortion due to voice separation by limiting the target range of voice separation to a time range in which mixed voice in which multiple voices are mixed exists. Note that the operating procedure of the voice processing device 100 corresponds to a voice processing method. Also, a program that realizes the operation of the voice processing device 100 corresponds to a voice processing program.

[0011] First, an example of the hardware configuration of the audio processing device 100 shown in FIG. 2 will be described.

[0012] The speech processing device 100 includes, as hardware, a processor 901, a main storage device 902, an auxiliary storage device 903, and a communication device 904. The speech processing device 100 also includes, as functional components, a feature extraction unit 101, a section detection unit 102, a speech separation unit 103, a speech restoration unit 104, and a speech connection unit 105 shown in FIG. 1 . The functional components of FIG. 1 are implemented, for example, by programs. The auxiliary storage device 903 stores programs that implement these functions. These programs are loaded from the auxiliary storage device 903 to the main storage device 902. The processor 901 then executes these programs to perform the operations of the functional components of FIG. 1 . FIG. 2 schematically illustrates a state in which the processor 901 is executing programs that implement the functions of the functional components of FIG. 1 . Although not shown in FIG. 2 , the speech processing device 100 may also include an input / output device. The input / output device may be, for example, a mouse, a keyboard, a recording medium reading device, a recording medium writing device, a display, etc.

[0013] Next, an example of the functional configuration of the voice processing device 100 shown in FIG. 1 will be described.

[0014] The feature extraction unit 101 acquires the mixed speech signal sequence 200. The feature extraction unit 101 acquires the mixed speech signal sequence 200 from a communication network via the communication device 904, for example. Alternatively, the feature extraction unit 101 may acquire the mixed speech signal sequence 200 from a recording medium reading device of an input / output device. The mixed speech signal sequence 200 is a time-series sequence of speech signals. The mixed speech signal sequence 200 includes a time range in which a single speech signal exists (hereinafter referred to as a single speech interval) and a time range in which a mixed speech signal exists (hereinafter referred to as a mixed speech interval). A single speech is a speech from a single speaker. A mixed speech is a speech in which multiple speeches from multiple speakers are mixed.

[0015] The feature extraction unit 101 extracts features of the audio signal included in the mixed audio signal sequence 200. For example, the feature extraction unit 101 extracts an amplitude spectrum obtained by short-time Fourier transform as a feature. Alternatively, the feature extraction unit 101 may extract features obtained by a neural network such as a multilayer perceptron or Conv1-D. The feature extraction unit 101 then outputs feature data 110, in which the extracted features are represented in time series, to the section detection unit 102. The feature data 110 represents the feature of a single audio segment in a single audio segment, and represents the feature of a mixed audio segment in a mixed audio segment. The feature extraction unit 101 also outputs the mixed audio signal sequence 200 to the section detection unit 102.

[0016] The section detection unit 102 acquires feature data 110 from the feature extraction unit 101 as time-series audio data. The section detection unit 102 then analyzes the features to detect an application section from the feature data 110. The application section is composed of a mixed audio section, a preceding section, and a following section. The mixed audio section is a time range in the feature data 110 in which mixed audio exists. The preceding section is a time range located immediately before the mixed audio section in the time series of the feature data 110 and in which a single audio exists. The following section is a time range located immediately after the mixed audio section in the time series of the feature data 110 and in which a single audio exists. The section detection unit 102 outputs application section features 120 to the audio separation unit 103. The application section features 120 are features of the application section. Furthermore, the section detection unit 102 outputs the mixed audio signal sequence 200 and non-mixed audio section definition information 1200 to the audio connection unit 105. The non-mixed voice section definition information 1200 is information that defines a non-mixed voice section in the mixed voice signal sequence 200. A non-mixed voice section is a section other than a mixed voice section in the mixed voice signal sequence 200. Specifically, a non-mixed voice section includes a forward section and a backward section. The forward section is a time range located before the mixed voice section in the time series of the mixed voice signal sequence 200. The backward section is a time range located after the mixed voice section in the time series of the mixed voice signal sequence 200. The processing performed by the section detection unit 102 corresponds to section detection processing.

[0017] The audio separation unit 103 acquires the application section feature 120 from the section detection unit 102. The audio separation unit 103 uses the application section feature 120 to separate the feature of the mixed audio in the mixed audio section into feature of multiple audios included in the mixed audio. As described above, strictly speaking, the audio separation unit 103 handles audio feature values ​​rather than the audio itself; however, for simplicity of explanation, the audio separation unit 103 may be described as handling the audio itself. In other words, the separation of the feature of the mixed audio by the audio separation unit 103 into feature values ​​of multiple audios may be described as the audio separation unit 103 separating the mixed audio into multiple audios. Hereinafter, each of the multiple audios obtained by separation by the audio separation unit 103 is referred to as a separated audio. In the processing by the audio separation unit 103, the term "separated audio" strictly refers to the feature of the separated audio. Furthermore, the terms "forward connected separated audio," "backward connected separated audio," "previous single audio," and "next single audio," which will be described later, strictly refer to the feature of each audio in the processing by the audio separation unit 103.

[0018] The audio separation unit 103 extracts forward-connected separated audio from the plurality of separated audio segments using the application section feature 120. The forward-connected separated audio is separated audio that is estimated to be the same audio as the immediately preceding single audio segment. The immediately preceding single audio segment is a single audio segment present in the immediately preceding section. The audio separation unit 103 also extracts backward-connected separated audio from the plurality of separated audio segments using the application section feature 120. The backward-connected separated audio is separated audio that is estimated to be the same audio as the immediately following single audio segment. The immediately following single audio segment is a single audio segment present in the immediately following section. The audio separation unit 103 outputs separated audio information 130 to the audio restoration unit 104. The separated audio information 130 includes features of each separated audio segment, an identifier of the separated audio corresponding to the forward-connected separated audio segment, an identifier of the separated audio corresponding to the backward-connected separated audio segment, a feature of the immediately preceding single audio segment, and a feature of the immediately following single audio segment. The processing performed by the audio separation unit 103 corresponds to audio separation processing.

[0019] The audio restoration unit 104 acquires separated audio information 130 from the audio separation unit 103. The audio restoration unit 104 restores the audio signal of each separated audio from the feature of each separated audio included in the separated audio information 130. The audio restoration unit 104 also restores the audio signal of the immediately preceding single audio from the feature of the immediately preceding single audio. The audio restoration unit 104 also restores the audio signal of the immediately succeeding single audio from the feature of the immediately succeeding single audio. The audio restoration unit 104 then outputs restored audio information 140 to the audio connection unit 105. The restored audio information 140 includes the audio signal of each restored separated audio, the audio signal of the immediately preceding single audio, the audio signal of the immediately succeeding single audio, an identifier of the separated audio corresponding to the forward connected separated audio, and an identifier of the separated audio corresponding to the backward connected separated audio.

[0020] The audio connection unit 105 acquires the mixed audio signal sequence 200 and unmixed audio section definition information 1200 from the section detection unit 102. The audio connection unit 105 also acquires restored audio information 140 from the audio restoration unit 104. The audio connection unit 105 connects the audio signal of the separated audio to the audio signal of the single audio in the mixed audio signal sequence 200 using the unmixed audio section definition information 1200 and the restored audio information 140. More specifically, the audio connection unit 105 connects the audio signal of the forward connected separated audio to the audio signal of the forward single audio. The forward single audio is a single audio present in the forward section. The audio connection unit 105 can identify the forward single audio by referring to the unmixed audio section definition information 1200. The audio connection unit 105 also connects the audio signal of the backward connected separated audio to the audio signal of the backward single audio. The backward single audio is a single audio present in the backward section. The audio connection unit 105 can identify the backward single audio by referring to the unmixed audio section definition information 1200. As a result of connecting the audio signals, the audio connection unit 105 generates a separated audio signal sequence 300 consisting only of audio signals of a single audio. The audio connection unit 105 then outputs the separated audio signal sequence 300. For example, the audio connection unit 105 transmits the separated audio signal sequence 300 to the sender of the mixed audio signal sequence 200 via the communication device 904. The audio connection unit 105 may also write the separated audio signal sequence 300 to a recording medium via a recording medium writing device of an input / output device.

[0021] ***Explanation of Operation*** Next, an example of operation of the audio processing device 100 according to this embodiment will be described with reference to FIGS.

[0022] Here, it is assumed that the feature data 110 shown in (a) of FIG. 3 has been generated by the feature extraction unit 101. The section detection unit 102 acquires the feature data 110 shown in (a) of FIG. 3. The waveforms in FIG. 3 show the transition of features over time. Note that the dashed waveform and the solid waveform in (a) of FIG. 3 represent features of the speech of different speakers. In the example of (a) of FIG. 3, as indicated by reference numeral 1101, while the speaker with the dashed waveform is speaking, the speaker with the solid waveform begins speaking, and the two voices are temporarily mixed together. Thereafter, the speaker with the dashed waveform stops speaking, and the speaker with the solid waveform continues speaking.

[0023] The section detection unit 102 analyzes the features shown in the feature data 110 in FIG. 3A and detects a time range in which speech exists from the feature data 110 as a speech section 121. Here, it is assumed that the section detection unit 102 has detected three speech sections 121 as shown in FIG. 3B. The section detection unit 102 can detect the speech section 121 by using, for example, the method shown in Reference 1 below. The section detection unit 102 can detect the speech section 121 by changing the configuration of the training data shown in Reference 1. Reference 1 uses a classifier that distinguishes between noise and speech. The section detection unit 102 can detect the speech section 121 by using a classifier that distinguishes between non-speech sections and speech sections instead of the classifier in Reference 1. Reference 1: International Publication WO2019162990A1

[0024] Next, the section detection unit 102 detects an application section 125 from the voice section 121. The application section 125 is made up of a previous section 122, a mixed voice section 123, and a subsequent section 124. The mixed voice section 123 is a time range in which mixed voice exists. The previous section 122 is a time range located immediately before the mixed voice section 123 in which a single voice exists. The subsequent section 124 is a time range located immediately after the mixed voice section 123 in which a single voice exists. The section detection unit 102 detects the mixed voice section 123 using the method described below. Then, the section detection unit 102 detects a predetermined time range immediately before the detected mixed voice section 123 as the previous section 122. Furthermore, the section detection unit 102 detects a predetermined time range immediately after the detected mixed voice section 123 as the subsequent section 124. This predetermined time can also be dynamically set depending on the positional relationship of the section detected by the section detection unit 102 with other sections. Specifically, when mixed voice segments 123 are adjacent to each other at a short time interval, the section detection unit 102 sets the preceding section 122 and the following section 124 based on the section information so that they do not overlap with other mixed voice segments. Dynamically setting the default time based on the positional relationship with other sections prevents mixed voices from another mixed voice segment from accidentally being mixed into the preceding section 122 or the following section 124, thereby improving the detection accuracy of the section detection unit 102. The section detection unit 102 may notify the user of the position and time section length of the detected mixed voice segment 123 using a display (not shown) or the like. Furthermore, when the default time is changed, the section detection unit 102 may notify the user of the position of the section whose default time has been changed and the new default time using a display (not shown) or the like. The user may be, for example, a speaker or an operator operating the speech processing device 100.

[0025] Method (1) of detecting the mixed speech section 123—rule-based The section detection unit 102 detects, for example, a point where the power of the speech signal suddenly increases in the speech section 121 as the start point of the mixed speech section 123. The section detection unit 102 also detects a point where the power of the speech signal suddenly decreases as the end point of the mixed speech section 123.

[0026] Method (2) for Detecting the Mixed Speech Section 123—Rule-Based The section detection unit 102 can also detect the mixed speech section 123 using the zero-crossing density. The zero-crossing density of the speech signal increases in the speech section 121. Utilizing this property, the section detection unit 102 first detects, as the speech section 121, a time range in which the zero-crossing density is equal to or greater than a threshold. It is also considered that the zero-crossing density will be even higher in the mixed speech section 123. For this reason, the section detection unit 102 detects, as the mixed speech section 123, a time range in the speech section 121 in which the zero-crossing density is even higher.

[0027] Method (3) of Detecting the Mixed Speech Section 123—Statistical Base The section detection unit 102 may detect the mixed speech section 123 by learning a Gaussian Mixture Model (GMM). For example, the section detection unit 102 learns a noise GMM and a speech GMM, compares the likelihood of each learning result, and detects the speech section 121. Furthermore, the section detection unit 102 learns a noise GMM, a single speech GMM, and a multiple speech GMM, and compares the likelihood of each learning result to detect the mixed speech section 123. In addition to using a GMM, the section detection unit 102 can detect the speech section 121 and the mixed speech section 123 by using high-dimensional features such as skewness. Furthermore, the section detection unit 102 may configure a Hidden Markov Model (HMM).

[0028] Method (4) of Detecting the Mixed Speech Section 123—Neural Network Based Alternatively, the section detection unit 102 may use a neural network to detect the mixed speech section 123. For example, the section detection unit 102 detects the speech section 121 and the mixed speech section 123 by learning using a multi-layer perceptron, a recurrent neural network (RNN), a long short term memory (LSTM), or the like.

[0029] The section detection unit 102 can use a combination of the above detection methods. For example, the section detection unit 102 can combine detection method (3) and detection method (4). In this case, the section detection unit 102 detects the voice section 121 and the mixed voice section 123 by adding together the statistically-based likelihood and the likelihood obtained from the neural network. Furthermore, the section detection unit 102 may exclude a detected section from subsequent processing if the length of the detected section is shorter than a set threshold. The threshold is a preset time section length and is set to a value shorter than a normal voice waveform. If the section detection unit 102 detects a section shorter than this threshold, it may have mistakenly detected a signal other than voice, such as noise. By excluding sections shorter than the threshold from subsequent processing, unnecessary voice separation processing is suppressed. This reduces the amount of processing and waveform distortion due to unnecessary voice separation processing.

[0030] The section detection unit 102 outputs the application section feature 120 , which is the feature of the application section 125 detected as described above, to the audio separation unit 103 .

[0031] The audio separation unit 103 separates the mixed audio in the mixed audio section 123 into multiple separated audio segments using the application section feature 120. The audio separation unit 103 can separate the mixed audio into multiple separated audio segments by using, for example, the technique disclosed in Patent Document 1. The audio separation unit 103 may also separate the mixed audio into multiple separated audio segments by learning the audio separation model disclosed in Patent Document 1. The audio separation unit 103 also extracts forward-connected separated audio segments and backward-connected separated audio segments from the multiple separated audio segments using the application section feature 120.

[0032] 4B shows a state in which the separated speech information 130 separates the mixed speech (a mixture of the dashed speech and the solid speech) of the mixed speech section 123 shown in FIG. 4A into separated speech. Separated speech 126 is the dashed speech separated from the mixed speech of the mixed speech section 123. Separated speech 127 is the solid speech separated from the mixed speech. Furthermore, the separated speech 126 is estimated to be the same speech as the immediately preceding single speech 128, and corresponds to forward-connected separated speech. The speech separation unit 103 estimates that the separated speech 126 and the immediately preceding single speech 128 are the same speech because the separated speech 126 is more similar to the immediately preceding single speech 128 than to the immediately following single speech 129 in terms of features. Furthermore, the separated speech 127 is estimated to be the same speech as the immediately following single speech 129, and corresponds to backward-connected separated speech. The audio separation unit 103 estimates that the separated audio 127 and the subsequent single audio 129 are the same audio because the separated audio 127 is more similar to the subsequent single audio 129 in terms of features than to the previous single audio 128. The audio separation unit 103 outputs separated audio information 130 to the audio restoration unit 104. As described above, the separated audio information 130 includes the features of the separated audio 126 and the separated audio 127, an identifier of the separated audio 126 corresponding to the forward connected separated audio, an identifier of the separated audio 127 corresponding to the backward connected separated audio, the features of the previous single audio 128, and the features of the subsequent single audio 129.

[0033] The audio restoration unit 104 restores the audio signals of the separated audio 126 and the separated audio 127 based on the separated audio information 130. The audio restoration unit 104 also restores the audio signal of the immediately preceding single audio 128 from the feature of the immediately preceding single audio 128. The audio restoration unit 104 also restores the audio signal of the immediately succeeding single audio 129 from the feature of the immediately succeeding single audio 129. The audio restoration unit 104 outputs the restored audio information 140 to the audio connection unit 105. As described above, the restored audio information 140 includes the audio signals of the restored separated audio 126 and the separated audio 127, the audio signal of the restored immediately preceding single audio 128, the audio signal of the restored immediately succeeding single audio 129, the identifier of the separated audio 126 corresponding to the forward connected separated audio, and the identifier of the separated audio 127 corresponding to the backward connected separated audio.

[0034] 4A shows a front section 1201 and a rear section 1202. The front section 1201 is a section in which a single voice is present and located ahead of the mixed voice section 123. The rear section 1202 is a section in which a single voice is present and located behind the mixed voice section 123. The non-mixed voice section definition information 1200 defines the front section 1201 and the rear section 1202 as non-mixed voice sections. As shown in FIG. 4C, the audio connection unit 105 refers to the non-mixed voice section definition information 1200 and identifies a front single voice signal 201, which is the voice signal of the front section 1201, and a rear single voice signal 202, which is the voice signal of the rear section 1202, in the mixed voice signal sequence 200. Then, the audio connection unit 105 connects the voice signal of the front connected separated voice 126 to the end of the front single voice signal 201 (the part corresponding to the previous single voice 128). Furthermore, the audio connection unit 105 connects the audio signal of the backward connected separated audio 127 to the beginning of the backward single audio signal 202 (the portion corresponding to the immediately following single audio 129).

[0035] The audio connection unit 105 may also determine whether the audio signal of the immediately preceding single audio 128 is similar to the tail portion of the forward single audio signal 201. In this case, if the audio signal of the immediately preceding single audio 128 is similar to the tail portion of the forward single audio signal 201, the audio connection unit 105 connects the audio signal of the forward connected separated audio 126 to the tail of the immediately preceding single audio signal 201. Similarly, the audio connection unit 105 may also determine whether the audio signal of the immediately following single audio 129 is similar to the head portion of the backward single audio signal 202. In this case, if the audio signal of the immediately following single audio 129 is similar to the head portion of the backward single audio signal 202, the audio connection unit 105 connects the audio signal of the backward connected separated audio 127 to the head of the backward single audio signal 202. The audio connection unit 105 determines the similarity using, for example, a correlation function, a mean square error, or another method. If the number of speakers is three or more, the audio connection unit 105 identifies the separated audio having the immediately preceding single audio 128 that is most similar to the end portion of the forward single audio signal 201 as the forward connected separated audio 126. Then, the audio connection unit 105 connects the audio signal of the forward connected separated audio 126 to the end of the immediately preceding single audio signal 201. Similarly, the audio connection unit 105 identifies the separated audio having the immediately following single audio 129 that is most similar to the start portion of the backward single audio signal 202 as the backward connected separated audio 127. Then, the audio connection unit 105 connects the audio signal of the backward connected separated audio 127 to the start of the backward single audio signal 202.

[0036] In the example of Figure 4(c), the audio connection unit 105 connects the audio signal of the forward connected separated audio 126 to the forward single audio signal 201 and the audio signal of the backward connected separated audio 127 to the backward single audio signal 202 using different audio output channels. Note that in this embodiment, it is not determined which audio signal belongs to which speaker. For this reason, as shown in the lower part of Figure 4(c), the audio signal indicated by the solid line and the audio signal indicated by the dashed line 2020, which are audio signals of different speakers, may be output on the same audio output channel.

[0037] Furthermore, as shown in Figure 5, there are cases where the continuously occurring voice of one speaker is temporarily overlapped with the voice of another speaker. In the example of Figure 5, the speaker represented by the dashed waveform starts speaking while the speaker represented by the solid waveform begins speaking, temporarily mixing the two voices. However, the speaker represented by the solid waveform finishes speaking in a short time, and thereafter only the speaker represented by the dashed waveform continues speaking.

[0038] In this case, as shown in (a) of FIG. 5, the section detection unit 102 detects the time range in which the solid line waveform and the dashed line waveform overlap as a mixed voice section 123. The section detection unit 102 also detects the time range of the single voice immediately before and after the mixed voice section 123 as a immediately before section 122 and a immediately after section 124. The voice separation unit 103 separates the mixed voice into separated voice 126 and separated voice 1270 as shown in (b) of FIG. 5. In (b) of FIG. 5, the separated voice 126 corresponds to the forward-connected separated voice and the backward-connected separated voice. That is, the separated voice 126 is estimated to be the same voice as the immediately before single voice 128 and the immediately after single voice 129. On the other hand, the separated voice 1270 corresponds to neither the forward-connected separated voice nor the backward-connected separated voice.

[0039] In this case, as shown in (c) of Figure 5, the audio connection unit 105 connects the audio signal of separated audio 126 to the end of the forward single audio signal 203 (the part corresponding to the immediately preceding single audio 128). The audio connection unit 105 also connects the audio signal of separated audio 126 to the beginning of the backward single audio signal 204 (the part corresponding to the immediately following single audio 129). The forward single audio signal 203 is the audio signal of the forward section 1203. The backward single audio signal 204 is the audio signal of the backward section 1204. On the other hand, since separated audio 1270 does not correspond to either forward-connected separated audio or backward-connected separated audio, the audio connection unit 105 does not connect the separated audio 1270 to any single audio in any time range. The audio connection unit 105 then sets the separated audio 1270 to an audio output channel (back channel) different from the audio output channels to which the audio signal of separated audio 126 is connected to the forward single audio signal 203 and the backward single audio signal 204.

[0040] As described above, the audio connection unit 105 may determine the similarity between the audio signal of the immediately preceding single audio 128 and the end portion of the forward single audio signal 203, and the similarity between the audio signal of the immediately following single audio 129 and the beginning portion of the backward single audio signal 204.

[0041] 6, the audio connection unit 105 may connect the audio signal of the forward connection separated audio 126 to the forward single audio signal 201 and connect the audio signal of the backward connection separated audio 127 to the backward single audio signal 202 on the same audio output channel. (a) and (b) of FIG. 6 are the same as (a) and (b) of FIG. 4. In (c) of FIG. 6, the audio connection unit 105 connects the audio signal of the forward connection separated audio 126 to the forward single audio signal 201 and connects the audio signal of the backward connection separated audio 127 to the backward single audio signal 202 on the same audio output channel. More specifically, the audio connection unit 105 sets the single audio signal obtained by connecting the audio signal of the backward connection separated audio 127 to the backward single audio signal 202 behind the single audio signal obtained by connecting the audio signal of the forward connection separated audio 126 to the forward single audio signal 201 on the same audio output channel. Here, the audio connection unit 105 shifts the single audio obtained by connecting the audio signal of the backward connected separated audio 127 and the backward single audio signal 202 backward in time.

[0042] 5 exists, the audio connection unit 105 may set the audio signal of the separated audio 1270 to an audio output channel where the audio signal of the separated audio 126 is connected to the front single audio signal 203 and the rear single audio signal 204, as shown in FIG. 7. (a) and (b) of FIG. 7 are the same as (a) and (b) of FIG. 5. In (c) of FIG. 7, the audio connection unit 105 sets the audio signal of the separated audio 1270 to an audio output channel where the audio signal of the separated audio 126 is connected to the front single audio signal 203 and the rear single audio signal 204-1. More specifically, the audio connection unit 105 does not connect the audio signal of the separated audio 1270 to any single audio in any time range, but sets the audio signal of the separated audio 1270 behind the audio signal of the single audio obtained by connecting the front single audio signal 203, the audio signal of the separated audio 126, and the rear single audio signal 204-1. 7(c), the rear single audio signal 204 is separated into a rear single audio signal 204-1 and a rear single audio signal 204-2. As in the case of FIG. 6, the audio connection unit 105 shifts the audio signal of the separated audio 1270 and the rear single audio signal 204-2 backward in time.

[0043] ***Description of Effects of the Embodiment*** In this embodiment, only mixed audio segments that require audio separation are detected as targets for audio separation. In conventional technology, audio separation is performed on the entire input audio. This causes processing distortion to occur on the entire audio, and the processing distortion has an adverse effect on post-processing. In this embodiment, by limiting the target range for audio separation, it is possible to reduce the adverse effects (processing distortion) caused by audio separation.

[0044] Furthermore, even if there is a processing delay in the audio separation, the processing delay can be reduced because the range of audio separation is limited in this embodiment.

[0045] Furthermore, even if the calculation cost of speech separation is high, the calculation cost can be reduced because the range of speech separation is limited in this embodiment.

[0046] Embodiment 2 In this embodiment, the adverse effects of processing distortion in audio restoration are alleviated. In this embodiment, differences from embodiment 1 will be mainly explained. Note that matters not explained below are the same as in embodiment 1.

[0047] Fig. 8 shows an example of the functional configuration of the speech processing device 100 according to this embodiment. Compared to Fig. 1 , in Fig. 8 , the segment detection unit 102 outputs a likelihood 1231 of the mixed speech segment 123 to the speech restoration unit 104.

[0048] In this embodiment, the section detection unit 102 calculates likelihood 1231 of the mixed speech section 123 in addition to the operation described in the first embodiment. Then, the section detection unit 102 outputs the calculated likelihood 1231 to the speech restoration unit 104.

[0049] The speech restoration unit 104 restores the speech signal of each separated speech using the likelihood 1231. Specifically, the speech restoration unit 104 first calculates a weighted feature W using the following equation 1, and then restores the speech signal of each separated speech using this weighted feature W as an input. In the following equation 1, alpha is the likelihood 1231. Furthermore, W_before is a feature of the mixed speech before separation. W_after is a feature of the speech after separation (in the example of FIG. 4, separated speech 126 and separated speech 127). W = alpha × W_after + (1 - alpha) × W_before Equation 1 However, in this embodiment, the above W, W_after, and W_before may be the speech signals themselves. In this case, the separated speech restored by the speech restoration unit 104 is W_after, the speech before separation is W_before, and the speech W obtained by taking the weighted sum of Equation 1 is the output of the speech restoration unit 104. For example, the following Reference 2 discloses that a weighted sum of speech before and after separation is calculated as a process for mitigating the adverse effects of processing distortion. Reference 2: Japanese Patent No. 7345702

[0050] ***Explanation of Effects of the Embodiment*** In this embodiment, processing distortion can be reduced by calculating a weighted sum of pre-separation speech and post-separation speech using the likelihood of a mixed speech segment. As a result, this embodiment can reduce the adverse effects of processing distortion on post-processing.

[0051] Embodiment 3 In this embodiment, an example in which feature data 110 is divided into predetermined sizes will be described. In this embodiment, differences from embodiment 1 will be mainly described. Note that matters not described below are the same as in embodiment 1.

[0052] Fig. 9 shows an example of the functional configuration of a speech processing device 100 according to this embodiment. Compared to Fig. 1 , Fig. 9 adds a first division unit 131 and a second division unit 132. In addition, the feature extraction unit 101 outputs feature data 110 and a mixed speech signal sequence 200 to the first division unit 131, and outputs the feature data 110 to the second division unit 132.

[0053] The functions of the first division unit 131 and the second division unit 132 are also realized by a program, similar to the feature extraction unit 101 etc. The program realizing the functions of the first division unit 131 and the second division unit 132 is executed by the processor 901.

[0054] The first division unit 131 divides the feature data 110 into pieces of a predetermined size. The first division unit 131 divides the feature data 110 into pieces of a size suitable for the section detection unit 102 to detect the mixed speech section 123. The first division unit 131 then outputs divided feature data 1131 obtained by dividing the feature data 110 and the mixed speech signal sequence 200 to the section detection unit 102. The section detection unit 102 acquires the divided feature data 1131 instead of the feature data 110 and detects the mixed speech section 123 using the divided feature data 1131. The detection process for the mixed speech section 123 in the section detection unit 102 is the same as that described in embodiment 1, and therefore will not be described again. Note that in this embodiment, the section detection unit 102 outputs application section definition information 1230 to the speech separation unit 103 instead of the application section feature 120. The application section definition information 1230 is information that defines the application section 125 .

[0055] The second division unit 132 divides the feature data 110 into predetermined sizes. The second division unit 132 divides the feature data 110 into sizes suitable for audio separation by the audio separation unit 103. The second division unit 132 may divide the feature data 110 into the same sizes as the division sizes used by the first division unit 131, or may divide the feature data 110 into sizes different from those used by the first division unit 131. The second division unit 132 then outputs divided feature data 1132 obtained by dividing the feature data 110 to the audio separation unit 103. The audio separation unit 103 acquires the divided feature data 1132 from the second division unit 132. The audio separation unit 103 also acquires application section definition information 1230 from the application section feature 120. The audio separation unit 103 identifies the application section 125 using the application section definition information 1230, and performs audio separation for the identified application section 125 using the division feature data 1132. The audio separation process itself in the audio separation unit 103 is the same as that described in embodiment 1, and therefore a description thereof will be omitted.

[0056] In this embodiment, the section detection unit 102 operates using division feature data 1131, and the audio separation unit 103 operates using division feature data 1132. Therefore, the section detection unit 102 and the audio separation unit 103 can operate in parallel. Furthermore, the first division unit 131 may divide the feature data 110 by providing overlapping portions between successive division feature data 1131. Similarly, the second division unit 132 may divide the feature data 110 by providing overlapping portions between successive division feature data 1132.

[0057] ***Description of Effects of the Embodiment*** In this embodiment, mixed speech segments are detected using division feature data. Therefore, in this embodiment, the processing delay in the detection of mixed speech segments can be reduced to the size of the division feature data. Similarly, in this embodiment, speech separation is performed using division feature data. Therefore, in this embodiment, the processing delay in speech separation can be reduced to the size of the division feature data.

[0058] Furthermore, in this embodiment, feature data is divided into sizes suitable for detecting mixed speech segments. Therefore, according to this embodiment, it is possible to reduce deterioration in accuracy in detecting mixed speech segments due to feature data division. Similarly, in this embodiment, feature data is divided into sizes suitable for speech separation. Therefore, according to this embodiment, it is possible to reduce deterioration in accuracy in speech separation due to feature data division.

[0059] Fourth Embodiment In this embodiment, an example of extracting features suitable for detecting mixed speech segments and features suitable for speech separation will be described. In this embodiment, differences from the first embodiment will be mainly described. Note that matters not described below are the same as those in the first embodiment.

[0060] Fig. 10 shows an example of the functional configuration of a speech processing device 100 according to this embodiment. Compared to Fig. 1, Fig. 10 includes a first feature extraction unit 141 and a second feature extraction unit 142 instead of the feature extraction unit 101.

[0061] The functions of the first feature amount extraction unit 141 and the second feature amount extraction unit 142 are also realized by a program, similar to the feature amount extraction unit 101 etc. The program realizing the functions of the first feature amount extraction unit 141 and the second feature amount extraction unit 142 is executed by the processor 901.

[0062] The first feature extraction unit 141 acquires the mixed audio signal sequence 200. The first feature extraction unit 141 acquires the mixed audio signal sequence 200 from a communication network via the communication device 904, for example. Alternatively, the first feature extraction unit 141 may acquire the mixed audio signal sequence 200 from a recording medium reading device of an input / output device. The first feature extraction unit 141 then extracts features from the mixed audio signal sequence 200 that are suitable for the section detection unit 102 to detect the mixed audio section 123. The first feature extraction unit 141 then outputs first feature data 1141 indicating the extracted features and the mixed audio signal sequence 200 to the section detection unit 102. The first feature extraction unit 141 also outputs the mixed audio signal sequence 200 to the second feature extraction unit 142.

[0063] The section detection unit 102 acquires first feature data 1141 instead of the feature data 110, and detects the mixed voice section 123 using the first feature data 1141. The detection process for the mixed voice section 123 by the section detection unit 102 is the same as that described in embodiment 1, and therefore a description thereof will be omitted. Note that in this embodiment, the section detection unit 102 outputs application section definition information 1240 to the voice separation unit 103 instead of the application section feature 120. The application section definition information 1240 is information that defines the application section 125.

[0064] The second feature extraction unit 142 extracts features suitable for speech separation by the speech separation unit 103 from the mixed speech signal sequence 200. The second feature extraction unit 142 may extract the same type of feature as the feature extracted by the first feature extraction unit 141, or may extract a different type of feature. The second feature extraction unit 142 then outputs second feature data 1142 indicating the extracted feature to the speech separation unit 103.

[0065] The audio separation unit 103 acquires second feature data 1142 from the second feature extraction unit 142. The audio separation unit 103 also acquires application section definition information 1240 from the application section feature 120. The audio separation unit 103 identifies an application section 125 using the application section definition information 1240, and performs audio separation for the identified application section 125 using the second feature data 1142. The audio separation process itself in the audio separation unit 103 is the same as that described in embodiment 1, and therefore description thereof will be omitted.

[0066] In this embodiment, the section detection unit 102 operates using the first feature data 1141, and the audio separation unit 103 operates using the second feature data 1142. Therefore, the section detection unit 102 and the audio separation unit 103 can operate in parallel.

[0067] ***Description of Effects of the Embodiment*** In this embodiment, features suitable for detecting mixed speech segments are extracted. Therefore, according to this embodiment, it is possible to improve the accuracy in detecting mixed speech segments. Furthermore, it is possible to use a model suitable for detecting mixed speech segments. Similarly, in this embodiment, features suitable for speech separation are extracted. Therefore, according to this embodiment, it is possible to improve the accuracy in speech separation. Furthermore, it is possible to use a model suitable for speech separation.

[0068] Embodiment 5 In this embodiment, an example of speech enhancement will be described. In this embodiment, differences from embodiment 1 will be mainly described. Note that matters not described below are the same as embodiment 1.

[0069] Fig. 11 shows an example of the functional configuration of a speech processing device 100 according to this embodiment. Compared to Fig. 1, Fig. 11 additionally includes a speech enhancement unit 106.

[0070] The function of the speech enhancement unit 106 is also realized by a program, similar to the feature extraction unit 101 etc. The program that realizes the function of the speech enhancement unit 106 is executed by the processor 901.

[0071] The speech enhancement unit 106 performs speech enhancement to separate noise from speech. The speech enhancement unit 106 uses, for example, the speech enhancement technology disclosed in the following Reference 3. Reference 3: https: / / www.merl.com / publications / docs / TR2016-113.pdf

[0072] In this embodiment, in addition to the processing described in the first embodiment, the section detection unit 102 determines the noise level in the feature data 110. Then, the section detection unit 102 determines a time range for which the speech enhancement unit 106 performs speech enhancement (hereinafter referred to as a speech enhancement section) according to the noise level. For example, as shown in FIG. 12 , the section detection unit 102 specifies a time range that is shorter than the speech section 121 and longer than the application section 125 as a speech enhancement section 1255. The section detection unit 102 detects the speech enhancement section 1255 using the method described below. Then, the section detection unit 102 outputs a speech enhancement section feature 1251 and application section definition information 1252 to the speech enhancement unit 106. The speech enhancement section feature 1251 is a feature of the speech enhancement section 1255. The application section definition information 1252 is information that defines the application section 125.

[0073] Method (1) of detecting the speech emphasis section 1255—rule-based When the power of the speech signal at a point in time slightly before the start point of the speech section 121 (hereinafter referred to as the candidate start point) is higher than the power of the speech signal at a point in time a fixed time before the candidate start point (in other words, when the power of the speech signal at the candidate start point is higher than that of the non-speech section before the candidate start point), the section detection unit 102 detects the candidate start point as the start point of the speech emphasis section 1255. Also, when the power of the speech signal at a point in time slightly after the end point of the speech section 121 (hereinafter referred to as the candidate end point) is higher than the power of the speech signal at a point in time a fixed time after the candidate end point, the section detection unit 102 detects the candidate end point as the end point of the speech emphasis section 1255.

[0074] Method (2) for Detecting the Speech Enhancement Section 1255—Statistical Base The section detection unit 102 may detect the speech enhancement section 1255 by learning a GMM. For example, the section detection unit 102 learns a noise GMM and a speech GMM, compares the likelihood of the respective learning results, and detects the speech section 121. When detecting the speech enhancement section 1255, the section detection unit 102 learns the noise GMM using high-power noise or special noise to which speech enhancement should be applied. Furthermore, the section detection unit 102 may configure an HMM.

[0075] Method (3) for Detecting the Emphasized Speech Section 1255—Neural Network Based The section detection unit 102 detects the speech section 121 and the emphasized speech section 1255 by learning using a multilayer perceptron, RNN, LSTM, or the like.

[0076] The speech enhancement unit 106 performs speech enhancement on the speech enhancement section feature 1251. Then, the speech enhancement unit 106 outputs the speech enhancement application section feature 160 to the speech separation unit 103. The speech enhancement application section feature 160 is a feature of the application section 125 after speech enhancement. The speech enhancement unit 106 outputs, as the speech enhancement application section feature 160, a feature of the speech enhancement section feature 1251 after speech enhancement that corresponds to the application section 125 defined in the application section definition information 1252.

[0077] The speech separation unit 103 performs speech separation using the speech enhancement application section feature 160 instead of the application section feature 120. The speech separation process itself in the speech separation unit 103 is the same as that described in the first embodiment, and therefore a description thereof will be omitted.

[0078] 10 , a configuration in which a first feature extraction unit 141 and a second feature extraction unit 142 are provided instead of the feature extraction unit 101 may be added with the speech enhancement unit 106. In this case, a third feature extraction unit that extracts a third feature suitable for speech enhancement may be provided. In this case, the third feature extraction unit inputs the third feature to the speech enhancement unit 106.

[0079] ***Explanation of Effects of the Embodiment*** According to this embodiment, noise can be removed by speech enhancement. Furthermore, in this embodiment, the range of speech enhancement is limited according to the noise level, so that processing distortion due to speech enhancement can be limited.

[0080] Furthermore, even if there is a processing delay in speech enhancement, the processing delay can be reduced because the target range of speech enhancement is limited in this embodiment. Also, even if the calculation cost of speech enhancement is high, the calculation cost can be reduced because the target range of speech enhancement is limited in this embodiment.

[0081] Sixth Embodiment In this embodiment, an example of generating a model for deriving the likelihood 1231 of the mixed speech section 123 described in the second embodiment will be described. In this embodiment, differences from the second embodiment will be mainly described. Note that matters not described below are the same as those in the second embodiment.

[0082] Fig. 13 shows an example of the functional configuration of the speech processing device 100 according to this embodiment. Compared to Fig. 8, Fig. 13 additionally includes a model generating unit 107 and a model storage unit 108.

[0083] The functions of the model generation unit 107 are also realized by a program, similar to the feature extraction unit 101 etc. The program that realizes the functions of the model generation unit 107 is executed by the processor 901. The model storage unit 108 is realized by, for example, the auxiliary storage device 903.

[0084] In the learning phase, the model generation unit 107 generates a likelihood derivation model 170 through learning. The likelihood derivation model 170 is a model for deriving the likelihood 1231 of the mixed speech section 123. The model generation unit 107 stores the generated likelihood derivation model 170 in the model storage unit 108. The model storage unit 108 stores the likelihood derivation model 170.

[0085] In this embodiment, in the inference phase, the section detection unit 102 derives the likelihood 1231 using the likelihood derivation model 170. Then, the section detection unit 102 outputs the derived likelihood 1231 to the speech restoration unit 104. As described in the second embodiment, the speech restoration unit 104 restores the speech signal of each separated speech using the likelihood 1231.

[0086] Specifically, the model generation unit 107 generates the likelihood derivation model 170 as follows. Here, it is assumed that the speech separation unit 103 learns the speech separation model described in Patent Document 1 and separates the mixed speech into a plurality of separated speeches. The loss function of the speech separation model used by the speech separation unit 103 to learn the speech separation model is defined as L_sep(W_true, W_after). Here, W_true is the true speech signal. W_after is the speech signal after separation. The likelihood 1231 output by the section detection unit 102 is defined as E_osd. In this case, the loss function of the speech separation model used to learn the speech separation model in this embodiment is defined as L_sep(W_true', W_after). W_true' can be expressed by the following equation 2. W_true'=E_osd×W_true+(1-E_osd)×W_before Formula 2

[0087] The model generation unit 107 is made up of, for example, training data 171, a voice mixing unit 172, a feature extraction unit 173, a teacher generation unit 174, and a model training unit 175, as shown in FIG.

[0088] ***Explanation of Effects of the Embodiment*** In this embodiment, a model for deriving the likelihood of a mixed speech segment is generated. Therefore, according to this embodiment, it is possible to derive the likelihood more accurately and effectively reduce processing distortion.

[0089] Although the first to sixth embodiments have been described above, two or more of these embodiments may be combined and implemented. Alternatively, one of these embodiments may be partially implemented. Alternatively, two or more of these embodiments may be partially combined and implemented. Furthermore, the configurations and procedures described in these embodiments may be modified as necessary.

[0090] *** Use Cases *** The following use cases are envisioned for the speech processing device 100 described in the first to sixth embodiments. (1) When multiple speakers' voices overlap in a meeting, lecture, etc., the voices of each speaker are separated and speech recognition is performed for recording, transcription, etc. (2) When the voices of a salesperson and a customer overlap in face-to-face sales, the voices of each speaker are separated and recorded, and each speaker's voice is saved as evidence. (3) When recognizing a user's instructions, questions, etc. to a voice assistant using a smartphone, smart speaker, etc., the voices of other speakers are removed. This allows the smartphone, smart speaker, etc. to accurately understand the user's instructions, questions, etc. (4) For people with hearing difficulties, hearing aids, headsets, etc. are developed that separate the voices of multiple speakers and amplify the voice of the conversation partner to make it easier to hear. (5) When evaluating spoken language communication skills in education, training, etc., the voices of multiple speakers are separated and speech content and fluency are measured. (6) When listening to the voices of patients and users in medical, nursing, and other settings, the voices of other speakers are removed. This will improve the quality of diagnostic and nursing care services.

[0091] *** Supplementary Explanation of Hardware Configuration *** Here, a supplementary explanation of the hardware configuration of the audio processing device 100 will be provided. The processor 901 shown in FIG. 2 is an IC (Integrated Circuit) that performs processing. The processor 901 is a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or the like. The main storage device 902 shown in FIG. 2 is a RAM (Random Access Memory). The auxiliary storage device 903 shown in FIG. 2 is a ROM (Read Only Memory), a flash memory, an HDD (Hard Disk Drive), or the like. The communication device 904 shown in FIG. 2 is an electronic circuit that executes data communication processing. The communication device 904 is, for example, a communication chip or a NIC (Network Interface Card).

[0092] The auxiliary storage device 903 also stores an OS (Operating System). At least a portion of the OS is executed by the processor 901. While executing at least a portion of the OS, the processor 901 executes programs that implement the functions of the functional components shown in FIGS. 1, 8-11, and 13 (hereinafter referred to as FIG. 1, etc.). Task management, memory management, file management, communication control, etc. are performed by the processor 901 executing the OS. At least one of information, data, signal values, and variable values ​​indicating the results of processing of the functional components shown in FIG. 1, etc. is stored in at least one of the main storage device 902, the auxiliary storage device 903, and registers and cache memory within the processor 901. The programs that implement the functions of the functional components shown in FIG. 1, etc. may be stored on portable recording media such as a magnetic disk, flexible disk, optical disk, compact disk, Blu-ray (registered trademark) disk, DVD, etc. Portable recording media storing programs that implement the functions of the functional components shown in FIG. 1, etc. may be distributed.

[0093] Furthermore, at least one "unit" of the functional components shown in FIG. 1 and the like may be read as a "circuit," a "step," a "procedure," a "process," or a "circuitry." Furthermore, the audio processing device 100 may be realized by a processing circuit. The processing circuit is, for example, a logic IC (Integrated Circuit), a GA (Gate Array), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array). In this case, the functional components shown in FIG. 1 and the like are each realized as part of the processing circuit. In this specification, the generic concept of a processor and a processing circuit is referred to as a "processing circuitry." In other words, a processor and a processing circuit are each specific examples of "processing circuitry."

[0094] Various aspects of the present disclosure are described below as appendices. (Appendix 1) An audio processing device having: a section detection unit that detects, from time-series audio data, a mixed audio section that is a time range in which a mixed audio containing a mixture of multiple audio segments exists; and an audio separation unit that separates the mixed audio from the mixed audio section into the multiple audio segments. (Appendix 2) The audio processing device according to Appendix 1, wherein the section detection unit detects, from the audio data, the mixed audio section, a previous section that is a time range in which a single audio segment exists and that is located immediately before the mixed audio section in the time series of the audio data, and a subsequent section that is a time range in which a single audio segment exists and that is located immediately after the mixed audio section in the time series of the audio data. (Appendix 3) The audio processing device according to Appendix 1 or 2, further having an audio connection unit that connects at least one separated audio from a plurality of separated audio segments that are the multiple audio segments obtained by separation by the audio separation unit to a single audio segment. (Supplementary Note 4) The audio processing device according to Supplementary Note 3, wherein the audio connection unit connects any of the plurality of separated audios to a forward single audio that is a single audio that exists in a forward section that is a time range that is located before the mixed audio section in the time series of the audio data, and connects any of the plurality of separated audios to a backward single audio that is a single audio that exists in a backward section that is a time range that is located after the mixed audio section in the time series of the audio data.(Supplementary Note 5) The audio processing device according to Supplementary Note 4, wherein the section detection unit detects from the audio data the mixed audio section, a previous section which is a time range located immediately before the mixed audio section in the time series of the audio data and in which a single audio section exists, and a subsequent section which is a time range located immediately after the mixed audio section in the time series of the audio data and in which a single audio section exists, the audio separation unit extracts, from the plurality of separated audio sections, a separated audio which is estimated to be the same as the previous single audio which is a single audio section existing in the previous section, as a forward-connected separated audio, and extracts, from the plurality of separated audio sections, a separated audio which is estimated to be the same as the subsequent single audio which is a single audio section existing in the subsequent section, as a backward-connected separated audio, the audio connection unit connects the forward-connected separated audio to the forward single audio, and connects the backward-connected separated audio to the backward single audio. (Supplementary Note 6) The audio processing device according to Supplementary Note 5, wherein the audio connection unit connects the forward-connected separated audio to the forward single audio and connects the backward-connected separated audio to the backward single audio using different audio output channels. (Supplementary Note 7) The audio processing device according to Supplementary Note 5, wherein, when there is a separated audio among the plurality of separated audios that does not correspond to either the forward-connected separated audio or the backward-connected separated audio, the audio connection unit does not connect the separated audio to any single audio in any time range, but sets the separated audio to an audio output channel that is different from the audio output channel to which the forward-connected separated audio is connected to the forward single audio and the audio output channel to which the backward-connected separated audio is connected to the backward single audio. (Supplementary Note 8) The audio processing device according to Supplementary Note 5, wherein the audio connection unit connects the forward-connected separated audio to the forward single audio and the backward-connected separated audio to the backward single audio on the same audio output channel. (Supplementary Note 9) The audio processing device according to Supplementary Note 8, wherein the audio connection unit sets a single audio obtained by connecting the backward-connected separated audio, the immediately following single audio, and the backward single audio behind the single audio obtained by connecting the forward single audio, the immediately preceding single audio, and the forward-connected separated audio on the same audio output channel.(Appendix 10) The audio processing device described in Appendix 8, wherein, when there is a separated audio among the plurality of separated audios that does not correspond to either the forward-connected separated audio or the backward-connected separated audio, the audio connection unit does not connect the separated audio to a single audio of any time range, but sets the separated audio to an audio output channel where the forward-connected separated audio is connected to the forward single audio and the backward-connected separated audio is connected to the backward single audio. (Supplementary Note 11) The section detection unit acquires, as the speech data, time-series feature data in which features of a single speech are indicated in a single speech section that is a time range in which a single speech exists, and features of the mixed speech are indicated in the mixed speech section, and detects the mixed speech section from the feature data by analyzing the features indicated in the feature data, the speech separation unit separates the features of the mixed speech in the mixed speech section into features of the multiple speeches, the speech processing device further has a speech restoration unit that restores multiple separated speeches that are the multiple speeches from the features of the multiple speeches obtained by separation by the separation unit, and the speech connection unit connects at least one separated speech of the multiple separated speeches obtained by restoration by the speech restoration unit to one single speech. (Supplementary Note 12) The speech processing device according to Supplementary Note 11, in which the speech restoration unit restores the multiple separated speeches using the likelihood of the mixed speech section. (Supplementary Note 13) The audio processing device further includes a first division unit that divides the audio data into sizes suitable for detection of the mixed audio section by the section detection unit, and a second division unit that divides the audio data into sizes suitable for audio division by the audio separation unit, wherein the section detection unit detects the mixed audio section from the audio data after division by the first division unit, and the audio separation unit separates the mixed audio of the mixed audio section into the multiple audio sections using the audio data after division by the second division unit.(Supplementary Note 14) The audio processing device further includes: a first feature extraction unit that generates first feature data indicating features suitable for detection of the mixed audio segment by the segment detection unit; and a second feature extraction unit that generates second feature data indicating features suitable for audio division by the audio separation unit, wherein the segment detection unit acquires the first feature data as the audio data and detects the mixed audio segment from the first feature data, and the audio separation unit separates the mixed audio of the mixed audio segment into the plurality of audio segments using the second feature data. (Supplementary Note 15) The audio processing device according to Supplementary Note 1, further includes a speech enhancement unit that performs speech enhancement to separate noise and audio, and the segment detection unit determines a time range for which the speech enhancement unit is to perform speech enhancement according to a noise level in the audio data. (Supplementary Note 16) The speech processing device according to Supplementary Note 12, further comprising: a model generation unit that generates a likelihood derivation model that is a model for deriving the likelihood of the mixed speech section; and the speech restoration unit that restores the multiple separated speeches using the likelihood of the mixed speech section derived using the likelihood derivation model. (Supplementary Note 17) A speech processing method in which a computer detects a mixed speech section, which is a time range in which a mixed speech in which multiple speeches are mixed, from time-series speech data, and separates the mixed speech in the mixed speech section into the multiple speeches. (Supplementary Note 18) A speech processing program that causes a computer to execute a section detection process that detects a mixed speech section, which is a time range in which a mixed speech in which multiple speeches are mixed, from time-series speech data, and a speech separation process that separates the mixed speech in the mixed speech section into the multiple speeches.

[0095] 100 Speech processing device, 101 Feature extraction unit, 102 Section detection unit, 103 Speech separation unit, 104 Speech restoration unit, 105 Speech connection unit, 106 Speech enhancement unit, 107 Model generation unit, 108 Model storage unit, 110 Feature data, 120 Application section feature, 121 Speech section, 122 Previous section, 123 Mixed speech section, 124 Next section, 125 Application section, 126 Separated speech, 127 Separated speech, 128 Previous single speech, 129 Next single speech, 130 Separated speech information, 131 First division unit, 132 Second division unit, 140 Restored speech information, 141 First feature extraction unit, 142 Second feature extraction unit, 160 Speech enhancement application section feature, 170 Likelihood derivation model, 171 Training data, 172 Speech mixing unit, 173 feature extraction unit, 174 teacher generation unit, 175 model learning unit, 200 mixed speech signal sequence, 201 front single speech signal, 202 rear single speech signal, 203 front single speech signal, 204 rear single speech signal, 300 separated speech signal sequence, 901 processor, 902 main storage device, 903 auxiliary storage device, 904 communication device, 1131 segmentation feature data, 1132 segmentation feature data, 1141 first feature data, 1142 second feature data, 1200 non-mixed speech section definition information, 1201 front section, 1202 rear section, 1203 front section, 1204 rear section, 1230 application section definition information, 1231 likelihood, 1240 application section definition information, 1251 speech enhancement section feature, 1252 Application section definition information, 1255 speech enhancement section, 1270 separated speech, 2020 speech signal.

Claims

1. A sound processing device having a section detection unit that detects a mixed sound section, which is a time range in which a mixed sound containing multiple sounds exists, from time-series sound data, and a sound separation unit that separates the mixed sound in the mixed sound section into the multiple sounds.

2. The audio processing device according to claim 1, wherein the section detection unit detects from the audio data the mixed audio section, an immediately preceding section which is a time range located immediately before the mixed audio section in the time series of the audio data and in which a single audio section exists, and an immediately following section which is a time range located immediately after the mixed audio section in the time series of the audio data and in which a single audio section exists.

3. The audio processing device according to claim 1 or 2, further comprising an audio connection unit that connects at least one of the separated audios, which are the multiple audios obtained by separation by the audio separation unit, to any single audio.

4. The audio processing device described in claim 3, wherein the audio connection unit connects any of the plurality of separated audio signals to a forward single audio signal, which is a single audio signal present in a forward section, which is a time range located before the mixed audio section in the time series of the audio data, and connects any of the plurality of separated audio signals to a backward single audio signal, which is a single audio signal present in a backward section, which is a time range located after the mixed audio section in the time series of the audio data.

5. The audio processing device described in claim 4, wherein the section detection unit detects from the audio data the mixed audio section, an immediately preceding section which is a time range located immediately before the mixed audio section in the time series of the audio data and in which a single audio section exists, and an immediately succeeding section which is a time range located immediately after the mixed audio section in the time series of the audio data and in which a single audio section exists; the audio separation unit extracts, from the plurality of separated audios, separated audio which is estimated to be the same audio as the immediately preceding single audio which is a single audio section existing in the immediately preceding section, as forward-connected separated audio; and extracts, from the plurality of separated audios, separated audio which is estimated to be the same audio as the immediately succeeding single audio which is a single audio section existing in the immediately succeeding section, as backward-connected separated audio; and the audio connection unit connects the forward-connected separated audio to the forward single audio and connects the backward-connected separated audio to the backward single audio.

6. An audio processing device according to claim 5, wherein the audio connection unit connects the forward connected separated audio to the forward single audio and connects the backward connected separated audio to the backward single audio using different audio output channels.

7. The audio processing device of claim 5, wherein, when there is a separated audio among the plurality of separated audios that does not correspond to either the forward-connected separated audio or the backward-connected separated audio, the audio connection unit does not connect the separated audio to any single audio of any time range, but sets the separated audio to an audio output channel different from the audio output channel to which the forward-connected separated audio is connected to the forward single audio and the audio output channel to which the backward-connected separated audio is connected to the backward single audio.

8. An audio processing device according to claim 5, wherein the audio connection unit connects the forward connected separated audio to the forward single audio and connects the backward connected separated audio to the backward single audio using the same audio output channel.

9. The audio processing device described in claim 8, wherein the audio connection unit sets a single audio obtained by connecting the backward connected separated audio, the immediately following single audio, and the backward single audio behind the single audio obtained by connecting the forward single audio, the immediately preceding single audio, and the forward connected separated audio in the same audio output channel.

10. The audio processing device described in claim 8 or 9, wherein, when there is a separated audio among the multiple separated audios that does not correspond to either the forward-connected separated audio or the backward-connected separated audio, the audio connection unit does not connect the separated audio to a single audio of any time range, but sets the separated audio to an audio output channel where the forward-connected separated audio is connected to the forward single audio and the backward-connected separated audio is connected to the backward single audio.

11. The section detection unit acquires, as the audio data, time-series feature data in which features of a single audio are indicated in a single audio section, which is a time range in which a single audio exists, and features of the mixed audio are indicated in the mixed audio section, and analyzes the features indicated in the feature data to detect the mixed audio section from the feature data; the audio separation unit separates the features of the mixed audio in the mixed audio section into features of the multiple audio sections; the audio processing device further has an audio restoration unit that restores multiple separated audio sections that are the multiple audio sections from the features of the multiple audio sections obtained by separation by the separation unit; and the audio connection unit connects at least one separated audio section of the multiple separated audio sections obtained by restoration by the audio restoration unit to one single audio section.

12. The audio processing device according to claim 11, wherein the audio restoration unit restores the plurality of separated audio segments using the likelihood of the mixed audio segments.

13. The audio processing device according to any one of claims 1 to 12, further comprising: a first division unit that divides the audio data into sizes suitable for detection of the mixed audio section by the section detection unit; and a second division unit that divides the audio data into sizes suitable for audio division by the audio separation unit, wherein the section detection unit detects the mixed audio section from the audio data after division by the first division unit, and the audio separation unit separates the mixed audio of the mixed audio section into the multiple audio sections using the audio data after division by the second division unit.

14. The audio processing device according to any one of claims 1 to 13, further comprising: a first feature extraction unit that generates first feature data indicating features suitable for detection of the mixed audio section by the section detection unit; and a second feature extraction unit that generates second feature data indicating features suitable for audio division by the audio separation unit, wherein the section detection unit obtains the first feature data as the audio data and detects the mixed audio section from the first feature data, and the audio separation unit separates the mixed audio of the mixed audio section into the multiple audio sections using the second feature data.

15. The audio processing device according to any one of claims 1 to 14, further comprising a speech enhancement unit that performs speech enhancement to separate noise from speech, and the section detection unit determines the time range for which the speech enhancement unit performs speech enhancement depending on the noise level in the audio data.

16. The speech processing device according to claim 12, further comprising a model generation unit that generates a likelihood derivation model that is a model for deriving the likelihood of the mixed speech section, and the speech restoration unit restores the multiple separated speeches using the likelihood of the mixed speech section derived using the likelihood derivation model.

17. A sound processing method in which a computer detects a mixed sound section, which is a time range in which a mixed sound containing multiple sounds is present, from time-series sound data, and the computer separates the mixed sound in the mixed sound section into the multiple sounds.

18. A sound processing program that causes a computer to execute a section detection process that detects a mixed sound section, which is a time range in which a mixed sound containing multiple sounds exists, from time-series sound data, and a sound separation process that separates the mixed sound in the mixed sound section into the multiple sounds.

Images