Audio processing device, audio processing method, and recording medium

Abstract

This audio processing device includes: a separation means for outputting a second audio signal obtained by separating a first audio signal; and a determination means for outputting the results of determining whether the separation of the first audio signal was valid, on the basis of the similarity between the first audio signal and the second audio signal.

Description

Audio processing device, audio processing method, and recording medium

[0001] The present disclosure relates to the technical fields of an audio processing device, an audio processing method, and a recording medium.

[0002] As a technology for audio separation, for example, a technology has been proposed in which sound source separation is recursively performed on an input audio signal using a predetermined sound source separation model that has been trained in advance to separate a predetermined sound source from a training audio signal that includes the predetermined sound source (see Patent Document 1).

[0003] International Application No. 2020-195924

[0004] An object of this disclosure is to provide a speech processing device, a speech processing method, and a recording medium that aim to improve upon the techniques related to the prior art documents mentioned above.

[0005] One aspect of an audio processing device includes a separation means for outputting a second audio signal obtained by separating a first audio signal, and a judgment means for outputting a judgment result as to whether the separation of the first audio signal is appropriate based on the similarity between the first audio signal and the second audio signal.

[0006] One aspect of the audio processing method is an audio processing method executed by a computer, which includes outputting a second audio signal separated from a first audio signal, and outputting a determination result as to whether the separation of the first audio signal is appropriate based on the similarity between the first audio signal and the second audio signal.

[0007] One aspect of the recording medium has recorded thereon a computer program for causing a computer to execute an audio processing method that includes outputting a second audio signal separated from a first audio signal, and outputting a judgment result as to whether the separation of the first audio signal is appropriate based on the similarity between the first audio signal and the second audio signal.

[0008] FIG. 1 is a block diagram showing an example of a configuration of a voice processing device according to an embodiment. FIG. 2 is a flowchart showing an example of an operation of a voice processing device according to an embodiment. FIG. 3 is a block diagram showing an example of a configuration of a voice processing device according to an embodiment. FIG. 4 is a block diagram showing an example of an operation of a voice processing device according to an embodiment. FIG. 5 is a block diagram showing an example of an operation of a voice processing device according to an embodiment. FIG. 6 is a block diagram showing an example of an operation of a voice processing device according to an embodiment. FIG. 7 is a block diagram showing an example of a configuration of a voice processing device according to an embodiment. FIG. 8 is a block diagram showing an example of an operation of a voice processing device according to an embodiment.

[0009] Hereinafter, embodiments of a voice processing device, a voice processing method, and a recording medium will be described with reference to the drawings. [1: First Embodiment]

[0010] A first embodiment of an audio processing device, an audio processing method, and a recording medium will be described with reference to Fig. 1 and Fig. 2. In the following, the first embodiment of an audio processing device, an audio processing method, and a recording medium will be described using an audio processing device 10.

[0011] 1, the audio processing device 10 includes a separation unit 11 and a determination unit 12. The operation of the audio processing device 10 will be described with reference to the flowchart of FIG.

[0012] 2, an audio signal is input to the separation unit 11 (step S11). The audio signal input to the separation unit 11 is referred to as a first audio signal. The separation unit 11 outputs an audio signal obtained by separating the first audio signal (step S12). The audio signal output from the separation unit 11 is referred to as a second audio signal.

[0013] The first audio signal is an audio signal before separation in the corresponding separation process. The first audio signal may be referred to as a "separation target signal." The separation target signal may be an audio signal that has not yet been determined to be inappropriate for separation. The second audio signal is an audio signal after separation in the corresponding separation process. The second audio signal may be referred to as a "post-separation signal."

[0014] If the corresponding separation process is the initial separation process, the first audio signal is an audio signal that has not been subjected to separation process by the separation unit 11. If the corresponding separation process is not the initial separation process, the first audio signal is an audio signal that has been subjected to separation process by the separation unit 11.

[0015] The determination unit 12 determines whether the corresponding separation is appropriate based on the similarity between the pre-separation signal and the separated signal (step S13). For example, the determination unit 12 may determine that the corresponding separation is inappropriate if the similarity between the pre-separation signal and the separated signal is equal to or greater than a predetermined value. For example, assume that the separation unit 11 has separated a first audio signal into two types of second audio signals. In this case, the determination unit 12 may determine that the corresponding separation is inappropriate if the similarity between the first audio signal and at least one of the two types of second audio signals is equal to or greater than a predetermined value. The determination unit 12 may determine that the corresponding separation is inappropriate if the similarity between the first audio signal and one of the two types of second audio signals is equal to or greater than a predetermined value. The determination unit 12 may determine that the corresponding separation is inappropriate if the similarity between the first audio signal and the other of the two types of second audio signals is equal to or greater than a predetermined value. The judgment unit 12 may judge that the corresponding separation is inappropriate if the similarity between the first audio signal and one of the two types of second audio signals, and the similarity between the first audio signal and the other of the two types of second audio signals, are each equal to or greater than a predetermined value.

[0016] The determination unit 12 outputs information indicating that the corresponding separation is invalid or information indicating that the corresponding separation is valid, depending on the determination result (step S14). That is, the determination unit 12 outputs a determination result as to whether the separation of the first audio signal is valid, based on the similarity between the first audio signal and the second audio signal.

[0017] In this way, the audio processing device 10 performs an audio processing method that includes outputting a second audio signal obtained by separating a first audio signal, and outputting a determination result as to whether the separation of the first audio signal is appropriate based on the similarity between the first audio signal and the second audio signal.

[0018] The audio processing device 10 described above may be realized by a computer reading a computer program recorded on a recording medium. In this case, the computer program may cause the computer to execute an audio processing method including outputting a second audio signal obtained by separating a first audio signal, and outputting a determination result as to whether the separation of the first audio signal is appropriate based on the similarity between the first audio signal and the second audio signal. [Technical Effect]

[0019] The audio processing device 10 according to the present disclosure determines whether the separation is appropriate based on the degree of similarity between the audio signal before separation and the audio signal after separation, and therefore can separate audio with high accuracy. [2: Second Embodiment]

[0020] A second embodiment of the audio processing device, audio processing method, and recording medium will be described with reference to Figures 3 to 6. Hereinafter, the second embodiment of the audio processing device, audio processing method, and recording medium will be described using the audio processing device 20. Note that, for the second embodiment, descriptions that overlap with the description of the first embodiment will be omitted as appropriate. [2-1: Configuration of the audio processing device 20]

[0021] The configuration of the audio processing device 20 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the audio processing device 20.

[0022] 3 , the voice processing device 20 includes a calculation device 21, a storage device 22, and a communication device 23. The voice processing device 20 may further include an input device 24 and an output device 25. However, the voice processing device 20 does not necessarily include at least one of the input device 24 and the output device 25. The calculation device 21, the storage device 22, the communication device 23, the input device 24, and the output device 25 may be connected via a data bus 26.

[0023] The arithmetic device 21 includes at least one processor (i.e., one processor or multiple processors) as hardware. The processor may include, for example, a processor conforming to a von Neumann computer architecture. The processor conforming to the von Neumann computer architecture may include at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor may include, for example, a processor conforming to a non-von Neumann computer architecture. The processor conforming to the non-von Neumann computer architecture may include at least one of an FPGA (Field Programmable Gate Array) and an ASIC (Application Specific Circuit).

[0024] The arithmetic device 21 reads a computer program 221 including at least one of computer program code and computer program instructions. For example, the arithmetic device 21 may read the computer program 221 stored in the storage device 22. For example, the arithmetic device 21 may read the computer program 221 stored in a computer-readable, non-transitory recording medium using a recording medium reading device (not shown) included in the audio processing device 20. The computer program 221 read from the recording medium may be stored in the storage device 22. The arithmetic device 21 may acquire (i.e., download or read) the computer program 221 from a device (not shown) located outside the audio processing device 20 via the communication device 23 (or another communication device). The downloaded computer program 221 may be stored in the storage device 22.

[0025] The arithmetic device 21 executes the loaded computer program 221. As a result, logical functional blocks for executing the audio processing that the audio processing device 20 should perform are realized within the arithmetic device 21. In other words, the arithmetic device 21, together with the storage device 22 or the like in which the computer program 221 is recorded (in other words, together with the storage device 22 and the computer program 221 recorded in the storage device 22 or the like), can function as a controller or computer for realizing the logical functional blocks for executing the processing that the audio processing device 20 should perform. In other words, the at least one processor included in the arithmetic device 21, the memory (recording medium) included in the storage device 22 or the like, and the computer program 221 are configured so that the audio processing device 20 performs the audio processing that the audio processing device 20 should perform.

[0026] A computational model that can be constructed by machine learning may be implemented in the computational device 21 by the computational device executing the computer program 221. An example of a computational model that can be constructed by machine learning is a computational model including a neural network (so-called artificial intelligence (AI)). In this case, learning of the computational model may include learning of parameters of the neural network (for example, at least one of a weight and a bias). The computational device 21 may perform voice processing using the computational model. In other words, the operation of performing voice processing may include the operation of performing voice processing using the computational model. Note that a computational model that has been constructed by offline machine learning using training data may be implemented in the computational device 21. Furthermore, the computational model implemented in the computational device 21 may be updated by online machine learning on the computational device 21. Alternatively, the calculation device 21 may perform audio processing using a calculation model implemented in a device external to the calculation device 21 (i.e., a device provided outside the audio processing device 20) in addition to or instead of the calculation model implemented in the calculation device 21.

[0027] The recording medium for recording the computer program 221 executed by the arithmetic device 21 may be at least one of a CD-ROM, CD-R, CD-RW, flexible disk, MO, DVD-ROM, DVD-RAM, DVD-R, DVD+R, DVD-RW, DVD+RW, Blu-ray (registered trademark), or other optical disk, a magnetic medium such as a magnetic tape, a magneto-optical disk, a semiconductor memory such as a USB memory, or any other medium capable of storing a program. The recording medium may include a device capable of recording a computer program (for example, a general-purpose device or a dedicated device in which the computer program 221 is implemented in a state in which it can be executed in at least one of the forms of software and firmware). Furthermore, each process or function included in the computer program 221 may be realized by a logical processing block realized within the arithmetic device 21 when the arithmetic device 21 (i.e., processor) executes the computer program 221, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) provided in the arithmetic device 21, or may be realized in a form that mixes logical processing blocks and partial hardware modules that realize some elements of the hardware.

[0028] The storage device 22 includes at least one memory capable of storing desired data. In other words, the storage device 22 includes at least one memory containing desired data. For example, the storage device 22 may store a computer program 221 executed by the arithmetic device 21. In this case, the storage device 22 (memory) may be used as the above-mentioned recording medium for recording the computer program 221 executed by the arithmetic device 21. The storage device 22 may temporarily store data used by the arithmetic device 21 when the arithmetic device 21 is executing the computer program 221. The storage device 22 may also store data to be stored long-term by the audio processing device 20. The storage device 22 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 22 may include a non-transitory recording medium.

[0029] The communication device 23 may be capable of communicating with devices external to the audio processing device 20. The communication device 23 may perform wired communication or wireless communication.

[0030] The input device 24 is a device capable of accepting information input to the voice processing device 20 from the outside. The input device 24 may include an operation device (e.g., a keyboard, a mouse, a touch panel, etc.) that can be operated by a user of the voice processing device 20. The input device 24 may include a recording medium reading device capable of reading information recorded on a recording medium that is detachable from the voice processing device 20, such as a USB (Universal Serial Bus) memory. Note that when information is input to the voice processing device 20 via the communication device 23 (in other words, when the voice processing device 20 acquires information via the communication device 23), the communication device 23 may function as an input device.

[0031] The output device 25 is a device capable of outputting information to the outside of the voice processing device 20. The output device 25 may output visual information such as text or images, auditory information such as sound, or tactile information such as vibration, as the information. The output device 25 may include, for example, at least one of a display, a speaker, a printer, and a vibration motor. The output device 25 may be capable of outputting information to a recording medium detachable from the voice processing device 20, such as a USB memory. Note that when the voice processing device 20 outputs information via the communication device 23, the communication device 23 may function as the output device.

[0032] FIG. 3 shows an example of logical functional blocks realized in the arithmetic device 21 to perform audio processing. As shown in FIG. 3, a separation unit 211, a determination unit 212, and a control unit 213 are realized in the arithmetic device 21. The "separation unit 211" is a component corresponding to the "separation unit 11" in the first embodiment described above, and the "determination unit 212" is a component corresponding to the "determination unit 12" in the first embodiment described above. When an audio signal contains audio signals from multiple sound sources, the audio processing device 20 may be configured to separate the audio signal for each sound source. [2-2: Audio processing method executed by audio processing device 20]

[0033] The audio processing method executed by the audio processing device 20 will be described with reference to Fig. 4 to Fig. 6. Fig. 4 is a block diagram showing an example of the flow of information in the audio processing method executed by the audio processing device 20. Fig. 5 is a diagram showing the correspondence between signals to be separated and similarities. Fig. 6 is a flowchart showing an example of the processing flow in the audio processing method executed by the audio processing device 20.

[0034] This embodiment separates an audio signal containing speech from one or more audio sources into audio signals for each audio source. For example, this embodiment may separate an audio signal containing speech from one or more speakers into audio signals for each speaker. An audio signal that has not yet been subjected to separation processing is referred to as a speech signal. The number of speakers included in the speech signal may be unknown.

[0035] 4 , first, the control unit 213 acquires a conversation signal. That is, the control unit 213 acquires an audio signal that has not yet been subjected to separation processing. The control unit 213 inputs the conversation signal to the separation unit 211 as a signal to be separated.

[0036] The separation unit 211 receives a separation target signal as an input and outputs a separated signal as a second audio signal obtained by separating the separation target signal. The separation unit 211 may separate the separation target signal into a plurality of separated signals. In this embodiment, a case will be described in which the separation unit 211 separates the separation target signal into two separated signals. For example, the separation unit 211 may separate the separation target signal into two separated signals using a speech separation model trained to separate the separation target signal into two separated signals. The speech separation model may be a computational model trained to separate speech into two depending on the speaker who is the sound source of the speech. Note that in this embodiment, a case will be described in which the speech separation model separates the separation target signal into two separated signals. However, the separation unit 211 may separate the separation target signal using a speech separation model that separates the separation target signal into three or more separated signals. Below, a case will be described in which the separation unit 211 separates the separation target signal S* into separated signals S*1 and separated signals S*2.

[0037] The determination unit 212 calculates the similarity. For example, the determination unit 212 may determine whether the similarity is "high," "medium," or "small." For example, the range of values ​​that the similarity can take may be divided into thirds, and the similarity in the first largest value range may be defined as "high," the similarity in the second largest value range as "medium," and the similarity in the third largest value range as "small." Alternatively, a similarity greater than a first value that is smaller than the maximum possible value of the similarity may be defined as "high." Alternatively, a similarity greater than a second value that is greater than the minimum possible value of the similarity and smaller than the first value may be defined as "small." Alternatively, a similarity smaller than the first value and greater than the second value may be defined as "medium." For simplicity of explanation, the following description will be given taking as an example a case where the determination unit 212 determines whether the similarity is "high," "medium," or "small."

[0038] 5, when the separation target signal S* contains speech from two or more speakers, the similarity between the separation target signal S* and the separated signal S*1 and the similarity between the separation target signal S* and the separated signal S*2 are both medium. In contrast, when the separation target signal S* contains speech from a single speaker, at least one of the similarity between the separation target signal S* and the separated signal S*1 and the similarity between the separation target signal S* and the separated signal S*2 is large.

[0039] When the separation target signal S* contains the voice of one speaker, separation may be performed such that one of the separated signals (e.g., separated signal S*1) contains the entire separation target signal S*, and the other separated signal (e.g., separated signal S*2) does not contain any voice signal. This may occur, for example, when the voice of one speaker is continuously present in a conversation signal section. In this case, the similarity between one of the separated signals (e.g., separated signal S*1) and the separation target signal S* is large, and the similarity between the other separated signal (e.g., separated signal S*2) and the separation target signal S* is zero. Note that even if the other separated signal contains only a small amount of the separation target signal S*, one of the separated signals may also be considered to contain the entire separation target signal S*.

[0040] Furthermore, when the separation target signal S* contains the voice of one speaker, separation may be performed such that one of the separated signals (e.g., separated signal S*1) contains the first half of the separation target signal, and the other separated signal (e.g., separated signal S*2) contains the second half of the separation target signal. This may occur when the voice of one speaker does not exist continuously in the section of the speaker signal, but exists, for example, in multiple discrete continuous sections. Note that the first half of the separation target signal may refer to, for example, the front part when the separation target signal is divided into two, and the second half of the separation target signal may refer to the back part when the separation target signal is divided into two.

[0041] The determination unit 212 determines whether separation of the separation target signals is appropriate based on the similarity between the separation target signals and the separated signals. The similarity between the separation target signals and the separated signals may be, for example, the similarity between the speaker characteristics of the audio signal to be separated and the speaker characteristics of the separated audio signal. Furthermore, the similarity used for speaker matching may be used as the similarity between the separation target signals and the separated signals. That is, the similarity calculated when speaker matching is performed between the separation target signals and the separated signals may be used as the similarity between the separation target signals and the separated signals. The determination unit 212 may determine that separation of the separation target signals is appropriate if both similarities are "medium." The determination unit 212 may determine that separation of the separation target signals is inappropriate if at least one similarity is "large." The determination unit 212 may determine that separation of the separation target signals is inappropriate if one similarity is "small." If all the similarities are "high," the determining unit 212 may determine that the separation of the separation target signals is inappropriate.

[0042] If the separation of the separation target signal is appropriate, the control unit 213 sets each of the multiple separated signals as the separation target signal. If the separation of the separation target signal is inappropriate, the control unit 213 sets the separation target signal as the output signal. The output signal is an audio signal for which separation is inappropriate, and is an audio signal that is not a separation target. The control unit 213 inputs any one of the multiple separated signals that have not yet been separated to the separation unit 211.

[0043] 4, the operations of the control unit 213, the separation unit 211, and the determination unit 212 are repeatedly performed. The control unit 213 may determine whether to continue or end the repeated operations.

[0044] Next, with reference to the flowchart shown in FIG. 6 , a case where conversation signal S1 indicates a conversation between three speakers will be described. That is, a case where conversation signal S1 includes three types of audio signals from different sound sources will be described. Conversation signal S1 does not have information indicating that it includes three types of audio signals from different sound sources. Therefore, separation unit 211, which performs the separation operation, determination unit 212, which performs the determination operation, and control unit 213, which controls the entire operation, do not know how many types of audio signals conversation signal S1 includes until control unit 213 makes a negative determination in step S22.

[0045] 6, the control unit 213 acquires the conversation signal S1 (step S21).

[0046] The control unit 213 determines whether there is a signal to be separated (step S22). Since the acquired conversation signal S1 is a signal to be separated (step S22: Yes), the control unit 213 inputs the conversation signal S1 to the separation unit 211 (step S23). The separation unit 211, to which the conversation signal S1 is input, outputs a separated signal S11 and a separated signal S12 (S24). The separation unit 211 may separate the conversation signal S1 containing three types of audio signals into a separated signal S11 containing two types of audio signals and a separated signal S12 containing one type of audio signal.

[0047] The determination unit 212 determines whether separation of the conversation signal S1 is appropriate based on the similarity between the conversation signal S1 and the separated signal S11 and the similarity between the conversation signal S1 and the separated signal S12 (step S25). Since the conversation signal S1 includes multiple audio signals and separation is appropriate, the similarity between the conversation signal S1 and the separated signal S11 and the similarity between the conversation signal S1 and the separated signal S12 are both "medium". The determination unit 212 outputs a determination result indicating that separation of the conversation signal S1 is appropriate to the control unit 213 based on the "medium" similarity (step S26). The control unit 213 determines that the separated signal S11 is the separation target signal S11 and the separated signal S12 is the separation target signal S12. The control unit 213 determines to continue the repeating operation. [Second Repeating Operation]

[0048] The control unit 213 determines whether there is a separation target signal (step S22). In the first repeat operation, the separated signal S11 becomes the separation target signal S11, and the separated signal S12 becomes the separation target signal S12 (step S22: Yes). The control unit 213 inputs one of the separation target signal S11 and the separation target signal S12 to the separation unit 211 (step S23). For example, in the second repeat operation, the control unit 213 may input the separation target signal S11 to the separation unit 211, and in the third repeat operation, the control unit 213 may input the separation target signal S12 to the separation unit 211. Below, a case will be described in which the control unit 213 inputs the separation target signal S11 to the separation unit 211 in the second repeat operation, and the control unit 213 inputs the separation target signal S12 to the separation unit 211 in the third repeat operation. Next, a case will be described in which the separation unit 211 separates the conversation signal S1 into a separated signal S11 including two types of audio signals and a separated signal S12 including one type of audio signal, as described above. The separation unit 211 receives the separation target signal S11 and outputs a separated signal S111 and a separated signal S112 (S24). Each of the separated signal S111 and the separated signal S112 may include one type of audio signal.

[0049] The determination unit 212 determines whether separation of the separation target signal S11 is appropriate based on the similarity between the separation target signal S11 and the separated signal S111 and the similarity between the separation target signal S11 and the separated signal S112 (step S25). Since the separation target signal S11 includes multiple audio signals and is appropriate for separation, the similarity between the separation target signal S11 and the separated signal S111 and the similarity between the separation target signal S11 and the separated signal S112 are both set to "medium." Based on the "medium" similarity, the determination unit 212 outputs a determination result indicating that separation of the separation target signal S11 is appropriate to the control unit 213 (step S26). The control unit 213 designates the separated signal S111 as the separation target signal S111 and the separated signal S112 as the separation target signal S112. The control unit 213 determines to continue the repeating operation. [Third Repeating Operation]

[0050] The control unit 213 determines whether there is a separation target signal (step S22). The separation target signal S12 has not yet been separated, and in the second repeat operation, the separated signal S111 becomes the separation target signal S111, and the separated signal S112 becomes the separation target signal S112 (step S22: Yes). As described above, the control unit 213 inputs the separation target signal S12 to the separation unit 211 (step S23). The separation unit 211, to which the separation target signal S12 has been input, outputs the separated signals S121 and S122 (S24).

[0051] The determination unit 212 determines whether separation of the separation target signal S12 is appropriate based on the similarity between the separation target signal S12 and the separated signal S121 and the similarity between the separation target signal S12 and the separated signal S122 (step S25). Since the separation target signal S12 contains one type of audio signal and separation is inappropriate, at least one of the similarity between the separation target signal S12 and the separated signal S121 and the similarity between the separation target signal S12 and the separated signal S122 is "large." Based on the similarity being "large," the determination unit 212 outputs a determination result indicating that separation of the separation target signal S12 is inappropriate to the control unit 213 (step S26). Note that the determination unit 212 may also determine that separation of the separation target signal S12 is inappropriate based on the similarity between the separation target signal S12 and the separated signal S121 and the similarity between the separation target signal S12 and the separated signal S122 being "small." The control unit 213 outputs the separation target signal S12. The control unit 213 determines to continue the repetitive operation. [Fourth repetitive operation]

[0052] The control unit 213 determines whether there is a separation target signal (step S22). In the second iteration, the separated signal S111 becomes the separation target signal S111, and the separated signal S112 becomes the separation target signal S112 (step S22: Yes). The control unit 213 inputs one of the separation target signal S111 and the separation target signal S112 to the separation unit 211 (step S23). For example, in the fourth iteration, the control unit 213 may input the separation target signal S111 to the separation unit 211, and in the fifth iteration, the control unit 213 may input the separation target signal S112 to the separation unit 211. Below, a case will be described in which the control unit 213 inputs the separation target signal S111 to the separation unit 211 in the fourth iteration, and inputs the separation target signal S112 to the separation unit 211 in the fifth iteration. The separation unit 211 receives the separation target signal S111 and outputs a separated signal S1111 and a separated signal S1112 (S24).

[0053] The determination unit 212 determines whether separation of the separation target signal S111 is appropriate based on the similarity between the separation target signal S111 and the separated signal S1111 and the similarity between the separation target signal S111 and the separated signal S1112 (step S25). Since the separation target signal S111 includes one type of audio signal and separation is inappropriate, at least one of the similarity between the separation target signal S111 and the separated signal S1111 and the similarity between the separation target signal S111 and the separated signal S1112 is "large." Based on the similarity being "large," the determination unit 212 outputs a determination result indicating that separation of the separation target signal S111 is inappropriate to the control unit 213 (step S26). The determination unit 212 may determine that separation of the separation target signal S111 is inappropriate based on the fact that either the similarity between the separation target signal S111 and the separated signal S1111 or the similarity between the separation target signal S111 and the separated signal S1112 is "small." The control unit 213 outputs the separation target signal S111. The control unit 213 determines to continue the repeating operation. [Fifth repeating operation]

[0054] The control unit 213 determines whether there is a signal to be separated (step S22). Since the signal to be separated S112 has not yet been separated (step S22: Yes), the control unit 213 inputs the signal to be separated S112 to the separation unit 211 (step S23). The separation unit 211, to which the signal to be separated S112 has been input, outputs separated signals S1121 and S1122 (S24).

[0055] The determination unit 212 determines whether separation of the separation target signal S112 is appropriate based on the similarity between the separation target signal S112 and the separated signal S1121 and the similarity between the separation target signal S112 and the separated signal S1122 (step S25). Since the separation target signal S112 includes one type of audio signal and separation is inappropriate, at least one of the similarity between the separation target signal S112 and the separated signal S1121 and the similarity between the separation target signal S112 and the separated signal S1122 is "large." Based on the similarity being "large," the determination unit 212 outputs a determination result indicating that separation of the separation target signal S112 is inappropriate to the control unit 213 (step S26). The determination unit 212 may determine that separation of the separation target signal S112 is inappropriate based on the fact that either the similarity between the separation target signal S112 and the separated signal S1121 or the similarity between the separation target signal S112 and the separated signal S1122 is "small." The control unit 213 outputs the separation target signal S112. The control unit 213 determines to continue the repetitive operation. [Sixth repetitive operation]

[0056] The control unit 213 determines whether there is a signal to be separated (step S22). Since there is no signal to be separated (step S22: No), the control unit 213 outputs output signals S12, S111, and S112 (step S28). The control unit 213 determines that the repeating operation is to end. [2-3: Technical Effects]

[0057] There is a technology that recognizes a speech signal and outputs text corresponding to the speech signal. The speech signal to be recognized may contain speech emitted from multiple sound sources. For example, the speech signal may be a conversation signal containing utterances from multiple speakers. In this case, if it is not possible to distinguish which speaker is uttering each of the utterances included in the conversation signal, the text corresponding to the conversation signal is difficult for humans to read. Furthermore, text corresponding to a conversation signal in which the speaker is not distinguished is difficult for artificial intelligence and the like to handle. If the speech signal is separated by sound source, the text corresponding to the speech signal becomes easier to handle for both humans and artificial intelligence and the like.

[0058] For example, a speech separation mechanism is known that receives a one-channel conversation signal as input and outputs N-channel speech signals. When one speech signal is input, this speech separation mechanism outputs speech signals for N sound sources. Since this speech separation mechanism has a fixed number of output channels N, it is necessary to estimate the number of sound sources and specify the number of channels before performing sound source separation. Therefore, this speech separation mechanism is not useful when the number of sound sources is unknown.

[0059] In contrast, Patent Document 1 discloses a technique (comparative example) for separating sounds without specifying the number of sound sources. The comparative example determines whether the signal separation is appropriate by comparing the separated signals with each other.

[0060] Example E1 in FIG. 7 illustrates a case where the separation target signal S1 includes audio signals emitted from multiple sound sources. For example, example E1 in FIG. 7 illustrates a case where an audio signal of a conversation including the audio of speaker 1 and the audio of speaker 2 is separated as the separation target signal S1. In this case, the separation target signal S1 is separated into a separated signal S11 mainly including the audio of speaker 1 and a separated signal S12 mainly including the audio of speaker 2. Because speakers 1 and 2 are different people, the separated signal S11 mainly including the audio of speaker 1 and the separated signal S12 mainly including the audio of speaker 2 are not similar. Therefore, the similarity between the separated signal S11 mainly including the audio of speaker 1 and the separated signal S12 mainly including the audio of speaker 2 is "low." In this way, it is expected that it can be determined that the speaker corresponding to the separated signal S11 and the speaker corresponding to the separated signal S12 are different people based on the similarity of the separated signals being "low."

[0061] In contrast, example E2 of Fig. 7 illustrates a case where the separation target signal S1 includes a speech signal emitted from one sound source. Example E2 of Fig. 7 illustrates a case where a speech signal including the speech of one speaker is separated as the separation target signal S1. In this case, the separation target signal S1 may be separated into a separated signal S11 including the first half of the speaker's speech and a separated signal S12 including the second half of the speaker's speech. Since the separated signal S11 and the separated signal S12 are speech signals emitted from the same sound source, they are expected to be similar.

[0062] However, it is known that speaker verification technology has a tendency to lose accuracy when the speech signals to be verified are short, resulting in low similarity even for speech signals from the same person. The separated signals have short signal durations. Therefore, the similarity between a separated signal S11, which is mainly composed of the speech of speaker 1, and a separated signal S12, which is mainly composed of the speech of speaker 2, may be "low." Thus, when the similarity of the separated signals is used, it may not be possible to determine whether the speaker corresponding to the separated signal S11 and the speaker corresponding to the separated signal S12 are the same person or different people. In other words, in the comparative example, since only the separated signals are referenced, it may not be possible to properly determine whether the separated signals are signals from different speakers or the same speaker.

[0063] Therefore, the speech processing device 20 according to this disclosure compares the signal before separation with the signal after separation to determine the similarity. The speech processing device 20 determines the validity of separation of the separation target signal S1 based on the similarity between the separation target signal S1 and the separated signal S11, and the similarity between the separation target signal S1 and the separated signal S12. The separation target signal S1 is longer than the separated signals S11 and S12. Therefore, the similarity between the separation target signal S1 and the separated signals, which are voices of the same speaker, does not decrease.

[0064] Example E3 in Fig. 7 illustrates a case where the separation target signal S1 includes a sound signal emitted from one sound source. Example E3 in Fig. 7 illustrates a case where a sound signal including the sound of one speaker is separated as the separation target signal S1. In this case, the separation target signal S1 may be separated into a separated signal S11 that includes all of the speaker's sound and a separated signal S12 that does not include the speaker's sound. In this case, too, the similarity between the separation target signal S1 and the separated signal S11 is "high," and the similarity between the separation target signal S1 and the separated signal S12 is "low," allowing the sound processing device 20 to determine that the separation of the separation target signal S1 is inappropriate.

[0065] In this way, even if the separated signal is short, the separation target signal is long, so the audio processing device 20 can accurately determine the validity of separation of the separation target signal S1. Furthermore, the audio processing device 20 can separate the audio signal into an appropriate number of audio sources without specifying the number of audio sources. [3: Third Embodiment]

[0066] A third embodiment of the audio processing device, audio processing method, and recording medium will be described with reference to Figures 8 and 9. Below, the third embodiment of the audio processing device, audio processing method, and recording medium will be described using an audio processing device 30. Note that, for the second embodiment, descriptions that overlap with the description of the first embodiment will be omitted as appropriate. Note that, in the drawings, parts common to the first and second embodiments are designated by the same reference numerals.

[0067] As shown in FIG. 8 , the arithmetic unit 21 included in the audio processing device 30 includes, as logical functional blocks, a separation unit 211, a determination unit 312, and a control unit 213. The determination unit 312 may include an evaluation unit 3121. The determination unit 312 determines whether separation of the separation target signal is appropriate based on the similarity between the separation target signal and the separated signal, as well as at least one of the similarity between each of the plurality of separated signals, the section length of each of the plurality of separated signals, and the overlap of each of the sections of the plurality of separated signals. The evaluation unit 3121 calculates an evaluation value based on the similarity between the separation target signal and the separated signal, as well as at least one of the similarity between each of the plurality of separated signals, the section length of each of the plurality of separated signals, and the overlap of each of the sections of the plurality of separated signals. The evaluation unit 3121 may determine which of four levels, "large," "medium," "small," or "very small," each index is. The evaluation unit 3121 may calculate the evaluation value based on which of the four levels each index is.

[0068] The evaluation unit 3121 may calculate the evaluation value so that the more inappropriate the separation of the separation target signals is, the larger the evaluation value becomes. In this case, the determination unit 312 may determine that the separation of the separation target signals is inappropriate when the evaluation value is equal to or greater than a predetermined value.

[0069] Alternatively, the evaluation unit 3121 may calculate the evaluation value such that the more appropriate the separation of the separation target signals is, the larger the evaluation value becomes. In this case, the determination unit 312 may determine that the separation of the separation target signals is appropriate when the evaluation value is equal to or greater than a predetermined value.

[0070] The audio processing method executed by the audio processing device 30 will be described with reference to Fig. 9. Fig. 9 is a diagram showing correspondence between the separation target, the similarity between the separation target signal and the separated signal, the similarity between each of the multiple separated signals, the section length of each of the multiple separated signals, and the overlapping separation target signal between each of the multiple separated signals. Below, an example will be described in which the evaluation unit 3121 calculates the evaluation value so that the evaluation value increases as the separation of the separation target signal becomes less appropriate.

[0071] If the separation of the separation target signal is inappropriate, at least one of the degrees of similarity between the separation target signal and the separated signal will be “high.” Therefore, if at least one of the degrees of similarity between the separation target signal and the separated signal is “high,” the evaluation unit 3121 may calculate the evaluation value so that the evaluation value is large.

[0072] Furthermore, if the separation of the separation target signal is inappropriate, both the similarities between the separation target signal and the separated signal will be "high," or one of the similarities between the separation target signal and the separated signal will be "high" and the other will be "very low." Therefore, if both the similarities between the separation target signal and the separated signal are "high," or one of the similarities between the separation target signal and the separated signal is "high" and the other is "very low," the evaluation unit 3121 may calculate the evaluation value so that the evaluation value becomes large.

[0073] Furthermore, if the separation of the separation target signals is inappropriate, the similarity between each of the plurality of separated signals will be “minimum.” Therefore, if the similarity between each of the plurality of separated signals is “minimum,” the evaluation unit 3121 may calculate the evaluation value so that the evaluation value becomes large.

[0074] Furthermore, if the separation of the separation target signals is appropriate, both of the section lengths of the separated signals will be “large.” Therefore, if at least one of the section lengths of the separated signals is not “large,” the evaluation unit 3121 may calculate the evaluation value so that the evaluation value becomes large.

[0075] Furthermore, if the separation of the separation target signals is inappropriate, the overlap of the section lengths of the separated signals will be "minimum." Therefore, if the overlap of the section lengths of the separated signals is "minimum," the evaluation unit 3121 may calculate the evaluation value so that the evaluation value becomes large. [Technical effect]

[0076] The similarity between each of the plurality of separated signals, the section length between each of the plurality of separated signals, and the overlap between each of the sections between the plurality of separated signals are useful information for determining the validity of the separation of the signals to be separated.

[0077] The audio processing device 30 according to this disclosure determines the appropriateness of separation of the separation target signals using at least one of the similarity between the separation target signal and the separated signal, the similarity between each of the plurality of separated signals, the section length of each of the plurality of separated signals, and the overlap of each of the sections of the plurality of separated signals, and therefore can more accurately determine whether the separation of the separation target signals is appropriate. [4: Supplementary Note]

[0078] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes. [Supplementary Note 1] A sound processing device comprising: separation means for outputting a second sound signal obtained by separating a first sound signal; and determination means for outputting a determination result as to whether separation of the first sound signal is appropriate based on the similarity between the first sound signal and the second sound signal. [Supplementary Note 2] The sound processing device according to Supplementary Note 1, comprising control means for inputting the second sound signal as the first sound signal to the separation means when it is determined that separation by the separation means is appropriate, and for outputting the first sound signal when it is determined that separation by the separation means is inappropriate. [Supplementary Note 3] The sound processing device according to Supplementary Note 1, wherein the determination means determines that separation of the first sound signal is inappropriate when the similarity is equal to or greater than a predetermined value. [Supplementary Note 4] The sound processing device according to Supplementary Note 1, wherein the determination means has evaluation means for calculating an evaluation value based on the similarity, and determines the appropriateness of separation by the separation means based on the evaluation value. [Supplementary Note 5] The audio processing device according to Supplementary Note 4, wherein the evaluation means calculates the evaluation value based on a section length of the second audio signal in addition to the similarity. [Supplementary Note 6] The audio processing device according to Supplementary Note 4, wherein the evaluation means calculates the evaluation value based on an overlap of sections of the plurality of second audio signals in addition to the similarity. [Supplementary Note 7] The audio processing device according to Supplementary Note 4, wherein the evaluation means calculates the evaluation value based on a first similarity that is the similarity and a second similarity that is the similarity of each of the plurality of second audio signals. [Supplementary Note 8] An audio processing method executed by a computer, comprising: outputting a second audio signal obtained by separating a first audio signal; and outputting a determination result as to whether separation of the first audio signal is appropriate based on the similarity between the first audio signal and the second audio signal. [Supplementary Note 9] A recording medium having recorded thereon a computer program for causing a computer to execute an audio processing method, the method including: outputting a second audio signal obtained by separating a first audio signal; and outputting a determination result as to whether the separation of the first audio signal is appropriate, based on the similarity between the first audio signal and the second audio signal.

[0079] Furthermore, some or all of the configurations described in Supplementary Notes 2 to 7, which are dependent on Supplementary Notes 1, and Supplementary Notes 8 and 9, may be dependent in the same manner as Supplementary Notes 2 to 7. Furthermore, not limited to Supplementary Notes 1, 8, and 9, some or all of the configurations described as Supplements may be dependent on various hardware, software, various recording means for recording software, or systems, within the scope of the above-mentioned embodiments.

[0080] This disclosure may be modified as appropriate within the scope that does not contradict the gist or idea of ​​the invention that can be read from the claims and the entire specification, and voice processing devices, voice processing methods, and programs that involve such modifications are also included in the technical idea of ​​this disclosure.

[0081] 10, 20, 30 Audio processing device 11, 211 Separation unit 12, 212, 312 Determination unit 213 Control unit 3121 Evaluation unit

Claims

1. An audio processing device comprising: a separation means for outputting a second audio signal obtained by separating a first audio signal; and a determination means for outputting a determination result as to whether the separation of the first audio signal is appropriate based on the similarity between the first audio signal and the second audio signal.

2. An audio processing device according to claim 1, further comprising control means for inputting the second audio signal to the separation means as the first audio signal when it is determined that the separation by the separation means is appropriate, and for outputting the first audio signal when it is determined that the separation by the separation means is inappropriate.

3. The audio processing device according to claim 1, wherein the determining means determines that the separation of the first audio signal is inappropriate when the similarity is equal to or greater than a predetermined value.

4. The audio processing device according to claim 1, wherein the determining means has an evaluating means for calculating an evaluation value based on the similarity, and determines the validity of the separation by the separating means based on the evaluation value.

5. The audio processing device according to claim 4, wherein the evaluation means calculates the evaluation value based on the similarity as well as the length of the second audio signal.

6. The audio processing device according to claim 4, wherein the evaluation means calculates the evaluation value based on the overlap of the sections of the plurality of second audio signals in addition to the similarity.

7. The audio processing device according to claim 4, wherein the evaluation means calculates the evaluation value based on a second similarity, which is the similarity between each of the plurality of second audio signals, in addition to the first similarity, which is the similarity.

8. A computer-implemented audio processing method comprising: outputting a second audio signal obtained by separating a first audio signal; and outputting a determination result as to whether the separation of the first audio signal is appropriate based on the similarity between the first audio signal and the second audio signal.

9. A recording medium having recorded thereon a computer program for causing a computer to execute an audio processing method including: outputting a second audio signal obtained by separating a first audio signal; and outputting a determination result as to whether the separation of the first audio signal is appropriate based on the similarity between the first audio signal and the second audio signal.

Images