Signal processing device, signal processing method and program

Abstract

Provided is a signal processing device comprising: a sound source separation unit that applies sound source separation processing to a mixed sound signal in which signals from a plurality of sound sources are mixed; and a band expanding unit that applies frequency band expansion processing to each sound source separation signal separated by the sound source separation unit.

Description

[Technical field]

[0001] The present disclosure relates to a signal processing device, a signal processing method, and a program. [Background technology]

[0002] There is known a sound source separation technique that extracts a signal of a sound source of interest from a mixed sound signal that contains sounds from multiple sound sources (see, for example, Patent Document 1). In addition, a frequency band expansion (enlargement) technique has been proposed that generates a signal of a wider frequency band by generating a high-frequency component from a low-frequency component signal and adding the obtained high-frequency component to the low-frequency component signal (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 047643

[0004] [Patent Document 2] International Publication No. 2015 / 079946 Summary of the Invention [Problem to be solved by the invention]

[0005] In this field, it is desirable to carry out appropriate frequency band expansion processing and the like.

[0006] An object of the present disclosure is to provide a signal processing device, a signal processing method, and a program that perform appropriate frequency band expansion processing and the like. [Means for solving the problem]

[0007] The present disclosure relates to, for example, a sound source separation unit that applies sound source separation processing to a mixed sound signal in which signals of a plurality of sound sources are mixed; a band expansion unit that applies frequency band expansion processing to each of the sound source separation signals separated by the sound source separation unit; 、 an adder that adds up outputs of the band extension units provided for each sound source separation signal; a frequency envelope shaping unit that shapes the frequency envelope of the combined output signal output from the adder unit; having The frequency envelope shaping unit shapes the frequency envelope of the synthesis output signal when a predetermined discontinuity is detected around f1, where f1 is the lower limit of the frequency extended by the frequency band extension process. It is a signal processing device. The present disclosure also relates to, for example, a sound source separation unit that applies sound source separation processing to a mixed sound signal in which signals of a plurality of sound sources are mixed; a band expansion unit that applies a frequency band expansion process to each of the sound source separation signals separated by the sound source separation unit; having The band extension unit outputs only an extension band signal, which is a signal of a band extended by the frequency band extension process; moreover, a downconverter that applies a downsampling process to a mixed sound signal including a signal of a sound source that includes a high-frequency component higher than a predetermined frequency; an adder that adds the mixed sound signal and the extended band signal; having The sound source separation unit applies sound source separation processing to the signal to which the downsampling processing has been applied. It is a signal processing device. The present disclosure also relates to, for example, a sound source separation unit that applies sound source separation processing to a mixed sound signal in which signals of a plurality of sound sources are mixed; a band expansion unit that applies a frequency band expansion process to each of the sound source separation signals separated by the sound source separation unit; an adder that adds a sound source separation signal to which frequency band extension processing has been applied and a sound source separation signal to which frequency band extension processing has not been applied; A determination unit that determines whether or not to apply frequency band extension processing to the sound source separation signal; having The determination unit determines not to apply the frequency band extension process to the sound source separation signal when the sound source separation signal contains high-frequency components equal to or higher than a predetermined frequency, and determines to apply the frequency band extension process to the sound source separation signal when the sound source separation signal does not contain high-frequency components equal to or higher than a predetermined frequency. It is a signal processing device.

[0008] The present disclosure relates to, for example, A sound source separation unit applies a sound source separation process to a mixed sound signal in which signals of a plurality of sound sources are mixed; The band expansion unit applies a frequency band expansion process to each of the sound source separation signals separated by the sound source separation unit. 、 an adder adding together outputs of the band extension units provided for each sound source separation signal; A frequency envelope shaping unit shapes a frequency envelope of the combined output signal output from the adder unit; The frequency envelope shaping unit shapes the frequency envelope of the synthesis output signal when a predetermined discontinuity is detected around f1, where f1 is the lower limit of the frequency extended by the frequency band extension process. A signal processing method. The present disclosure also relates to, for example, A sound source separation unit applies a sound source separation process to a mixed sound signal in which signals of a plurality of sound sources are mixed; A band expansion unit applies a frequency band expansion process to each of the sound source separation signals separated by the sound source separation unit; The band extension unit outputs only an extension band signal, which is a signal of a band extended by the frequency band extension process; A downconverter applies a downsampling process to the mixed sound signal including a signal of a sound source including a high-frequency component higher than a predetermined frequency; an adder unit adds the mixed sound signal and the extended band signal; The sound source separation unit applies sound source separation processing to the signal to which the downsampling processing has been applied. A signal processing method. The present disclosure also relates to, for example, A sound source separation unit applies a sound source separation process to a mixed sound signal in which signals of a plurality of sound sources are mixed; A band expansion unit applies a frequency band expansion process to each of the sound source separation signals separated by the sound source separation unit; an adder adding the sound source separation signal to which the frequency band extension processing has been applied and the sound source separation signal to which the frequency band extension processing has not been applied; A determination unit determines whether or not to apply a frequency band extension process to the sound source separation signal; The determination unit determines not to apply the frequency band extension process to the sound source separation signal when the sound source separation signal contains high-frequency components equal to or higher than a predetermined frequency, and determines to apply the frequency band extension process to the sound source separation signal when the sound source separation signal does not contain high-frequency components equal to or higher than a predetermined frequency. A signal processing method.

[0009] The present disclosure relates to, for example, A sound source separation unit applies a sound source separation process to a mixed sound signal in which signals of a plurality of sound sources are mixed; The band expansion unit applies a frequency band expansion process to each of the sound source separation signals separated by the sound source separation unit. 、 an adder adding together outputs of the band extension units provided for each sound source separation signal; A frequency envelope shaping unit shapes a frequency envelope of the combined output signal output from the adder unit; The frequency envelope shaping unit shapes the frequency envelope of the synthesis output signal when a predetermined discontinuity is detected around f1, where f1 is the lower limit of the frequency extended by the frequency band extension process. This is a program that causes a computer to execute the signal processing method. The present disclosure also relates to, for example, A sound source separation unit applies a sound source separation process to a mixed sound signal in which signals of a plurality of sound sources are mixed; A band expansion unit applies a frequency band expansion process to each of the sound source separation signals separated by the sound source separation unit; The band extension unit outputs only an extension band signal, which is a signal of a band extended by the frequency band extension process; A downconverter applies a downsampling process to the mixed sound signal including a signal of a sound source including a high-frequency component higher than a predetermined frequency; an adder unit adds the mixed sound signal and the extended band signal; The sound source separation unit applies sound source separation processing to the signal to which the downsampling processing has been applied. This is a program that causes a computer to execute the signal processing method. The present disclosure also relates to, for example, A sound source separation unit applies a sound source separation process to a mixed sound signal in which signals of a plurality of sound sources are mixed; A band expansion unit applies a frequency band expansion process to each of the sound source separation signals separated by the sound source separation unit; an adder adding the sound source separation signal to which the frequency band extension processing has been applied and the sound source separation signal to which the frequency band extension processing has not been applied; A determination unit determines whether or not to apply a frequency band extension process to the sound source separation signal; The determination unit determines not to apply the frequency band extension process to the sound source separation signal when the sound source separation signal contains high-frequency components equal to or higher than a predetermined frequency, and determines to apply the frequency band extension process to the sound source separation signal when the sound source separation signal does not contain high-frequency components equal to or higher than a predetermined frequency. This is a program that causes a computer to execute the signal processing method. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a signal processing device according to the first embodiment. [Diagram 2] FIG. 2 is a diagram to be referred to when explaining an operation example of the band extending unit according to the first embodiment. [Diagram 3] FIG. 3 is a diagram to be referred to when explaining an example of the configuration of a signal processing device according to the second embodiment. [Figure 4] FIG. 4 is a diagram to be referred to when explaining the processing performed in the signal processing device according to the second embodiment. [Diagram 5] FIG. 5 is a diagram to be referred to when explaining a modified example of the signal processing device according to the second embodiment. [Figure 6] FIG. 6 is a diagram to be referred to when explaining an example of the configuration of a signal processing device according to the third embodiment. [Figure 7] FIG. 7 is a diagram to be referred to when explaining a modified example of the signal processing device according to the third embodiment. [Figure 8] FIG. 8 is a diagram to be referred to when explaining a modified example of the signal processing device according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The description will be made in the following order. <Issues to be considered in the implementation> <First embodiment> <Second embodiment> <Third embodiment> <Modification> The embodiments and the like described below are preferred specific examples of the present disclosure, and the contents of the present disclosure are not limited to these embodiments and the like.

[0012] <Issues to be considered in the implementation> First, in order to facilitate understanding of the present disclosure, problems to be considered in the embodiments will be described. As described above, a device that performs frequency band extension processing (hereinafter, appropriately abbreviated as band extension processing) is known. When extending the band of a band-limited sound source, it has been difficult to perform the band extension processing correctly because the frequency envelope (spectral envelope) differs depending on the type of sound source, such as a musical instrument. For example, percussion instruments such as cymbals and percussions and Japanese musical instruments such as shakuhachi, shamisen, and koto contain components up to very high frequencies, whereas instruments such as pianos and violins have a characteristic that attenuation increases toward higher frequencies. When the sound sources do not overlap in time, it is possible to estimate the type of sound source at each time and appropriately change the behavior (processing contents) of the band extension processing depending on the type, but in the case of music, etc., since multiple types of sound sources generally exist simultaneously, it has been difficult to perform appropriate band extension processing depending on the type of sound source.

[0013] In recent years, high-resolution audio (hereinafter referred to as high-resolution sound sources) with a sampling rate higher than 48 kHz has become widespread. When producing high-resolution sound sources, some sounds such as vocals are recorded with high-resolution sound sources, but many instruments are recorded with standard resolution audio (hereinafter referred to as standard resolution sound sources) with a sampling rate of 48 kHz or less, and there is a demand to make the sounds of all instruments high-resolution in a remastering process (remastering). In this case, it is preferable to leave the high-resolution recorded sound sources untouched and apply band expansion processing only to sound sources that are not recorded in high resolution, but since the sounds of all sound sources are mixed in the mixing process, there is a problem that it is not possible to select whether or not to apply band expansion processing for each sound source in the remastering process. The present disclosure has been made in consideration of these points. Details of the present disclosure will be described below.

[0014] <First embodiment> [Signal processing device according to the first embodiment] (Configuration example) FIG. 1 is a block diagram showing an example of the configuration of a signal processing device (signal processing device 1) according to a first embodiment. The signal processing device 1 has, for example, a sound source separation unit 11, a band expansion unit 12, and an adder unit 13. In this embodiment, a mixed sound signal x in which sounds (signals) of a plurality of (for example, N (N is a natural number)) sound sources are mixed is input to the sound source separation unit 11. The signal processing device 1 has N band expansion units (band expansion unit 121, band expansion unit 122, ... band expansion unit 12) corresponding to the number of sound sources. N In addition, when there is no need to distinguish between the individual band extension units, the band extension units are collectively referred to as band extension unit 12 as appropriate.

[0015] The sound source separation unit 11 applies a sound source separation process to the mixed sound signal x to generate sound source separation signals s1, s2, s N The sound source separation signal s1 is supplied to the band extension unit 121. The sound source separation signal s2 is supplied to the band extension unit 122. N Bandwidth expansion section 12 N are supplied to.

[0016] The sound source separation process performed by the sound source separation unit 11 is not limited to a specific process, but for example, a sound source separation process based on a multi-channel Wiener filter (MWF) using deep neural networks (DNN) can be applied, and the sound source separation process described in the above-mentioned Patent Document 1 can be applied. The sound source separation process described in Patent Document 1 is, in outline, a process of estimating an amplitude spectrum using different sound source separation methods (specifically, DNN and long short term memory (LSTM)) having outputs with different properties over time, and combining the estimation results using a predetermined combination parameter to generate a sound source separation signal. Of course, the sound source separation unit 11 may perform a sound source separation process different from the above-mentioned sound source separation process.

[0017] The band extension unit 12 applies band extension processing to each sound source separation signal s separated by the sound source separation unit 11. For example, the band extension unit 12 receives the sound source separation signal s, which is a low-frequency signal component, as an input signal, performs band extension processing on the sound source separation signal s, and outputs the resultant output signal as output signals j (output signal j1, output signal j2, output signal j3, output signal j4, output signal j5, N ) The band extension unit 12 applies a known band extension process, such as the band extension process described in the above-mentioned Patent Document 2, to the sound source separation signal s. Each band extension unit 12 is associated with a type of sound source separation signal s to be input.

[0018] In the following, the lowest frequency end of the frequency components to be expanded by the band expansion process will be referred to as the expansion start band, and signals in a band with a higher frequency than the expansion start band will be referred to as high-frequency components, and signals in a band with a lower frequency than the expansion start band will be referred to as low-frequency components, as appropriate.

[0019] The adder 13 receives the output signal j (specifically, the output signal j1, the output signal j2, the output signal j3, the output signal j4, the output signal j5, the output signal j6, the output signal j7, the output signal j8, the output signal j9, the output signal j10, the output signal j11, the output signal j12 ...3, the output signal j14, the output signal j15, the output signal j16, the output N ) to generate and output a synthetic output signal S. In this embodiment, the band-extended excitation signal that is the output of the signal processing device 1 is used as the synthetic output signal S.

[0020] (Overall operation example) Next, an example of the operation performed by the signal processing device 1 will be described. A mixed sound signal x is input to the sound source separation unit 11. The sound source separation unit 11 applies sound source separation processing to the mixed sound signal x to generate and output a sound source separation signal s. The band extension unit 12 applies band extension processing to the sound source separation signal s to generate and output an output signal j. The adder unit 13 adds each output signal j to generate and output a combined output signal S.

[0021] (Example of operation of the band expander) However, the band extension process described in the above-mentioned Patent Document 2 is based on the premise of a mixed sound, and therefore does not take into consideration the attributes of the sound source, specifically, performing optimal band extension process according to the type of sound source. For example, the envelope of a drum cymbal extends up to high frequencies without attenuation. Therefore, in this embodiment, in order to perform optimal band extension process for each type of sound source, the frequency envelope of the high-frequency component (high-frequency band) to be estimated is set for each type of sound source. Specifically, parameters of the band extension process corresponding to the type of sound source are set, and the band extension process is performed using the parameters. A device that estimates a high-frequency band that has been trained using only the type of sound source (for example, a cymbal sound) as teacher data may be applied as the band extension unit.

[0022] FIG. 2 shows an example of a frequency envelope according to the type of sound source. The horizontal axis of FIG. 2 indicates frequency (Hz), and the vertical axis indicates sound pressure (dB). Also, f1 in FIG. 2 indicates the expansion start band. Also, the frequency envelope FE1 after the expansion start band f1 in FIG. 2 is a schematic representation of the frequency envelope of a vocal sound source, and the frequency envelope FE2 after the expansion start band f1 is a schematic representation of the frequency envelope of a cymbal sound source, for example. The band expansion unit 12 corresponding to the vocal has parameters set therein for generating the frequency envelope FE1. Also, the band expansion unit 12 corresponding to the cymbal has parameters set therein for generating the frequency envelope FE2. This allows each band expansion unit 12 to perform an appropriate band expansion process according to the attributes of the sound source input thereto. The parameters are set appropriately according to the contents of the band expansion process.

[0023] <Second embodiment> Next, the second embodiment of the present disclosure will be described. The matters described in the first embodiment can also be applied to the second embodiment unless otherwise specified. The same reference symbols are used for the same or similar configurations as those in the first embodiment, and duplicated descriptions are omitted as appropriate.

[0024] [Overview of the second embodiment] When band extension processing is performed independently for each sound source separation signal, depending on the algorithm of the band extension processing, the high frequency components of the synthesis output signal S may be unnaturally emphasized. For example, if the algorithm of the band extension processing estimates only the amplitude spectrum or its envelope and replicates the phase in a certain manner (for example, uses the same one as the low frequency components (low frequency range)), and if the sound source separation algorithm does not change phase significantly for each separated sound source, the high frequency signals of each sound source separation signal whose band is extended all have similar phases. Therefore, even if the amplitude spectrum or its envelope of each sound source separation signal is correctly estimated, the high frequency signals all have similar phases, so there is a risk that the high frequency components of the synthesis output signal S may be unnaturally emphasized more than they should be. This embodiment is a signal processing device having a configuration corresponding to the above matters.

[0025] [Signal processing device according to the second embodiment] (Configuration example) 3 is a block diagram showing an example of the configuration of a signal processing device (signal processing device 2) according to the second embodiment. The signal processing device 2 differs from the signal processing device 1 in that it includes a frequency envelope shaping unit 21 at the rear stage of the addition unit 13. In this embodiment, the output of the frequency envelope shaping unit 21 is used as a band-extended sound source signal.

[0026] The frequency envelope shaping unit 21 shapes the frequency envelope of the synthesis output signal S output from the adder 13. For example, when a predetermined discontinuity is detected around the extension start band (the lower limit of the frequency extended by the band extension process) f1, the frequency envelope shaping unit 21 shapes the frequency envelope of the synthesis output signal S. In this embodiment, the detection of the predetermined discontinuity is performed by the frequency envelope shaping unit 21, but may be performed by another functional block. By shaping the frequency envelope by the frequency envelope shaping unit 21, the amplitude of the extended high frequency components is suppressed, and it is possible to prevent the high frequency components from being unnaturally emphasized.

[0027] (Example of operation) In this embodiment, the presence of a discontinuity is detected when the difference in signal energy before and after the extension start band f1 is equal to or greater than a predetermined value. A specific example will be described with reference to FIG.

[0028] The horizontal axis of Fig. 4 indicates frequency (Hz), and the vertical axis indicates sound pressure (dB). Moreover, f1 in Fig. 4 indicates the extension start band. Moreover, the frequency envelopes after the extension start band f1 in Fig. 4 (frequency envelopes FE3 to FE6) indicate examples of the frequency envelopes of the high frequency components of the synthesis output signal S.

[0029] For example, as shown in FIG. 4, predetermined frequency bands (f1-Δf) and (f1+Δf) are set before and after the extension start band f1, and the energy e of each frequency band (the shaded area in FIG. 4) is calculated for each frequency envelope. The energy in the low frequency band is e L , the energy in the high frequency band is e H With a threshold value for detecting discontinuity set to Th, if the following formula 1 is satisfied, it is determined that discontinuity exists before and after the extension start band f1. (e H / e L )>Th ···(1)

[0030] 4, when the frequency envelope of the high frequency components of the synthesis output signal S is frequency envelope FE3, the above-mentioned formula 1 is satisfied, and therefore the presence of discontinuity is detected. Since the frequency envelope FE3 unnaturally emphasizes the high frequency components, the frequency envelope shaping unit 21 performs a process of shaping the frequency envelope, specifically, a process of suppressing the amplitude of the high frequency components. The process of suppressing the amplitude may uniformly suppress the amplitude of the high frequency components, or may suppress only the amplitudes larger than a predetermined threshold value.

[0031] 4, when the frequency envelope of the high frequency components of the synthesis output signal S is the frequency envelope FE4 to FE6, it is determined that no discontinuity exists because the above-mentioned formula 1 is not satisfied. In this case, since there is no risk of the high frequency components being unnaturally emphasized, no processing is performed by the frequency envelope shaping unit 21, and the synthesis output signal S is output from the frequency envelope shaping unit 21.

[0032] According to the second embodiment described above, when band extension processing is performed, it is possible to prevent high-frequency components after the extension start band from being unnaturally emphasized.

[0033] (Modification) Next, a description will be given of a modified example of the signal processing device according to the second embodiment. Fig. 5 is a block diagram showing an example of the configuration of a signal processing device (signal processing device 2A) according to the modified example.

[0034] The signal processing device 2A does not have a frequency envelope shaping unit 21, and instead has a phase rotation unit 22. The phase rotation unit 22 is provided between the band expansion unit 12 and the addition unit 13. Specifically, the signal processing device 2A has a number of phase rotation units 22 (phase rotation units 221, 222, ... 22) corresponding to the number of band expansion units 12. N The output signals from the phase rotation units 22 are added by the adder 13.

[0035] The phase rotation unit 22 rotates (changes) the phase of the high-frequency components of the output signal j, the band of which has been expanded by the band expansion unit 12, so that the high-frequency components have different phases depending on the sound source. The phase rotation unit 22 is configured, for example, by a filter that can shift the phase without affecting the amplitude, specifically, an all-pass filter.

[0036] For example, the phase is randomly rotated by the phase rotation unit 22, so that it is possible to prevent the high-frequency components of the band-extended sound source signal from being unnaturally emphasized. Also, since the human hearing characteristic is insensitive to phase changes in high frequencies, it is possible to prevent the high-frequency components of the band-extended sound source signal from being unnaturally emphasized without giving the user a sense of discomfort in hearing.

[0037] <Third embodiment> Next, the third embodiment of the present disclosure will be described. The matters described in the first and second embodiments can also be applied to the third embodiment unless otherwise specified. The same reference symbols are used for the same or similar configurations as the first and second embodiments, and duplicated descriptions are omitted as appropriate.

[0038] [Overview of the third embodiment] As described above, there is a demand to apply band extension processing only to a standard resolution sound source among a sound source (hereinafter, appropriately referred to as a mixed sound source) including a high resolution sound source (e.g., a sound source including high frequency components after the extension start band f1) and a standard resolution sound source (e.g., a sound source not including high frequency components after the extension start band f1). This embodiment is an embodiment that meets such a demand. Note that the band of the mixed sound source includes high frequencies after the extension start band f1.

[0039] [Signal processing device according to the third embodiment] (Configuration example) 6 is a block diagram showing an example of the configuration of a signal processing device (signal processing device 3) according to a third embodiment. Like the signal processing device 1, the signal processing device 3 has a sound source separation unit 11, a band expansion unit 12 (for example, band expansion units 121 and 122), and an adder unit 13. A mixed sound source signal (hereinafter, appropriately referred to as a mixed sound source signal x1) is input to the sound source separation unit 11. The signal processing device 3 differs from the signal processing device 1 in that it has a system in which the mixed sound source signal x1 is input not only to the sound source separation unit 11 but also to the adder unit 13.

[0040] (Example of operation) Next, an operation example of the signal processing device 3 will be described. The mixed sound source signal x1 is separated for each sound source type by the sound source separation unit 11 to generate a sound source separation signal s. Of the sound source separation signals s for each sound source type, only the sound source separation signals that are not recorded in high resolution (in this example, the sound source separation signals s1 and s2) are supplied to the corresponding band extension units 121 and 122. The band extension unit 121 extends the band of the sound source separation signal s1 by performing band extension processing. Moreover, the band extension unit 122 extends the band of the sound source separation signal s2 by performing band extension processing.

[0041] Of the output signals obtained by applying the band extension process, the band extension unit 121 outputs an extension band signal p1, which is a signal containing only high-frequency components after the extension start band f1, to the adder 13. Moreover, the band extension unit 122 outputs an extension band signal p2, which is a signal containing only high-frequency components after the extension start band f1, to the adder 13, from the output signals obtained by applying the band extension process. Here, the band extension units 121 and 122 output only the extension band signals to the adder 13 because the low-frequency components of the sound source separation signals s1 and s2 are included in the mixed sound source signal x1 input to the adder 13.

[0042] The adder 13 adds the extension band signals p1, p2 and the mixed excitation signal x1 to generate and output a band extension excitation signal.

[0043] According to the third embodiment described above, it is possible to band-extend only the sound source signals that are not recorded in high resolution, without changing the high-frequency components of the sound source signals that are recorded in high resolution. In the above description, the sound source separation signals s1 and s2 are exemplified as sound source separation signals that are not recorded in high resolution, but the mixed sound source signal x1 may contain many sound source separation signals that are not recorded in high resolution.

[0044] (Variation 1) 7 is a block diagram showing a modified example of the signal processing device according to the third embodiment. In the above example, it is assumed that the sound source separation unit 11 of the signal processing device 3 has a performance capable of separating a sound source including a high-resolution sound source, but it is also assumed that the performance of the sound source separation unit 11 is not sufficient to separate a sound source including a high-resolution sound source.

[0045] In this case, as shown in Fig. 7, the sound source separation unit 11 of the signal processing device (signal processing device 3A) according to this modification has a downconverter 11A that applies downsampling processing to the mixed sound source signal x1. By performing downsampling in the downconverter 11A, the sound source separation unit 11 can perform downsampling on the mixed sound source signal x1. In the case of such a configuration, for example, the band expansion unit 121 is an upconverter 12 A1 After upsampling, the band expansion unit 121 performs band expansion processing. A2 After upsampling, the band extension process is performed by the band extension unit 122. A1 , 12 A2 The processing by may be performed before each of the band expansion units 121 and 122.

[0046] (Variation 2) 8 is a block diagram showing another modified example of the signal processing device according to the third embodiment. The sound source separation unit 11 of the signal processing device (signal processing device 3B) according to this modified example has a determination unit 11B. Note that it is assumed that the sound source separation unit 11 of the signal processing device 3B has a performance capable of separating sound sources including high-resolution sound sources.

[0047] In the signal processing device 3B, the mixed sound source signal x1 is supplied only to the sound source separation unit 11 without being supplied to the addition unit 13. The sound source separation unit 11 performs sound source separation processing on the mixed sound source signal x1 to generate sound source separation signals s1, s2 and a sound source separation signal hm corresponding to a high-resolution recorded sound source signal. The determination unit 11B determines whether or not to apply band extension processing to each sound source separation signal in a later stage. When a high-frequency component is included in the sound source separation signal, the determination unit 11B determines that it is not necessary to apply band extension processing to the sound source separation signal, and outputs the sound source separation signal to the addition unit 13. In this modification, the determination unit 11B determines that it is not necessary to apply band extension processing to the sound source separation signal hm, and the sound source separation signal is supplied from the sound source separation unit 11 to the addition unit 13.

[0048] Furthermore, when the sound source separation signal does not contain high frequency components, the determination unit 11B determines that band extension processing needs to be applied to the sound source separation signal, and outputs the sound source separation signal to the band extension unit 12. In this modification, the determination unit 11B determines that band extension processing needs to be applied to the sound source separation signals s1 and s2, and supplies them to the band extension units 121 and 122, respectively.

[0049] The band extension unit 121 generates an output signal j1 by applying band extension processing to the sound source separation signal s1. In the configuration related to the signal processing device 3B, since the mixed sound source signal x1 is not supplied to the adder 13, the band extension unit 121 outputs an output signal j1 including low-frequency components, not an extended band signal, to the adder 13. Moreover, the band extension unit 122 generates an output signal j2 by applying band extension processing to the sound source separation signal s2. In the configuration related to the signal processing device 3B, since the mixed sound source signal x1 is not supplied to the adder 13, the band extension unit 122 outputs an output signal j2 including low-frequency components, not an extended band signal, to the adder 13. The adder 13 adds the sound source separation signal hm, the output signal j1, and the output signal j2.

[0050] The signal processing device 3B according to this modification can provide the same effects as those obtained based on the above-described configuration of the signal processing device 3. Moreover, the signal processing device 3B according to this modification automatically determines whether or not to apply band extension processing, so that, for example, in a remastering process, the user does not need to know in advance which sound source separation signal the band extension processing should be applied to and select whether or not to apply the band extension processing.

[0051] <Modification> Although several embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and various modifications are possible without departing from the spirit and scope of the present disclosure.

[0052] In the above embodiment, the type of sound source is listed as an attribute of the sound source, but other attributes such as the signal properties of the sound source may be used.

[0053] When a DNN or LSTM is applied as the sound source separation unit, the input to the network is generally the amplitude spectrum of the mixed sound signal, and the training data is the amplitude spectrum of the sound of the target sound source, but the sound source separation signal after sound source separation may also be used as training data in learning.

[0054] The present disclosure may also employ a cloud computing configuration in which a single function is shared and processed collaboratively by multiple devices via a network.

[0055] The present disclosure can also be realized in any form, such as a device, a method, a program, a system, etc. For example, a program that performs the functions described in the above-mentioned embodiments can be made downloadable, and an apparatus that does not have the functions described in the embodiments can download and install the program, thereby enabling the apparatus to perform the control described in the embodiments. The present disclosure can also be realized by a server that distributes such a program. The matters described in each embodiment and modified example can be appropriately combined. The contents of the present disclosure should not be interpreted as being limited by the effects exemplified in this specification.

[0056] The present disclosure may also have the following configurations. (1) a sound source separation unit that applies sound source separation processing to a mixed sound signal in which signals of a plurality of sound sources are mixed; a band extension unit that applies a frequency band extension process to each of the sound source separation signals separated by the sound source separation unit; A signal processing device comprising: (2) The band extension unit applies a frequency band extension process according to an attribute of the sound source separation signal. A signal processing device as described in (1). (3) an adder that adds up outputs of the band extension units provided for each sound source separation signal; a frequency envelope shaping unit that shapes the frequency envelope of the combined output signal output from the adder; have A signal processing device according to (1) or (2). (4) The frequency envelope shaping unit shapes the frequency envelope of the synthesis output signal when a predetermined discontinuity is detected around f1, where f1 is a lower limit of the frequency extended by the frequency band extension process. A signal processing device according to (3). (5) If the difference in signal energy before and after f1 is equal to or greater than a predetermined value, the discontinuity is detected. A signal processing device according to (4). (6) A phase rotation unit that applies a process for rotating a phase to an output signal of the band expansion unit. A signal processing device according to (1) or (2). (7) The phase rotation unit is composed of an all-pass filter. A signal processing device according to (6). (8) The band extension unit outputs only an extension band signal, which is a signal of a band extended by the frequency band extension process. A signal processing device as described in (1). (9) a down-converter that applies a down-sampling process to the mixed sound signal including a signal of a sound source that includes a high-frequency component higher than a predetermined frequency; an adder that adds the mixed sound signal and the extended band signal, The sound source separation unit applies a sound source separation process to the signal to which the downsampling process has been applied. A signal processing device according to (8). (10) an adder that adds the sound source separation signal to which the frequency band extension processing has been applied and the sound source separation signal to which the band extension processing has not been applied; A signal processing device as described in (1). (11) a determination unit that determines whether or not to apply the frequency band extension process to the sound source separation signal; A signal processing device according to (10). (12) The determination unit determines not to apply the frequency band extension process to the sound source separation signal when the sound source separation signal contains high-frequency components equal to or higher than a predetermined frequency, and determines to apply the frequency band extension process to the sound source separation signal when the sound source separation signal does not contain high-frequency components equal to or higher than a predetermined frequency. A signal processing device according to (11). (13) A sound source separation unit applies a sound source separation process to a mixed sound signal in which signals of a plurality of sound sources are mixed; A band extension unit applies a frequency band extension process to each of the sound source separation signals separated by the sound source separation unit. Signal processing methods. (14) A sound source separation unit applies a sound source separation process to a mixed sound signal in which signals of a plurality of sound sources are mixed; A band extension unit applies a frequency band extension process to each of the sound source separation signals separated by the sound source separation unit. A program for causing a computer to execute a signal processing method. [Explanation of symbols]

[0057] 1,2,2A,3,3A,3B...Signal processing device 11...Sound source separation section 11A···Down Converter 12 Bandwidth expansion section 13 Addition section 21...Frequency envelope shaping section 22 Phase rotation section

Claims

1. a sound source separation unit that applies sound source separation processing to a mixed sound signal in which signals of a plurality of sound sources are mixed; a band extension unit that applies a frequency band extension process to each of the sound source separation signals separated by the sound source separation unit; an adder that adds up outputs of the band extension units provided for each sound source separation signal; a frequency envelope shaping unit that shapes the frequency envelope of the combined output signal output from the adder; having The frequency envelope shaping unit shapes the frequency envelope of the synthesis output signal when a predetermined discontinuity is detected around f1, where f1 is a lower limit of the frequency extended by the frequency band extension process. Signal processing device.

2. The band extension unit applies a frequency band extension process according to an attribute of the sound source separation signal. The signal processing device according to claim 1 .

3. If the difference in signal energy before and after f1 is equal to or greater than a predetermined value, the discontinuity is detected. The signal processing device according to claim 1 .

4. A phase rotation unit that applies a process for rotating a phase to an output signal of the band expansion unit. The signal processing device according to claim 1 .

5. The phase rotation unit is composed of an all-pass filter. The signal processing device according to claim 4.

6. A sound source separation unit that applies sound source separation processing to a mixed sound signal in which signals of multiple sound sources are mixed; a band extension unit that applies a frequency band extension process to each of the sound source separation signals separated by the sound source separation unit; having The band extension unit outputs only an extension band signal, which is a signal of a band extended by the frequency band extension process; moreover, a down-converter that applies a down-sampling process to the mixed sound signal including a signal of a sound source that includes a high-frequency component higher than a predetermined frequency; an adder that adds the mixed sound signal and the extended band signal; having The sound source separation unit applies a sound source separation process to the signal to which the downsampling process has been applied. Signal processing device.

7. A sound source separation unit that applies sound source separation processing to a mixed sound signal in which signals of a plurality of sound sources are mixed; a band extension unit that applies a frequency band extension process to each of the sound source separation signals separated by the sound source separation unit; an adder that adds the sound source separation signal to which the frequency band extension processing has been applied and the sound source separation signal to which the frequency band extension processing has not been applied; a determination unit that determines whether or not to apply the frequency band extension process to the sound source separation signal; having The determination unit determines not to apply the frequency band extension process to the sound source separation signal when the sound source separation signal contains high-frequency components equal to or higher than a predetermined frequency, and determines to apply the frequency band extension process to the sound source separation signal when the sound source separation signal does not contain high-frequency components equal to or higher than a predetermined frequency. Signal processing device.

8. A sound source separation unit applies a sound source separation process to a mixed sound signal in which signals of a plurality of sound sources are mixed; A band extension unit applies a frequency band extension process to each of the sound source separation signals separated by the sound source separation unit; an adder adding together outputs of the band extension units provided for each sound source separation signal; a frequency envelope shaping unit that shapes a frequency envelope of the combined output signal output from the adder unit; The frequency envelope shaping unit shapes the frequency envelope of the synthesis output signal when a predetermined discontinuity is detected around f1, where f1 is a lower limit of the frequency extended by the frequency band extension process. Signal processing methods.

9. A sound source separation unit applies a sound source separation process to a mixed sound signal in which signals of a plurality of sound sources are mixed; A band extension unit applies a frequency band extension process to each of the sound source separation signals separated by the sound source separation unit; an adder adding together outputs of the band extension units provided for each sound source separation signal; a frequency envelope shaping unit that shapes a frequency envelope of the combined output signal output from the adder unit; The frequency envelope shaping unit shapes the frequency envelope of the synthesis output signal when a predetermined discontinuity is detected around f1, where f1 is a lower limit of the frequency extended by the frequency band extension process. A program that causes a computer to execute a signal processing method.

10. A sound source separation unit applies a sound source separation process to a mixed sound signal in which signals of a plurality of sound sources are mixed, A band extension unit applies a frequency band extension process to each of the sound source separation signals separated by the sound source separation unit; The band extension unit outputs only an extension band signal, which is a signal of a band extended by the frequency band extension process; A down-converter applies a down-sampling process to the mixed sound signal including a signal of a sound source including a high-frequency component higher than a predetermined frequency; an adder unit adds the mixed sound signal and the extended band signal; The sound source separation unit applies a sound source separation process to the signal to which the downsampling process has been applied. Signal processing methods.

11. A sound source separation unit applies a sound source separation process to a mixed sound signal in which signals of a plurality of sound sources are mixed, A band extension unit applies a frequency band extension process to each of the sound source separation signals separated by the sound source separation unit; The band extension unit outputs only an extension band signal, which is a signal of a band extended by the frequency band extension process; A down-converter applies a down-sampling process to the mixed sound signal including a signal of a sound source including a high-frequency component higher than a predetermined frequency; an adder unit adds the mixed sound signal and the extended band signal; The sound source separation unit applies a sound source separation process to the signal to which the downsampling process has been applied. A program that causes a computer to execute a signal processing method.

12. A sound source separation unit applies a sound source separation process to a mixed sound signal in which signals of a plurality of sound sources are mixed, A band extension unit applies a frequency band extension process to each of the sound source separation signals separated by the sound source separation unit; an adder adding the sound source separation signal to which the frequency band extension processing has been applied and the sound source separation signal to which the frequency band extension processing has not been applied; a determination unit determining whether or not to apply the frequency band extension process to the sound source separation signal; The determination unit determines not to apply the frequency band extension process to the sound source separation signal when the sound source separation signal contains high-frequency components equal to or higher than a predetermined frequency, and determines to apply the frequency band extension process to the sound source separation signal when the sound source separation signal does not contain high-frequency components equal to or higher than a predetermined frequency. Signal processing methods.

13. A sound source separation unit applies a sound source separation process to a mixed sound signal in which signals of a plurality of sound sources are mixed, A band extension unit applies a frequency band extension process to each of the sound source separation signals separated by the sound source separation unit; an adder adding the sound source separation signal to which the frequency band extension processing has been applied and the sound source separation signal to which the frequency band extension processing has not been applied; a determination unit determining whether or not to apply the frequency band extension process to the sound source separation signal; The determination unit determines not to apply the frequency band extension process to the sound source separation signal when the sound source separation signal contains high-frequency components equal to or higher than a predetermined frequency, and determines to apply the frequency band extension process to the sound source separation signal when the sound source separation signal does not contain high-frequency components equal to or higher than a predetermined frequency. A program that causes a computer to execute a signal processing method.

Images