Sound signal processing method and sound signal processing device

Abstract

According to one embodiment, a sound signal processing method involves receiving a sound signal, performing vowel determination on the basis of the spectral shape of the sound signal during a prescribed first period of time or performing voice determination on the basis of the rate of change in the pitch of the sound signal during a prescribed second period of time, and adjusting the time characteristics or the frequency characteristics of the sound signal on the basis of the results of the vowel determination or the voice determination.

Description

Sound signal processing method and sound signal processing device

[0001] An embodiment according to the present invention relates to a sound signal processing method and a sound signal processing device.

[0002] Patent Literature 1 describes a voice enhancement device that enhances voice. The voice enhancement device generates a first mixed signal by mixing a frequency band component including a first formant with a fundamental frequency band component. The voice enhancement device generates a second mixed signal by mixing a frequency band component including a second formant with a fundamental frequency band component. The voice enhancement device generates a first voice signal to be output to an L speaker by delaying the first mixed signal based on a first delay amount. The voice enhancement device generates a second voice signal to be output to an R speaker by delaying the second mixed signal based on a second delay amount. This makes it possible to align the time it takes for a sound output from the L speaker to reach a user's ear and the time it takes for a sound output from the R speaker to reach the user's ear. As a result, the user can hear the voice more easily.

[0003] International Publication No. 2018 / 105077

[0004] When a microphone picks up a user's voice, it also picks up background noise. Conventional voice enhancement processes enhance the user's voice, but in environments with a lot of background noise, they also enhance the background noise, making it difficult to effectively enhance the user's voice.

[0005] An object of one embodiment of the present invention is to provide a sound signal processing method that can efficiently emphasize a user's voice even in an environment with a lot of background noise.

[0006] A sound signal processing method according to one embodiment of the present invention includes receiving a sound signal, making a vowel determination based on the spectral shape of the sound signal within a predetermined first time period, or making a speech determination based on the rate of change of the pitch of the sound signal within a predetermined second time period, and adjusting the time characteristics or frequency characteristics of the sound signal based on the result of the vowel determination or the speech determination.

[0007] According to the sound signal processing method of the embodiment of the present invention, it is possible to efficiently emphasize speech even in an environment with a lot of background noise.

[0008] FIG. 1 is a diagram showing an example of a sound signal processing system. FIG. 2 is a block diagram showing an example of the configuration of a PC 1. FIG. 3 is a flowchart showing an example of processing P of the processor 15. FIG. 4 is a diagram showing an example of frequency characteristics of a sound signal including vowel components and consonant components. FIG. 5 is a flowchart showing an example of processing by the processor 15 according to Modification 1. FIG. 6 is a flowchart showing an example of processing by the processor 15 according to Modification 2. FIG. 7 is a diagram showing an example of signal processing by the processor 15 according to Modification 2. FIG. 8 is a flowchart showing an example of processing by the processor 15 according to Modification 3. FIG. 9 is a flowchart showing an example of processing by the processor 15 according to Modification 4.

[0009] First Embodiment A sound signal processing method according to a first embodiment will now be described with reference to the drawings. Fig. 1 is a diagram showing an example of a sound signal processing system.

[0010] The sound signal processing system includes a PC (Personal Computer) 1 and a PC 2 connected via a communication line 3 such as the Internet. The PC 1 is an example of the sound signal processing device of the present application. Note that the communication line 3 does not necessarily have to be the Internet, and may be a LAN (Local Area Network).

[0011] In this embodiment, PC1 is used to connect to PC2 on the far-end side and conduct remote conversation. As shown in FIG. 1 , PC1 is placed in room R1. PC1 establishes communication with PC2 placed in room R2, which is different from room R1, via communication line 3. PC1 and PC2 communicate wirelessly or via a wired connection. User U1 in room R1 conducts remote conversation with user U2 in room R2 by using PC1, which has established communication with PC2. PC1 corresponds to the near-end information processing device in this application. PC2 corresponds to the far-end information processing device in this application. Note that the configuration of PC2 is the same as that of PC1, so a description of the configuration of PC2 will be omitted below.

[0012] Note that each of the rooms R1 and R2 does not necessarily have to be a closed space such as a conference room, but may be an unclosed space such as an open space.

[0013] Fig. 2 is a block diagram showing an example of the configuration of the PC 1. As shown in Fig. 2, the PC 1 includes a communication interface 10, a USB (Universal Serial Bus) interface 11, a user interface 12, a flash memory 13, a RAM (Random Access Memory) 14, and a processor 15. The processor 15 is, for example, a CPU (Central Processing Unit).

[0014] The communication interface 10 is a network interface and communicates with the PC 2 via the communication line 3 .

[0015] The USB interface 11 is an interface based on the USB standard. As shown in Fig. 2, a microphone 4 and a speaker 5 are connected to the USB interface 11. The microphone 4 acquires the voice of the user U1. The USB interface 11 transmits an audio signal received from the PC 2 via the communication interface 10 to the speaker 5. The speaker 5 outputs sound based on the audio signal. Note that a device such as a headset in which a microphone and a speaker are integrated may also be connected to the USB interface 11.

[0016] The PC 1 may be provided with an audio interface instead of the USB interface 11. In this case, the microphone 4 and the speaker 5 are connected to the audio interface. The audio interface transmits sound signals received from the PC 2 via the communication interface 10 to the speaker 5.

[0017] The microphone 4 does not necessarily have to be an external microphone, but may be a microphone built into the PC 1. The speaker 5 does not necessarily have to be an external speaker, but may be a speaker built into the PC 1.

[0018] The user interface 12 includes a mouse, a keyboard, a touch panel, etc. The user interface 12 accepts operations from the user U1.

[0019] The flash memory 13 stores various programs. The various programs include programs that operate the PC 1. However, the flash memory 13 does not necessarily have to store the various programs. The various programs may be stored in a separate device, such as a server. In this case, the PC 1 receives the various programs from the separate device, such as the server.

[0020] The processor 15 executes various operations by reading out the programs stored in the flash memory 13 into the RAM 14 .

[0021] In this embodiment, the processor 15 executes a process (hereinafter referred to as a process P) for adjusting the sound signal acquired by the microphone 4. FIG. 3 is a flowchart showing an example of the process P of the processor 15.

[0022] The processor 15 starts the process P when, for example, an application program related to a remote conversation with the PC 2 is executed (FIG. 3: START).

[0023] After starting, the processor 15 receives a sound signal from the microphone 4 (FIG. 3: step S11). Next, the processor 15 performs a process of determining whether the received sound signal contains a vowel component based on the spectral shape of the sound signal (hereinafter, referred to as vowel determination) (FIG. 3: step S12).

[0024] 4 is a diagram showing an example of the frequency characteristics of a sound signal including vowel components and consonant components. As shown in FIG. 4 , the spectrum of the sound signal including vowel components and consonant components has a shape including multiple spectral peak components (formants). Specifically, the sound signal including vowel components and consonant components includes a first formant F1, a second formant F2 in a band higher than the band of the first formant F1, and a third formant F3 in a band higher than the band of the second formant F2. In FIG. 4 , the formant frequency of the first formant F1 is 280 Hz, the formant frequency of the second formant F2 is 2 kHz, and the formant frequency of the third formant F3 is 8 kHz. The bands around 280 Hz and 2 kHz are the vowel bands of speech, and the band around 8 kHz is the consonant band of speech.

[0025] Furthermore, in a sound signal containing vowel and consonant components, the signal levels of the formants decrease from low to high frequencies. As shown in Figure 4, the signal level of the second formant F2 is lower than the signal level of the first formant F1. The signal level of the third formant F3 is lower than the signal level of the second formant F2.

[0026] The processor 15 measures the signal level of the first formant F1 (low-frequency formant) and the signal level of the second formant F2 (high-frequency formant). Specifically, the processor 15 extracts a sound signal over a predetermined period of time, such as 30 msec, and obtains a smoothed spectrum. The processor 15 measures the signal level of the first formant F1 and the signal level of the second formant F2 in the spectrum for each frame. The predetermined period of time, such as 30 msec, corresponds to the predetermined first time in this application.

[0027] The processor 15 determines whether the sound signal contains a vowel component based on the difference between the signal level of the first formant F1 and the signal level of the second formant F2. If the signal level of the second formant F2 is lower than the signal level of the first formant F1, the processor 15 determines that the sound signal contains a vowel component.

[0028] In addition, the processor 15 may further measure the signal level of the third formant F3, and if the signal level of the third formant F3 is lower than the signal level of the second formant F2, determine that the sound signal contains a vowel component.

[0029] Next, if the processor 15 determines that the sound signal contains a vowel component, it adjusts the frequency characteristics of the sound signal. As an example, the processor 15 performs a process of emphasizing the vowel band in the sound signal (hereinafter referred to as vowel emphasis processing) as an adjustment of the frequency characteristics (FIG. 3: step S13). The vowel emphasis processing is, for example, a process of increasing the signal level of the vowel band in the sound signal. The processor 15 uses, for example, a dynamic equalizer to increase the signal level of the vowel bands near 280 Hz and 2 kHz. The dynamic equalizer here is a function that increases the dynamic range of a specific band in the sound signal. This makes it possible to emphasize only the vowel band of the sound signal.

[0030] On the other hand, if the processor 15 determines that the sound signal does not contain a vowel component, it does not perform vowel emphasis processing. This prevents the signal level from increasing even though the sound signal does not contain a vowel component (is not speech), thereby preventing the background noise of room R1 from being emphasized.

[0031] Finally, the processor 15 outputs the adjusted sound signal to the communication interface 10 ( FIG. 3 : step S14). The communication interface 10 transmits the adjusted sound signal to the PC 2. The PC 2 receives the sound signal from the PC 1. The speaker of the PC 2 emits sound based on the received sound signal.

[0032] The processor 15 in the present application performs vowel determination based on the spectral shape (based on the difference in signal levels of multiple formants). In other words, the processor 15 does not determine whether a sound signal contains a vowel component solely based on the magnitude of the signal level of a specific band. Therefore, even if the signal level of the vowel band is high due to background noise, for example, the processor 15 will not erroneously determine that the sound signal contains a vowel component if the spectral shape differs from that of a vowel. Therefore, the processor 15 does not erroneously emphasize the background noise. As a result, even if there is a lot of background noise in room R1, the processor 15 can efficiently emphasize the voice of user U1 without emphasizing the background noise.

[0033] The sound output from the speaker of PC2 emphasizes vowels, which are characteristic of human speech, but does not emphasize background noise, so that even if there is a lot of background noise in room R1, user U2 can enjoy a customer experience in which user U1's voice is easy to hear.

[0034] The processor 15 ends the process P when the application program related to the remote conversation with the PC 2 is ended (FIG. 3: END). The processor 15 repeatedly performs the vowel determination and the adjustment of the frequency characteristics of the sound signal until the process P is ended.

[0035] It should be noted that the sound signal processing device in the present application does not need to transmit a sound signal to the far-end side. For example, the sound signal processing device may be a hearing aid. The hearing aid may output the adjusted sound signal to a built-in earphone and allow the user to hear a sound with emphasized vowels.

[0036] [Modification 1] Fig. 5 is a flowchart showing an example of the processing of the processor 15 according to Modification 1. The processing of steps S11, S13, and S14 in Fig. 5 is the same as the processing of steps S11, S13, and S14 in Fig. 3, and therefore description thereof will be omitted.

[0037] After the process of step S11, the processor 15 performs a voice determination based on the rate of change in the pitch of the sound signal within a predetermined time ( FIG. 5 : step S12a). Specifically, the processor 15 measures the average pitch of the sound signal over a predetermined time (predetermined second time), such as 30 msec. The processor 15 also calculates the rate of change of the average pitch over time.

[0038] For example, the pitch of a person's voice does not change suddenly during conversation. Therefore, a sound with a large rate of change in pitch is unlikely to be speech. Therefore, the processor 15 executes a process to determine whether the picked-up sound is speech or not based on the rate of change in pitch of the sound signal.

[0039] If the rate of change of the pitch is large (for example, if the rate of change of the pitch is equal to or greater than a predetermined threshold), the processor 15 determines that the picked-up sound is not speech. In this case, the processor 15 does not perform vowel emphasis processing on the sound signal. This prevents the signal level of the sound signal from increasing and emphasizing the background noise in room R1, even though the picked-up sound is not speech.

[0040] On the other hand, if the rate of change of the pitch is small, the picked-up sound is likely to be speech. Therefore, if the rate of change of the pitch is small (for example, if the rate of change of the pitch is less than a predetermined threshold), the processor 15 determines that the picked-up sound is speech. In this case, the processor 15 performs vowel emphasis processing on the sound signal.

[0041] The processor 15 in Modification 1 performs voice determination based on the rate of change of pitch. In other words, the processor 15 does not determine whether a picked-up sound is voice based solely on the magnitude of the signal level in a specific band. Therefore, even if the signal level in the vowel band is high due to background noise, the processor 15 determines that the sound is not voice if the rate of change of pitch is large. Therefore, the processor 15 does not erroneously emphasize background noise. As a result, even if there is a lot of background noise in room R1, the processor 15 can efficiently emphasize the voice of user U1 without emphasizing the background noise. This allows user U2 to enjoy a customer experience in which the voice of user U1 is easier to hear, even if there is a lot of background noise in room R1.

[0042] The processor 15 may perform the vowel determination according to the first embodiment in addition to the voice determination according to the first modification. In this case, the processor 15 determines whether a sound contains voice based on the results of both the voice determination and the vowel determination. This improves the accuracy of the determination by the processor 15 as to whether a sound is voice.

[0043] [Modification 2] Fig. 6 is a flowchart showing an example of the processing of the processor 15 according to Modification 2. The processing of steps S11, S12, and S14 in Fig. 6 is the same as the processing of steps S11, S12, and S14 in Fig. 3, and therefore description thereof will be omitted.

[0044] The processor 15 according to the second modification performs vowel emphasis processing by attenuating the signal level of bands other than the vowel band in the sound signal. In the second modification, after the processing of step S12, the processor 15 attenuates the signal level of the consonant band using DeEsser ( FIG. 6 : step S13b). The DeEsser referred to here is a type of compressor, and has a function of compressing the dynamic range of a specific band in the sound signal.

[0045] 7 is a diagram showing an example of signal processing by the processor 15 according to Modification 2. As an example, the consonant band is a band around 8 kHz (the band of the third formant F3). As shown in FIG. 7, the processor 15 attenuates the signal level in the band around 8 kHz. By attenuating the signal level of the consonant band, the vowel band in the sound signal is relatively emphasized.

[0046] In the vowel emphasis process, when the signal level of the vowel band is amplified, the signal level of the consonant band may also be amplified along with the signal level of the vowel band. Amplifying the signal level of the consonant band may emphasize harsh sounds such as sibilants in the output sound, potentially causing discomfort to the user who hears such sounds. In the second modification, the processor 15 attenuates the signal level of the consonant band. The processor 15 reduces harsh sounds such as sibilants from the sound emitted by the speaker of the PC 2. This allows user U2 to enjoy a customer experience in which he or she can converse with user U1 without feeling uncomfortable.

[0047] The processor 15 may also perform processing to increase the signal level of the vowel band using a dynamic equalizer (processing according to the first embodiment) in addition to processing to attenuate the level of the consonant band signal using a DeEsser (processing according to the second modification), thereby further emphasizing the vowel band of the sound signal.

[0048] [Modification 3] Fig. 8 is a flowchart showing an example of the processing of the processor 15 according to Modification 3. The processing of steps S11, S12, S13, and S14 in Fig. 8 is the same as the processing of steps S11, S12, S13, and S14 in Fig. 3, and therefore description thereof will be omitted.

[0049] The processor 15 stores the sound signal after the vowel emphasis processing (after step S13) in the flash memory 13 ( FIG. 8 : step S20c). The flash memory 13 is an example of a storage device in the present application. In Modification 3, the PC 1 records the vowel-emphasized sound. That is, the PC 1 according to Modification 3 functions as a voice recorder. This allows the user U1 to enjoy the customer experience of being able to listen to the recorded voice of the user U1 that has been efficiently emphasized without emphasizing background noise. Note that the sound signal processing device according to Modification 3 may be a voice recorder.

[0050] In addition, the processor 15 according to the third variant example may record the adjusted sound signal in the flash memory 13 before outputting it to the communication interface 10 (before step S14 in FIG. 8), or may record it in the flash memory 13 after outputting it to the communication interface 10 (after step S14 in FIG. 8).

[0051] [Modification 4] The processor 15 according to Modification 4 performs vowel determination and frequency characteristic adjustment on the sound signal received from the PC 2 on the far-end side.

[0052] 9 is a flowchart showing an example of the process of the processor 15 according to Modification 4. The processor 15 receives a sound signal related to the voice of the user U2 from the PC 2 via the communication interface 10 (FIG. 9: step S11d).

[0053] Next, the processor 15 performs a vowel determination on the sound signal received from the PC 2 (FIG. 9: step S12). Based on the result of the vowel determination, the processor 15 performs a vowel emphasis process on the sound signal received from the PC 2 (FIG. 9: step S13). The processes of steps S12 and S13 in FIG. 9 are the same as the processes of steps S12 and S13 in FIG. 3, and therefore a description thereof will be omitted.

[0054] Finally, the processor 15 outputs the adjusted sound signal to the USB interface 11 (FIG. 9: step S14d). The USB interface 11 transmits the adjusted sound signal to the speaker 5.

[0055] The processor 15 according to the fourth modification can efficiently emphasize the voice of the far-end user U2 without emphasizing the background noise, even if there is a lot of background noise in the far-end room R2. Therefore, even if there is a lot of background noise in the far-end room R2, the user U1 can enjoy a customer experience in which the voice of the user U2 can be easily heard.

[0056] The sound signal processing device in Modification 4 may receive sound data related to content such as a movie, instead of a sound signal acquired by a microphone on the far-end side. The sound signal processing device performs adjustment processing such as vowel emphasis on a sound signal obtained by decoding the sound data of the content. The sound signal processing device may transmit the adjusted sound signal to a speaker, thereby allowing the user to hear the sound with vowel emphasis.

[0057] The above-described embodiments and modifications should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined not by the above-described embodiments or modifications, but by the claims. Furthermore, the scope of the present invention is intended to include all modifications that are equivalent to the scope of the claims and fall within the scope thereof.

[0058] For example, the first to fourth modifications may be combined as appropriate.

[0059] 1, 2: PC, 10: communication interface, 11: USB interface, 12: user interface, 13: flash memory, 14: RAM, 15: processor, 3: communication line, 4: microphone, 5: speaker

Claims

1. A sound signal processing method comprising: receiving a sound signal; making a vowel determination based on the spectral shape of the sound signal within a predetermined first time period; or making a voice determination based on the rate of change of the pitch of the sound signal within a predetermined second time period; and adjusting the time characteristics or frequency characteristics of the sound signal based on the result of the vowel determination or the voice determination.

2. The sound signal processing method according to claim 1, wherein the vowel determination is performed based on a difference between a signal level in a low-frequency formant and a signal level in a high-frequency formant of the sound signal.

3. The sound signal processing method according to claim 2, wherein the low-frequency formant includes a first formant of the sound signal, and the high-frequency formant includes a second formant in a band higher than the band of the first formant, and a third formant in a band higher than the band of the second formant.

4. A sound signal processing method according to any one of claims 1 to 3, comprising the steps of: performing the vowel determination or the voice determination on a sound signal received by a microphone; and adjusting the time characteristics or frequency characteristics of the sound signal received by the microphone based on the result of the vowel determination or the voice determination.

5. A sound signal processing method as described in claim 4, wherein an information processing device on the near-end side performs the vowel determination or the voice determination and adjusts the time characteristics or the frequency characteristics, and the information processing device on the near-end side outputs a sound signal whose time characteristics or frequency characteristics have been adjusted to an information processing device on the far-end side.

6. The sound signal processing method according to any one of claims 1 to 3, further comprising recording the sound signal after adjusting the time characteristic or the frequency characteristic.

7. A sound signal processing method according to any one of claims 1 to 3, wherein the adjustment includes vowel emphasis processing for emphasizing a vowel band in the sound signal.

8. The sound signal processing method according to claim 7, wherein in the vowel emphasis processing, the signal level of a band other than the band of the vowel in the sound signal is attenuated.

9. The sound signal processing method according to claim 7, wherein in said vowel emphasis processing, a signal level of a consonant band in said sound signal is attenuated.

10. The sound signal processing method according to claim 7, wherein in the vowel emphasis processing, a signal level of a vowel band in the sound signal is increased.

11. A sound signal processing device comprising a processor that receives a sound signal, performs a vowel determination based on the spectral shape of the sound signal within a predetermined first time period, or performs a voice determination based on the rate of change of the pitch of the sound signal within a predetermined second time period, and adjusts the time characteristics or frequency characteristics of the sound signal based on the result of the vowel determination or the voice determination.

12. The sound signal processing device according to claim 11, wherein the processor performs the vowel determination based on a difference between a signal level in a low-frequency formant and a signal level in a high-frequency formant of the sound signal.

13. The sound signal processing device according to claim 12, wherein the low-frequency formant includes a first formant of the sound signal, and the high-frequency formant includes a second formant in a band higher than the band of the first formant, and a third formant in a band higher than the band of the second formant.

14. A sound signal processing device as described in any one of claims 11 to 13, wherein the processor performs the vowel determination or the voice determination on a sound signal received by a microphone, and adjusts the time characteristics or frequency characteristics of the sound signal received by the microphone based on the result of the vowel determination or the voice determination.

15. The sound signal processing device according to claim 14, wherein the processor outputs the sound signal, the time characteristic or the frequency characteristic of which has been adjusted, to a far-end information processing device.

16. The sound signal processing device according to any one of claims 11 to 13, further comprising a storage device, wherein the storage device records the sound signal after the time characteristic or the frequency characteristic has been adjusted.

17. The sound signal processing device according to claim 11, wherein the adjustment includes vowel emphasis processing for emphasizing a vowel band in the sound signal.

18. The sound signal processing device according to claim 17, wherein the processor attenuates a signal level of a band other than the band of the vowel in the sound signal in the vowel emphasis processing.

19. The sound signal processing device according to claim 17, wherein the processor attenuates a signal level of a consonant band in the sound signal in the vowel emphasis processing.

20. The sound signal processing device according to claim 17, wherein the processor increases a signal level of a vowel band in the sound signal in the vowel emphasis processing.

Images