Sound signal processing method and sound signal processor

Abstract

To provide a sound signal processing method capable of efficiently emphasizing voice of a user even in environment with much background noise.SOLUTION: A sound signal processing method includes: a step of receiving a sound signal; a step of determining vowels based on a spectral shape pf the sound signal within a prescribed first time period or determining voice based on a change rate of pitches of the sound signal within a prescribed second time period; and a step of adjusting a time characteristic or a frequency characteristic of the sound signal based on a result of vowel determination or voice determination.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] An embodiment according to the present invention relates to a sound signal processing method and a sound signal processing device. [Background technology]

[0002] Patent Document 1 describes a voice enhancement device that enhances voice. The voice enhancement device generates a first mixed signal by mixing a frequency band component including a first formant and a fundamental frequency band component. The voice enhancement device generates a second mixed signal by mixing a frequency band component including a second formant and a fundamental frequency band component. The voice enhancement device generates a first voice signal to be output to the L speaker by delay-controlling the first mixed signal based on a first delay amount. The voice enhancement device generates a second voice signal to be output to the R speaker by delay-controlling the second mixed signal based on a second delay amount. This makes it possible to align the time it takes for the sound output from the L speaker to reach the user's ear and the time it takes for the sound output from the R speaker to reach the user's ear. As a result, the user can easily hear the voice. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 105077 Summary of the Invention [Problem to be solved by the invention]

[0004] When a microphone picks up a user's voice, it also picks up background noise. Conventional voice enhancement processes enhance the user's voice, but in environments with a lot of background noise, the process also enhances the background noise, making it difficult to enhance the user's voice efficiently.

[0005] An object of one embodiment of the present invention is to provide a sound signal processing method that can efficiently emphasize a user's voice even in an environment with a lot of background noise. [Means for solving the problem]

[0006] A sound signal processing method according to an embodiment of the present invention includes: Accepts sound signals, performing a vowel determination based on a spectral shape of the sound signal within a predetermined first time period, or performing a voice determination based on a rate of change of the pitch of the sound signal within a predetermined second time period; Based on the result of the vowel determination or the voice determination, the time characteristic or the frequency characteristic of the sound signal is adjusted. Effect of the Invention

[0007] According to the audio signal processing method according to an embodiment of the present invention, it is possible to effectively emphasize a voice even in an environment with a lot of background noise. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a sound signal processing system. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of the PC 1. [Diagram 3] FIG. 3 is a flowchart showing an example of a process P of the processor 15. [Figure 4] FIG. 4 is a diagram showing an example of frequency characteristics of a sound signal including vowel components and consonant components. [Diagram 5] FIG. 5 is a flowchart showing an example of the process of the processor 15 according to the first modification. [Figure 6] FIG. 6 is a flowchart showing an example of the process of the processor 15 according to the second modification. [Figure 7] FIG. 7 is a diagram showing an example of signal processing by the processor 15 according to the second modification. [Figure 8]FIG. 8 is a flowchart showing an example of the process of the processor 15 according to the third modification. [Figure 9] FIG. 9 is a flowchart showing an example of the process of the processor 15 according to the fourth modification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [First embodiment] The sound signal processing method according to the first embodiment will be described below with reference to the drawings. Fig. 1 is a diagram showing an example of a sound signal processing system.

[0010] The sound signal processing system includes a PC (Personal Computer) 1 and a PC 2 connected via a communication line 3 such as the Internet. The PC 1 is an example of a sound signal processing device in the present application. Note that the communication line 3 does not necessarily have to be the Internet, and may be a LAN (Local Area Network).

[0011] In this embodiment, PC1 is used to connect to PC2 on the far end side and conduct remote conversation. As shown in FIG. 1, PC1 is placed in room R1. PC1 establishes communication with PC2 placed in room R2 different from room R1 via communication line 3. PC1 and PC2 communicate wirelessly or wired. User U1 in room R1 conducts remote conversation with user U2 in room R2 by using PC1 with which communication with PC2 has been established. PC1 corresponds to the near end side information processing device in this application. PC2 corresponds to the far end side information processing device in this application. Note that the configuration of PC2 is the same as that of PC1, and therefore, hereinafter, a description of the configuration of PC2 will be omitted.

[0012] Each of the rooms R1 and R2 does not necessarily have to be a closed space such as a conference room, etc. Each of the rooms R1 and R2 may be an unclosed space such as an open space, for example.

[0013] Fig. 2 is a block diagram showing an example of the configuration of the PC 1. As shown in Fig. 2, the PC 1 includes a communication interface 10, a USB (Universal Serial Bus) interface 11, a user interface 12, a flash memory 13, a RAM (Random Access Memory) 14, and a processor 15. The processor 15 is, for example, a CPU (Central Processing Unit).

[0014] The communication interface 10 is a network interface and communicates with the PC 2 via the communication line 3.

[0015] The USB interface 11 is an interface based on the USB standard. As shown in FIG. 2, a microphone 4 and a speaker 5 are connected to the USB interface 11. The microphone 4 acquires the voice of the user U1. The USB interface 11 transmits an audio signal received from the PC 2 via the communication interface 10 to the speaker 5. The speaker 5 outputs sound based on the audio signal. Note that a device such as a headset in which a microphone and a speaker are integrated may be connected to the USB interface 11.

[0016] The PC 1 may include an audio interface instead of the USB interface 11. In this case, the microphone 4 and the speaker 5 are connected to the audio interface. The audio interface transmits a sound signal received from the PC 2 via the communication interface 10 to the speaker 5.

[0017] The microphone 4 does not necessarily have to be an external microphone, but may be a microphone built into the PC 1. The speaker 5 does not necessarily have to be an external speaker, but may be a speaker built into the PC 1.

[0018] The user interface 12 includes a mouse, a keyboard, a touch panel, etc. The user interface 12 accepts operations by a user U1.

[0019] The flash memory 13 stores various programs. The various programs are programs that operate the PC 1, etc. It is not necessary for the flash memory 13 to store the various programs. The various programs may be stored in another device, such as a server. In this case, the PC 1 receives the various programs from the other device, such as the server.

[0020] The processor 15 reads out the programs stored in the flash memory 13 into the RAM 14 to execute various operations.

[0021] In this embodiment, the processor 15 executes a process (hereinafter, referred to as a process P) for adjusting the sound signal acquired by the microphone 4. FIG.

[0022] The processor 15 starts the process P when, for example, an application program related to a remote conversation with the PC 2 is executed (FIG. 3: START).

[0023] After starting, the processor 15 receives a sound signal from the microphone 4 (FIG. 3: step S11). Next, the processor 15 performs a process of determining whether or not the received sound signal contains a vowel component based on the spectral shape of the sound signal (hereinafter, vowel determination) (FIG. 3: step S12).

[0024] FIG. 4 is a diagram showing an example of frequency characteristics of a sound signal including a vowel component and a consonant component. As shown in FIG. 4, the spectrum of the sound signal including a vowel component and a consonant component has a shape including a plurality of spectral peak components (formants). Specifically, the sound signal including a vowel component and a consonant component includes a first formant F1, a second formant F2 in a band higher than the band of the first formant F1, and a third formant F3 in a band higher than the band of the second formant F2. In FIG. 4, the formant frequency of the first formant F1 is 280 Hz, the formant frequency of the second formant F2 is 2 kHz, and the formant frequency of the third formant F3 is 8 kHz. The bands around 280 Hz and 2 kHz are the vowel bands of the voice, and the band around 8 kHz is the consonant band of the voice.

[0025] In addition, in a sound signal including vowel and consonant components, the signal levels of the formants decrease from low to high frequencies. As shown in Fig. 4, the signal level of the second formant F2 is lower than the signal level of the first formant F1. The signal level of the third formant F3 is lower than the signal level of the second formant F2.

[0026] The processor 15 measures the signal level of the first formant F1 (low-frequency formant) and the signal level of the second formant F2 (high-frequency formant). Specifically, the processor 15 extracts a sound signal of a predetermined time, such as 30 msec, and obtains a smoothed spectrum. The processor 15 measures the signal level of the first formant F1 and the signal level of the second formant F2 in the spectrum for each frame. The predetermined time, such as 30 msec, corresponds to the predetermined first time in this application.

[0027] The processor 15 performs a vowel determination based on the difference between the signal level of the first formant F1 and the signal level of the second formant F2. When the signal level of the second formant F2 is lower than the signal level of the first formant F1, the processor 15 determines that the sound signal contains a vowel component.

[0028] Furthermore, the processor 15 may further measure the signal level of the third formant F3, and determine that the sound signal contains a vowel component if the signal level of the third formant F3 is lower than the signal level of the second formant F2.

[0029] Next, when the processor 15 determines that the sound signal contains a vowel component, it adjusts the frequency characteristics of the sound signal. As an example, the processor 15 performs a process of emphasizing the vowel band in the sound signal (hereinafter referred to as vowel emphasis processing) as an adjustment of the frequency characteristics (FIG. 3: step S13). As an example, the vowel emphasis processing is a process of increasing the signal level of the vowel band in the sound signal. The processor 15 increases the signal level of the vowel band near 280 Hz and near 2 kHz, for example, using a dynamic equalizer. The dynamic equalizer referred to here is a function that increases the dynamic range of a specific band in the sound signal. This makes it possible to emphasize only the vowel band of the sound signal.

[0030] On the other hand, when the processor 15 determines that the sound signal does not contain a vowel component, it does not perform the vowel emphasis process. This prevents the signal level from increasing even though the sound signal does not contain a vowel component (is not a voice), and the background noise of the room R1 from being emphasized.

[0031] Finally, the processor 15 outputs the adjusted sound signal to the communication interface 10 (FIG. 3: step S14). The communication interface 10 transmits the adjusted sound signal to the PC2. The PC2 receives the sound signal from the PC1. The speaker of the PC2 emits a sound based on the received sound signal.

[0032] The processor 15 in the present application performs vowel determination based on the spectral shape (based on the difference in signal levels of multiple formants). In other words, the processor 15 does not determine whether a sound signal contains a vowel component based only on the magnitude of the signal level of a specific band. For this reason, even if the signal level of the vowel band is high due to background noise, for example, the processor 15 will not erroneously determine that the sound signal contains a vowel component if the spectral shape differs from that of a vowel. Therefore, the processor 15 does not erroneously emphasize the background noise. As a result, even if there is a lot of background noise in room R1, the processor 15 can efficiently emphasize the voice of user U1 without emphasizing the background noise.

[0033] In the sound output from the speaker of PC2, vowels, which are characteristic of human speech, are emphasized, while background noise is not emphasized. Therefore, even if there is a lot of background noise in room R1, user U2 can enjoy a customer experience in which the voice of user U1 is easily audible.

[0034] The processor 15 ends the process P when the application program related to the remote conversation with the PC 2 is ended (FIG. 3: END). The processor 15 repeats the vowel determination and the adjustment of the frequency characteristics of the sound signal until the process P is ended.

[0035] It should be noted that the sound signal processing device in the present application does not need to transmit a sound signal to the far-end side. For example, the sound signal processing device may be a hearing aid. The hearing aid may output the adjusted sound signal to a built-in earphone and allow the user to hear a sound with emphasized vowels.

[0036] [Variation 1] Fig. 5 is a flowchart showing an example of the process of the processor 15 according to the modified example 1. The processes of steps S11, S13, and S14 in Fig. 5 are the same as the processes of steps S11, S13, and S14 in Fig. 3, and therefore description thereof will be omitted.

[0037] After the process of step S11, processor 15 performs voice determination based on the rate of change of the pitch of the sound signal within a predetermined time (FIG. 5: step S12a). Specifically, processor 15 measures the average pitch of the sound signal for a predetermined time (predetermined second time), such as 30 msec. Processor 15 also calculates the rate of change of the average pitch over time.

[0038] For example, in conversation, the pitch of a person's voice does not change suddenly. Therefore, a sound with a large rate of change in pitch is likely not a voice. Therefore, the processor 15 executes a process to determine whether a picked-up sound is a voice or not based on the rate of change in pitch of the sound signal.

[0039] When the rate of change of the pitch is large (for example, when the rate of change of the pitch is equal to or greater than a predetermined threshold), the processor 15 determines that the "picked sound is not speech." In this case, the processor 15 does not perform vowel emphasis processing on the sound signal. This prevents the signal level of the sound signal from increasing and the background noise of the room R1 from being emphasized, even though the picked-up sound is not speech.

[0040] On the other hand, if the rate of change of the pitch is small, the picked up sound is likely to be a voice. Therefore, if the rate of change of the pitch is small (for example, if the rate of change of the pitch is less than a predetermined threshold value), the processor 15 determines that the "picked up sound is a voice." In this case, the processor 15 performs a vowel emphasis process on the sound signal.

[0041] The processor 15 in the first modification performs voice determination based on the rate of change of pitch. That is, the processor 15 does not determine whether a picked-up sound is voice or not based only on the magnitude of the signal level of a specific band. For this reason, even if the signal level of the vowel band is high due to background noise, the processor 15 determines that it is not voice if the rate of change of pitch is large. Therefore, the processor 15 does not erroneously emphasize the background noise. As a result, even if there is a lot of background noise in the room R1, the processor 15 can efficiently emphasize the voice of the user U1 without emphasizing the background noise. This allows the user U2 to enjoy a customer experience in which the voice of the user U1 is easily audible even if there is a lot of background noise in the room R1.

[0042] The processor 15 may execute the vowel determination according to the first embodiment in addition to the voice determination according to the first modification. In this case, the processor 15 determines whether the sound contains voice based on the results of both the voice determination and the vowel determination. This improves the accuracy of the determination by the processor 15 as to whether the sound is voice.

[0043] [Variation 2] Fig. 6 is a flowchart showing an example of the process of the processor 15 according to the modified example 2. The processes of steps S11, S12, and S14 in Fig. 6 are the same as the processes of steps S11, S12, and S14 in Fig. 3, and therefore description thereof will be omitted.

[0044] The processor 15 according to the second modification performs vowel emphasis processing by attenuating the signal level of bands other than the vowel band in the sound signal. In the second modification, after the processing of step S12, the processor 15 attenuates the signal level of the consonant band by DeEsser (FIG. 6: step S13b). The DeEsser referred to here is a type of compressor, and has a function of compressing the dynamic range of a specific band in the sound signal.

[0045] Fig. 7 is a diagram showing an example of signal processing by the processor 15 according to the second modification. As an example, the consonant band is a band around 8 kHz (the band of the third formant F3). As shown in Fig. 7, the processor 15 attenuates the signal level in the band around the frequency of 8 kHz. By attenuating the level of the signal in the consonant band, the vowel band in the sound signal is relatively emphasized.

[0046] In the vowel emphasis process, when the signal level of the vowel band is amplified, the signal level of the consonant band may also be amplified together with the signal level of the vowel band. By amplifying the signal level of the consonant band, harsh sounds such as sibilants are emphasized in the output sound, and the user who hears the sound may feel uncomfortable. In the second modification, the processor 15 attenuates the signal level of the consonant band. The processor 15 reduces harsh sounds such as sibilants from the sound emitted by the speaker of the PC2. As a result, the user U2 can enjoy a customer experience of being able to converse with the user U1 without feeling uncomfortable.

[0047] The processor 15 may perform processing to increase the signal level of the vowel band using a dynamic equalizer (processing according to the first embodiment) in addition to processing to attenuate the level of the consonant band signal using the DeEsser (processing according to the second modification). This further emphasizes the vowel band of the sound signal.

[0048] [Variation 3] Fig. 8 is a flowchart showing an example of the process of the processor 15 according to the modified example 3. The processes of steps S11, S12, S13, and S14 in Fig. 8 are the same as the processes of steps S11, S12, S13, and S14 in Fig. 3, and therefore description thereof will be omitted.

[0049] The processor 15 stores the sound signal after the vowel emphasis process (after step S13) in the flash memory 13 (FIG. 8: step S20c). The flash memory 13 is an example of a storage device in the present application. In the third modification, the PC1 records the vowel-emphasized sound. That is, the PC1 according to the third modification functions as a voice recorder. This allows the user U1 to enjoy a customer experience in which the user U1 can hear the recorded voice of the user U1 that has been efficiently emphasized without emphasizing background noise. The sound signal processing device according to the third modification may be a voice recorder.

[0050] In addition, the processor 15 in accordance with variant example 3 may record the adjusted sound signal in the flash memory 13 before outputting it to the communication interface 10 (before step S14 in FIG. 8), or may record it in the flash memory 13 after outputting it to the communication interface 10 (after step S14 in FIG. 8).

[0051] [Variation 4] The processor 15 according to the fourth modification performs vowel determination and frequency characteristic adjustment on the sound signal received from the PC 2 on the far-end side.

[0052] 9 is a flowchart showing an example of the process of the processor 15 according to the modified example 4. The processor 15 receives a sound signal related to the voice of the user U2 from the PC 2 via the communication interface 10 (FIG. 9: step S11d).

[0053] Next, the processor 15 performs vowel determination on the sound signal received from the PC 2 (FIG. 9: step S12). The processor 15 performs vowel emphasis processing on the sound signal received from the PC 2 based on the result of the vowel determination (FIG. 9: step S13). The processing of steps S12 and S13 in FIG. 9 is the same as the processing of steps S12 and S13 in FIG. 3, and therefore description thereof will be omitted.

[0054] Finally, the processor 15 outputs the adjusted sound signal to the USB interface 11 (FIG. 9: step S14d). The USB interface 11 transmits the adjusted sound signal to the speaker 5.

[0055] The processor 15 according to the fourth modification can efficiently emphasize the voice of the user U2 on the far-end side without emphasizing the background noise even if there is a lot of background noise in the room R2 on the far-end side. Therefore, even if there is a lot of background noise in the room R2 on the far-end side, the user U1 can enjoy a customer experience in which the voice of the user U2 can be easily heard.

[0056] The sound signal processing device in the fourth modification may receive sound data related to content such as a movie, instead of a sound signal acquired by a microphone on the far end side. The sound signal processing device performs adjustment processing such as vowel emphasis on a sound signal obtained by decoding the sound data of the content. The sound signal processing device may transmit the adjusted sound signal to a speaker, thereby allowing the user to hear the sound with vowel emphasis.

[0057] The above-mentioned embodiments and the description of the modifications should be considered as illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims, not by the above-mentioned embodiments or modifications. Furthermore, the scope of the present invention is intended to include all modifications within the meaning and scope of the claims.

[0058] For example, the first to fourth modifications may be combined as appropriate. [Explanation of symbols]

[0059] 1, 2: PC, 10: communication interface, 11: USB interface, 12: user interface, 13: flash memory, 14: RAM, 15: processor, 3: communication line, 4: microphone, 5: speaker

Claims

1. Accepts sound signals, A vowel determination is performed based on a spectral shape of the sound signal within a predetermined first time, or a voice determination is performed based on a rate of change of the pitch of the sound signal within a predetermined second time, adjusting a time characteristic or a frequency characteristic of the sound signal based on a result of the vowel determination or the voice determination; A method for processing an audio signal.

2. the vowel determination is performed based on a difference between a signal level of a low-frequency formant and a signal level of a high-frequency formant of the sound signal. The method for processing a sound signal according to claim 1 .

3. the low-frequency formant includes a first formant of the sound signal; The high frequency formants are a second formant having a frequency band higher than the frequency band of the first formant; a third formant having a frequency band higher than the frequency band of the second formant; Including, The method for processing a sound signal according to claim 2.

4. performing the vowel determination or the voice determination on the sound signal received by the microphone; adjusting a time characteristic or a frequency characteristic of the sound signal received by the microphone based on a result of the vowel determination or the voice determination; The sound signal processing method according to any one of claims 1 to 3.

5. A near-end information processing device The vowel determination or the voice determination; Adjusting the time characteristic or the frequency characteristic; Do the following: The near-end information processing device outputs the sound signal, the time characteristic or the frequency characteristic of which has been adjusted, to the far-end information processing device. The sound signal processing method according to claim 4.

6. Recording the sound signal after adjusting the time characteristic or the frequency characteristic. The sound signal processing method according to any one of claims 1 to 3.

7. The adjustment includes a vowel emphasis process for emphasizing a vowel band in the sound signal. The sound signal processing method according to any one of claims 1 to 3.

8. In the vowel emphasis processing, a signal level of a band other than the band of the vowel in the sound signal is attenuated. The method for processing a sound signal according to claim 7.

9. In the vowel emphasis processing, a signal level of a consonant band in the sound signal is attenuated. The method for processing a sound signal according to claim 7.

10. In the vowel emphasis processing, a signal level of a vowel band in the sound signal is increased. The method for processing a sound signal according to claim 7.

11. Accepts sound signals, A vowel determination is performed based on a spectral shape of the sound signal within a predetermined first time, or a voice determination is performed based on a rate of change of the pitch of the sound signal within a predetermined second time, adjusting a time characteristic or a frequency characteristic of the sound signal based on a result of the vowel determination or the voice determination; A processor is provided. Sound signal processing device.

12. The processor performs the vowel determination based on a difference between a signal level of a low-frequency formant and a signal level of a high-frequency formant of the sound signal. The sound signal processing device according to claim 11 .

13. the low-frequency formant includes a first formant of the sound signal; The high frequency formants are a second formant having a frequency band higher than the frequency band of the first formant; a third formant having a frequency band higher than the frequency band of the second formant; Including, The sound signal processing device according to claim 12.

14. The processor, performing the vowel determination or the voice determination on the sound signal received by the microphone; adjusting a time characteristic or a frequency characteristic of the sound signal received by the microphone based on a result of the vowel determination or the voice determination; The sound signal processing device according to any one of claims 11 to 13.

15. The processor outputs the sound signal having the time characteristic or the frequency characteristic adjusted to a far-end information processing device. The sound signal processing device according to claim 14.

16. Further comprising a storage device; The storage device records the sound signal after adjusting the time characteristic or the frequency characteristic. The sound signal processing device according to any one of claims 11 to 13.

17. The adjustment includes a vowel emphasis process for emphasizing a vowel band in the sound signal. The sound signal processing device according to any one of claims 11 to 13.

18. In the vowel emphasis processing, the processor attenuates a signal level of a band other than the band of the vowel in the sound signal. The sound signal processing device according to claim 17.

19. The processor attenuates a signal level of a consonant band in the sound signal in the vowel emphasis processing. The sound signal processing device according to claim 17.

20. The processor increases a signal level of a vowel band in the sound signal in the vowel emphasis processing. The sound signal processing device according to claim 17.

Images