Sound signal processing method in conference system

Abstract

To accurately determine whether a sound signal acquired in a conference system is generated by a musical performance, and appropriately switch signal processing for obtaining a sound signal to be transmitted to a communication partner.SOLUTION: A sound signal processing method includes acquiring an image signal with an imaging unit 7, acquiring a sound signal with a sound collection unit 6, and executing a performance determination P01 to determine whether the sound signal has been generated by a performance based on the image signal and the sound signal. If the sound signal is generated by a performance, a first signal process for the performance is executed (step S31) to obtain a sound signal to be transmitted to a communication partner. If the sound signal is not generated by a performance, a second signal process for conversation is executed (step S32) to obtain a sound signal to be transmitted to the communication partner.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an audio signal processing method for obtaining an audio signal to be transmitted to a communication partner in a conference system. [Background technology]

[0002] As an example of a technology related to sound signal processing for obtaining a sound signal to be transmitted to a communication partner in a conference system, there is a technology disclosed in Patent Document 1. In the technology disclosed in Patent Document 1, sound signals collected in a conference room are subjected to acoustic processing optimized for the acoustic environment of the conference room, and the signals are transmitted to the communication partner.

[0003] With the spread of conference systems, conference systems have come to be used not only for transmitting audio signals of conversations in conferences, but also for transmitting audio signals of music. However, the conversation signal processing for obtaining the audio signal to be transmitted to the communication partner includes, for example, processing for removing noise components that interfere with the transmission of the conversation audio signal. When such conversation signal processing is applied to the audio signal obtained by playing a musical performance, the obtained audio signal loses the musical expression of the performance sound.

[0004] In view of this, the technique disclosed in Patent Document 2 acquires a sound signal in a conference room, and determines whether or not musical content is present in the sound signal, thereby switching signal processing for the sound signal. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2007-060460 A

[0006] [Patent Document 2] U.S. Patent No. 11,562,761 Summary of the Invention [Problem to be solved by the invention]

[0007] However, it has been difficult to accurately determine whether musical elements are present in a sound signal captured in a conference room.

[0008] This invention has been made in consideration of the above circumstances, and has as its object to provide a technical means for accurately determining whether or not a sound signal acquired in a conference system has been generated by a musical performance, and for appropriately switching signal processing to obtain a sound signal to be transmitted to the other party. [Means for solving the problem]

[0009] This invention provides a sound signal processing method for a conference system, which acquires an image signal by an imaging unit, acquires a sound signal by a sound collection unit, and performs a performance determination based on the image signal and the sound signal to determine whether the sound signal has been generated by a performance, and if the sound signal has been generated by a performance, performs a first signal processing for performance to obtain a sound signal to be transmitted to a communication partner, and if the sound signal has not been generated by a performance, performs a second signal processing for conversation to obtain a sound signal to be transmitted to a communication partner. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram of a conference terminal that executes a sound signal processing method according to an embodiment of the present invention. [Diagram 2] 4 is a flowchart of a performance determination process executed in the embodiment. [Diagram 3] 4 is a flowchart of a sound signal generation process executed in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] 1 is a block diagram showing the configuration of a conference terminal 10 that executes a sound signal processing method according to an embodiment of the present invention. The conference terminal 10 is a terminal that constitutes a conference system that connects to other conference terminals in remote locations to hold remote conferences. The conference terminal 10 may be, for example, a personal computer.

[0013] In Fig. 1, a processor 1 is the control center of a conference terminal 10. A storage unit 2 is composed of a non-volatile storage unit such as a hard disk, and a volatile storage unit such as a RAM. The non-volatile storage unit stores various programs executed by the processor 1. The volatile storage unit is used by the processor 1 as a work area.

[0014] The display unit 3 is a means for displaying various information, and is configured, for example, by a liquid crystal display panel. The operation unit 4 is a means for receiving operation information from a user, and includes operators such as a keyboard and a mouse. The communication unit 5 is a means for communicating with other conference terminals, etc., via a network.

[0015] The sound collection unit 6 is a means for collecting sounds generated in the conference room and acquiring sound signals, and includes one or more microphones. The sound collection unit 6 may include a microphone installed in the conference room, or may include a microphone carried by the user (or the conference participant). The sound collection unit 6 may also include a microphone array capable of beam control. The imaging unit 7 includes one or more cameras that capture images in the conference room and acquire image signals. The sound emission unit 8 includes a speaker installed in the conference room. Sound signals received from the conference terminal of the communication partner are emitted from this stationary speaker. The sound emission unit 8 may include earphones or headphones worn by the user (or the conference participant). In the conference terminal 10, the sound signals acquired by the sound collection unit 6 may be emitted from these earphones or headphones.

[0016] One of the programs stored in the storage unit 2 is a conference application. When the conference application is started by the user, the processor 1 connects to another conference terminal in a remote location and executes processing for the remote conference. This processing for the remote conference includes a process of transmitting an image signal acquired by the imaging unit 7 and an audio signal acquired by the sound collection unit 6 to a connected communication partner (e.g., another conference terminal). The latter process, i.e., a process of transmitting an audio signal acquired by the sound collection unit to the communication partner, includes a sound signal generation process of generating an audio signal to be transmitted to the communication partner from the audio signal acquired by the sound collection unit.

[0017] In the conference room, the sound signal acquired by the sound pickup unit 6 contains noise that hinders the transmission of conversational voices, such as noise from an air conditioner, noise caused by conference participants turning over papers, etc. Therefore, in the sound signal generation process, when transmitting a sound signal of conversational voices, a process for improving the transmission capability of conversational voices, specifically, a signal process for conversation is executed that includes a process for removing the above-mentioned noises from the sound signal.

[0018] On the other hand, in this embodiment, for example, a musical instrument may be played in the conference room, and a sound signal generated by the performance may be transmitted to the communication partner by the conference terminal 10. If the above-mentioned signal processing for conversation is performed on such a sound signal generated by the performance, the components generating the expressiveness of the performance in the sound signal are removed as noise, which is undesirable. In addition, it is desirable to perform processing on the sound signal generated by the performance, such as an effecter, to enhance the expressiveness of the performance. Therefore, in the sound signal generation processing, when the sound signal generated by the performance is to be transmitted, signal processing for performance is performed to enhance the transmission capability of the sound signal generated by the performance.

[0019] In order to switch between signal processes as described above, it is necessary to determine whether or not the sound signal acquired by the sound collection unit 6 is generated by a musical performance. Therefore, the processor 1 in this embodiment acquires an image signal by the imaging unit 7, acquires a sound signal by the sound collection unit 6, and performs a performance determination as to whether or not the acquired sound signal is generated by a musical performance based on the image signal and the sound signal. If the sound signal is generated by a musical performance, the processor 1 executes a first signal processing for musical performance to obtain a sound signal to be transmitted to the communication partner, and if the sound signal is not generated by a musical performance, the processor 1 executes a second signal processing for conversation to obtain a sound signal to be transmitted to the communication partner.

[0020] Next, the operation of this embodiment will be described. In this embodiment, when a conference application is started, the processor 1 executes a performance judgment P01 shown in FIG. 2 and a sound signal generation process P02 shown in FIG. 3 in parallel in accordance with the conference application. The performance judgment P01 and the sound signal generation process P02 constitute a sound signal processing method according to this embodiment. Here, the performance judgment P01 is a process for judging whether or not a sound signal acquired by the sound collection unit 6 has been generated by a performance. Moreover, the sound signal generation process P02 is a process for executing a first signal process for a performance (step S110) or a second signal process for a conversation (step S120) in accordance with the judgment result of the performance judgment P01 in order to obtain a sound signal to be transmitted to a communication partner.

[0021] 2, first, the processor 1 determines whether or not a musical instrument has been detected, that is, whether or not the image signal acquired by the imaging unit 7 indicates some musical instrument such as a guitar, a trumpet, etc. (step S11). If the result of the determination in step S11 is "YES", the process of the processor 1 proceeds to step S12, and if the result is "NO", the process of the processor 1 proceeds to step S22.

[0022] Next, in step S12, the processor 1 judges whether the sound signal acquired by the sound collection unit 6 is generated by playing. Specifically, when the image signal acquired by the imaging unit 7 indicates a playing action and the sound pressure of the sound signal acquired by the sound collection unit 6 is equal to or higher than a threshold, the processor 1 judges that the sound signal is generated by playing. Here, when the image signal indicates a playing action, the playing action corresponds to, for example, an action of keeping the mouth in contact with the instrument for a predetermined time or more, an action of touching the finger to the instrument and moving the finger for a predetermined time or more, an action of keeping an accessory (stick, pick, drumstick, etc.) in contact with the instrument for a predetermined time or more, etc. If the judgment result in step S12 is "YES", the process of the processor 1 proceeds to step S21, and if the judgment result is "NO", the process of the processor 1 proceeds to step S22.

[0023] Steps S21 to S25 constitute a hysteresis process (step S20) for preventing signal processing from switching unnecessarily frequently or unnaturally.

[0024] When the determination results in steps S11 and S12 are "YES" and the process proceeds to step S21, the processor 1 clears the non-performance duration to 0. Here, the non-performance duration is the duration during which the determination result in step S11 is "NO" or the determination result in step S12 is "NO", that is, the duration during which the determination result that the sound signal acquired by the sound pickup unit 6 is not generated by a performance continues to occur.

[0025] Next, the processor 1 judges whether the input volume of the sound pickup unit 6 is equal to or greater than a predetermined value (step S25). If the judgment result of step S25 is "YES", the processor 1 sets the flag F to "1" (step S31). This flag F="1" is a flag that instructs the execution of a first signal processing for performance (step S110). When the process of step S31 ends, the process of the processor 1 returns to step S11.

[0026] When the result of the determination in step S11 or S12 is "NO", the process proceeds to step S22, where the processor 1 measures the non-performance duration. Specifically, the processor 1 updates the non-performance duration by adding a predetermined time to the current non-performance duration.

[0027] Next, the processor 1 judges whether or not the non-performance duration time is equal to or greater than a threshold value (step S23). If the judgment result in step S23 is "YES", the processor 1 judges whether or not the input volume of the sound collection unit 6 is equal to or greater than a predetermined value (step S24). Then, if the judgment result in step S24 is "NO", the processor 1 sets the flag F to "0" (step S31). On the other hand, if the judgment result in step S24 is "YES", the processor 1 sets the flag F to "0" (step S32). The processor 11 also sets the flag F to "0" (step S32) if the judgment result in step S25 is "NO".

[0028] This flag F="0" is a flag that instructs execution of the second signal processing for conversation (step S120). When the process of step S32 ends, the process of the processor 1 returns to step S11. In the performance determination P01, the above processes are repeatedly executed.

[0029] 3, first, the processor 1 judges whether the flag F is "1" or not (step S100). If the judgment result of step S100 is "YES", the processor 1 executes a first signal processing for musical performance (step S110) to obtain a sound signal to be transmitted to the communication partner. If the judgment result of step S100 is "NO", the processor 1 executes a second signal processing for conversation (step S120) to obtain a sound signal to be transmitted to the communication partner.

[0030] The first signal processing (step S110) is a time domain signal processing with low latency (short delay time). In this first signal processing (step S110), the channel strip (step S111) is a process for one channel in a general mixer, and includes processes such as COMP (Compressor), EQ (Equalizer), and HPF (High Pass Filter).

[0031] The echo canceller (step S112) is a process for preventing the occurrence of echoes of the performance sound by removing a sound signal corresponding to a sound signal received from the communication partner from the sound signal acquired by the sound pickup unit 6. The low-level suppressor (step S113) is a process for suppressing low-level sound signals acquired by the sound pickup unit 6. Here, a low-level signal means a weak signal that is not possible for the performance sound of a musical instrument.

[0032] The loopback process (step S114) is a process of adding a sound signal of a sound reproduced from a storage medium such as a DVD to a sound signal acquired by the sound collection unit 6 and transmitting the sound to the communication partner. The direct monitoring process (step S115) is a process of emitting the sound signal acquired by the sound collection unit 6 from the sound emission unit 8. The microphone volume adjustment process (step S116) is a process of adjusting the volume of a desired microphone in the sound collection unit 6 in response to an operation of the operation unit 4, etc. The loopback process (step S114), the direct monitoring process (step S115), and the microphone volume adjustment process (step S116) are each executed by performing a predetermined operation on the operation unit 4.

[0033] The second signal processing (step S120) is a frequency domain signal processing, which has a high latency (long delay time). In the second signal processing (step S120), a time domain sound signal acquired by the sound collection unit 6 is converted into a frequency domain signal in FFT (Fast Fourier Transform) (step S121). This frequency domain signal is the processing target of steps S122 to S125.

[0034] The echo canceller (step S122) is a process for removing a sound signal corresponding to a sound signal received from a communication partner from the sound signal acquired by the sound pickup unit 6, thereby preventing echoes from occurring in conversation voices.

[0035] Beam control (step S123) is a process of controlling the directivity of the microphone array in the sound collection unit 6 so that sound from the direction of the sound source of the conversation voice (e.g., the mouth of the speaker) is amplified and sound from other directions is attenuated. Suppressor (step S124) is a process of removing noise that interferes with the transmission of the conversation voice, such as noise generated by an air conditioner, from the sound signal acquired by the sound collection unit 6. Automatic adjustment (step S125) includes processes such as EQ and volume adjustment to clearly transmit the conversation voice to the communication partner.

[0036] IFFT (Inverse Fast Fourie Transform) is a process (step S126) for converting the frequency domain signal that has been subjected to the processes in steps S122 to S125 described above into a time domain signal.

[0037] When the first signal processing (step S110) or the second signal processing (step S120) ends, the process of the processor 1 returns to step S100. In the sound signal generation process P02, the above processes are repeatedly executed. The above is the operation of this embodiment.

[0038] As described above, according to this embodiment, a performance determination is made as to whether or not a sound signal is generated by a performance based on the sound signal acquired by the sound collection unit 6 and the image signal acquired by the imaging unit 7, so that the performance determination can be performed accurately. Therefore, according to this embodiment, appropriate signal processing is performed based on the result of the performance determination, and a sound signal to be transmitted to the communication partner can be obtained. [Explanation of symbols]

[0039] 10...conference terminal, 1...processor, 2...memory unit, 3...display unit, 4...operation unit, 5...communication unit, 6...sound pickup unit, 7...imaging unit, 8...sound emission unit.

Claims

1. An image signal is acquired by an imaging unit; The sound signal is acquired by the sound pickup section, performing a performance determination as to whether or not the sound signal is generated by a performance based on the image signal and the sound signal; When the sound signal is generated by a musical performance, a first signal processing for musical performance is executed to obtain a sound signal to be transmitted to a communication partner; if the sound signal is not generated by a musical performance, a second signal processing for conversation is executed to obtain a sound signal to be transmitted to a communication partner. A method for processing an audio signal in a conference system.

2. 2. The sound signal processing method according to claim 1, wherein said performance determination step determines that said sound signal is generated by a performance if said image signal indicates a musical performance and if a sound pressure of said sound signal is equal to or higher than a threshold value.

3. 3. The sound signal processing method according to claim 2, wherein said performance determination step determines that said performance has occurred when said image signal indicates contact between a musical instrument or an accessory thereof and a human body.

4. The sound signal processing method according to claim 1 , wherein the second signal processing includes a process of removing noise that interferes with transmission of conversational voice from the sound signal acquired by the sound pickup section.

5. The sound signal processing method according to claim 1 , wherein the first signal processing includes a loop-back process of adding a sound signal of a reproduced sound to the sound signal acquired by the sound pickup section.

6. The sound signal processing method according to claim 1 , wherein the first signal processing includes a direct monitoring process of outputting the sound signal acquired by the sound collection section from a sound emission section.

7. The sound signal processing method according to claim 1 , wherein the first signal processing includes a process of adjusting a volume of the sound pickup unit.

Images