Acoustic processing method and acoustic processing system

Abstract

To appropriately control a processing parameter that is applied to acoustic processing for a sound collection signal in an environment in which sound emission of far end sound received from a far end device and sound collection of near end sound pronounced by a near end user are performed in parallel.SOLUTION: A sound emission processing system 100a receives an acoustic signal X representing far end sound pronounced by a user of a far end from a far end device, and performs sound emission of the far end sound represented by the acoustic signal X by means of a sound emission device 15. An acoustic processing unit 30 generates an acoustic signal Y by performing acoustic processing applying a processing parameter for sound collection signal Ra generated by a sound collection device 14 on the basis of sound collection of audio including near end sound pronounced by a user Ua in a near end. A communication control unit 20 transmits the acoustic signal Y to the far end device. An operation control unit 60 controls processing parameters such that a processing parameter in a case where at least one of the near end sound and the far end sound includes musical performance sound differs from a processing parameter in a case where the at least one does not include the musical performance sound.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to techniques for processing acoustic signals. [Background technology]

[0002] In an environment where multiple communication devices each having a sound emitting device and a sound collecting device communicate with each other via a communication network, echoes caused by feedback sounds propagating from the sound emitting device to the sound collecting device become a problem. For example, Patent Document 1 discloses an echo reduction device that generates a pseudo echo signal that approximates the feedback sound using an adaptive filter and subtracts the pseudo echo signal from a collected sound signal generated by a sound collecting device. In the technology of Patent Document 1, multiple coefficients of the adaptive filter are updated when the user on the far end side is speaking, and the update of the coefficients is stopped when the user on the far end side is not speaking. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-163305 A Summary of the Invention [Problem to be solved by the invention]

[0004] By the way, there are cases where multiple users play music in remote locations using communication devices. For example, remote music lessons in which a remote instructor teaches a player of an instrument, or remote ensemble in which multiple players play a common piece of music in remote locations are assumed. However, if the frequency response of the adaptive filter fluctuates due to the update of the coefficients during the period in which each user is playing, the performance expression intended by the user may be diminished. Note that the above description focuses on the update of the coefficients of the adaptive filter, but similar problems are assumed in other types of acoustic processing of a collected signal by a sound collection device. In consideration of the above circumstances, one of the objectives of the present disclosure is to appropriately control processing parameters applied to acoustic processing of a collected signal in an environment in which the emission of a far-end sound received from a far-end device and the collection of a near-end sound produced by a near-end user are executed in parallel. [Means for solving the problem]

[0005] In order to solve the above problems, an acoustic processing method according to one embodiment of the present disclosure receives a first acoustic signal representing a far-end sound produced by a first user from a far-end device, emits the far-end sound represented by the first acoustic signal by a sound emission device, generates a second acoustic signal by applying acoustic processing to a picked-up signal generated by a sound collection device by collecting sound including a near-end sound produced by a near-end second user, transmits the second acoustic signal to the far-end device, updates the processing parameter according to the first acoustic signal or the picked-up signal, and stops updating the processing parameter when at least one of the near-end sound and the far-end sound includes a performance sound.

[0006] An acoustic processing method according to another aspect of the present disclosure includes receiving a first acoustic signal representing a far-end sound produced by a first user from a far-end device, emitting the far-end sound represented by the first acoustic signal by a sound emitting device, generating a second acoustic signal by applying acoustic processing to a picked-up signal generated by a sound collecting device by collecting sound including a near-end sound produced by a second user at the near-end, transmitting the second acoustic signal to the far-end device, updating the processing parameter according to the first acoustic signal or the picked-up signal, and controlling the update of the processing parameter so that an update speed of the processing parameter when at least one of the near-end sound and the far-end sound includes a performance sound differs from an update speed of the processing parameter when the near-end sound does not include the performance sound.

[0007] An acoustic processing system according to one embodiment of the present disclosure is an acoustic processing system that receives a first acoustic signal representing a far-end sound produced by a first user from a far-end device and emits the far-end sound represented by the first acoustic signal by an audio output device, and includes an acoustic processing unit that generates a second acoustic signal by applying acoustic processing parameters to a picked-up signal generated by the sound input device by collecting sound including a near-end sound produced by a near-end second user, a communication control unit that transmits the second acoustic signal to the far-end device, an update processing unit that updates the processing parameters in accordance with the first acoustic signal or the picked-up signal, and an operation control unit that stops updating the processing parameters when at least one of the near-end sound and the far-end sound includes a performance sound.

[0008] An acoustic processing system according to another aspect of the present disclosure is an acoustic processing system that receives a first acoustic signal representing a far-end sound produced by a first user from a far-end device and emits the far-end sound represented by the first acoustic signal by an acoustic emission device, and includes an acoustic processing unit that generates a second acoustic signal by applying acoustic processing parameters to a picked-up signal generated by the sound pickup device by collecting sound including a near-end sound produced by a second user at the near-end, a communication control unit that transmits the second acoustic signal to the far-end device, an update processing unit that updates the processing parameter in accordance with the first acoustic signal or the picked-up signal, and an operation control unit that controls the update of the processing parameter so that an update speed of the processing parameter when at least one of the near-end sound and the far-end sound includes a performance sound differs from an update speed of the processing parameter when the near-end sound does not include the performance sound. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating a configuration of a communication system according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram illustrating a configuration of a sound processing system. [Diagram 3] FIG. 2 is a block diagram illustrating an example of a functional configuration of the sound processing system. [Figure 4] 4 is a block diagram illustrating a specific configuration of an echo suppressor; FIG. [Diagram 5] 11 is a flowchart illustrating a specific procedure of a determination process. [Figure 6] FIG. 4 is an explanatory diagram of an estimation model used by the determination processing unit. [Figure 7] 5 is an explanatory diagram of the operation of the operation control unit. FIG. [Figure 8] 10 is a flowchart illustrating a specific procedure of a control process. [Figure 9] FIG. 11 is a block diagram illustrating an example of a functional configuration of a sound processing system according to a second embodiment. [Figure 10] FIG. 11 is a block diagram illustrating a specific configuration of a setting unit in the second embodiment. [Figure 11]13 is an explanatory diagram of the operation of the operation control unit in the second embodiment. FIG. [Figure 12] 10 is a flowchart illustrating a specific procedure of a control process in the second embodiment. [Figure 13] FIG. 11 is a block diagram illustrating an example of a functional configuration of a sound processing system according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] A: First embodiment FIG. 1 is a block diagram illustrating a configuration of a communication system 1 according to a first embodiment. The communication system 1 is a computer system used, for example, for music lessons, and includes an audio processing system 100a and an audio processing system 100b. Each of the audio processing system 100a and the audio processing system 100b is realized by an information terminal such as a mobile phone, a smartphone, a tablet terminal, or a personal computer. Note that a remote conference device (so-called speakerphone) used for transmitting and receiving sound between remote locations is also used as the audio processing system 100a or the audio processing system 100b. The audio processing system 100a and the audio processing system 100b communicate with each other via a communication network 200 such as the Internet. Note that the communication method between the audio processing system 100a and the audio processing system 100b is arbitrary. For example, a part of the communication path established between the audio processing system 100a and the audio processing system 100b may be a wireless section.

[0011] The user Ua uses the sound processing system 100a, and the user Ub uses the sound processing system 100b. The user Ua plays the musical instrument 300a, and the user Ub plays the musical instrument 300b. The user Ua is, for example, an instructor who teaches the user Ub how to play the musical instrument 300b. The user Ub is, for example, a trainee who is trained by the user Ua. The sound processing system 100a and the musical instrument 300a are installed in an acoustic space where the user Ua is located (for example, a music classroom), and the sound processing system 100b and the musical instrument 300b are installed in an acoustic space where the user Ub is located (for example, the home of the user Ub). The musical instruments 300a and 300b are natural musical instruments that produce sound when played. For example, various natural musical instruments such as keyboard instruments, string instruments, or wind instruments are used as the musical instruments 300a and 300b. It is also assumed that the user Ub is an instructor and the user Ua is a trainee.

[0012] The user Ua plays the musical instrument 300a and speaks to the user Ub. For example, the user Ua plays the musical instrument 300a as a model for the user Ub to refer to, and speaks to instruct the user Ub. In the following description, for convenience, it is assumed that the speech by the user Ua and the performance of the musical instrument 300a are performed in different periods on the time axis. On the other hand, the user Ub plays the musical instrument 300b and speaks to the user Ua. For example, the user Ub plays the musical instrument 300b for practice, and speaks to the user Ua to ask a question, etc. In the following description, for convenience, it is assumed that the speech by the user Ub and the performance of the musical instrument 300b are performed in different periods on the time axis.

[0013] The sound processing system 100b transmits a sound signal X to the sound processing system 100a. The sound signal X is a signal representing the sound around the sound processing system 100b. Specifically, the sound signal X represents a performance sound produced from the musical instrument 300b when the user Ub is playing, or a speech sound produced by the user Ub. Furthermore, the sound processing system 100a transmits a sound signal Y to the sound processing system 100b. The sound signal Y is a signal representing the sound around the sound processing system 100a. Specifically, the sound signal Y represents a performance sound produced from the musical instrument 300a when the user Ua is playing, or a speech sound produced by the user Ua.

[0014] The performance sounds include instrument sounds produced by the musical instrument 300a or 300b, as well as singing sounds produced by the singing of the user Ua or user Ub. In other words, the performance sounds are collectively expressed as sounds expressing music (musical sounds). Furthermore, "performance" includes the operation of producing sounds from the musical instrument 300a or 300b, as well as singing by the user Ua or user Ub. On the other hand, speech sounds are sounds expressing language (language sounds).

[0015] The sound processing system 100a emits sound represented by the sound signal X to the user Ua. The user Ua plays the musical instrument 300a or speaks to the user Ub while listening to the sound of the musical instrument 300b being played by the user Ub or the sound of the speech by the user Ub. The sound processing system 100b also emits sound represented by the sound signal Y to the user Ub. The user Ub plays the musical instrument 300b or speaks to the user Ub while listening to the sound of the musical instrument 300a being played by the user Ua or the sound of the speech by the user Ua.

[0016] 2 is a block diagram illustrating a specific configuration of the sound processing system 100a. Since the configuration of the sound processing system 100b is similar to that of the sound processing system 100a, detailed description of the sound processing system 100b will be omitted. When focusing on the sound processing system 100a, the sound processing system 100b is an example of a "far-end device".

[0017] The sound processing system 100a includes a control device 11, a storage device 12, a communication device 13, a sound collection device 14, and a sound emission device 15. The sound processing system 100a may be realized as a single device, or may be realized as a plurality of devices configured separately from each other.

[0018] The control device 11 is a single or multiple processors that control each element of the sound processing system 100a. Specifically, the control device 11 is configured by one or more types of processors, such as a CPU (Central Processing Unit), an SPU (Sound Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit).

[0019] The storage device 12 is one or more memories that store programs executed by the control device 11 and various data used by the control device 11. The storage device 12 is configured with a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. In addition, a portable recording medium that is detachably attached to the sound processing system 100a, or a recording medium that the control device 11 can access via the communication network 200 (e.g., cloud storage) may be used as the storage device 12.

[0020] The communication device 13 communicates with the acoustic processing system 100b via the communication network 200. Specifically, the communication device 13 receives an acoustic signal X transmitted from the acoustic processing system 100b. The communication device 13 also transmits an acoustic signal Y to the acoustic processing system 100b. The acoustic signal X is an example of a "first acoustic signal," and the acoustic signal Y is an example of a "second acoustic signal."

[0021] The sound emitting device 15 is a speaker that emits sound (hereinafter referred to as "far-end sound") represented by the sound signal X received by the communication device 13 from the sound processing system 100b. That is, the performance sound of the musical instrument 300b or the speech sound of the user Ub is emitted as the far-end sound from the sound emitting device 15. For convenience, a D / A converter that converts the sound signal X from digital to analog is omitted from the illustration. In addition, the sound emitting device 15 configured separately from the sound processing system 100a may be connected to the sound processing system 100a by wire or wirelessly.

[0022] The sound collection device 14 is a microphone that collects surrounding sounds to generate a collected sound signal Ra. The sound collection device 14 configured separately from the sound processing system 100a may be connected to the sound processing system 100a in a wired or wireless manner.

[0023] Specifically, the sound collection device 14 collects sound produced by the near-end user Ua (hereinafter referred to as "near-end sound"). The near-end sound is a performance sound of the musical instrument 300a or a speech sound of the user Ua. The near-end sound can also be said as a target sound (target sound) transmitted from the sound processing system 100a to the sound processing system 100b. In the first embodiment, the sound emitting device 15 emits the far-end sound and the sound collection device 14 collects the near-end sound in parallel.

[0024] Sounds other than the near-end sound also reach the sound collection device 14. For example, feedback sound from the sound emitting device 15 reaches the sound collection device 14. The feedback sound is sound reflected by the wall of the acoustic space after sound is emitted by the sound emitting device 15, or sound that arrives directly from the sound emitting device 15 to the sound collection device 14. Noise present in the acoustic space also reaches the sound collection device 14. The noise is stationary environmental noise such as the operating noise of air conditioning equipment. As can be understood from the above explanation, the collected sound signal Ra contains predominantly acoustic components of the near-end sound, but may also contain acoustic components other than the near-end sound.

[0025] 3 is a block diagram illustrating a functional configuration of the sound processing system 100a. The control device 11 of the sound processing system 100a executes a program stored in the storage device 12 to realize a plurality of functions (a communication control unit 20, a playback processing unit 25, a sound processing unit 30, an update processing unit 40, a determination processing unit 50, and an operation control unit 60).

[0026] The communication control unit 20 receives the acoustic signal X transmitted from the acoustic processing system 100b through the communication device 13. The reproduction processing unit 25 performs signal processing, such as equalization, on the acoustic signal X received by the communication control unit 20. The acoustic signal X processed by the reproduction processing unit 25 is supplied to the sound emitting device 15. By supplying the acoustic signal X to the sound emitting device 15, the speech sound of the user Ub or the performance sound of the musical instrument 300b is emitted as a far-end sound.

[0027] The acoustic processing unit 30 generates an acoustic signal Y by performing acoustic processing on the collected sound signal Ra generated by the sound collection device 14. The communication control unit 20 transmits the acoustic signal Y generated by the acoustic processing unit 30 from the communication device 13 to the acoustic processing system 100b (far-end device). The acoustic processing unit 30 of the first embodiment includes an echo suppression unit 31, a noise suppression unit 32, and a volume adjustment unit 33.

[0028] The echo suppressor 31 generates a collected sound signal Rb by executing echo suppression processing on the collected sound signal Ra. The echo suppression processing is signal processing (AEC: Adaptive Echo Canceller) that suppresses feedback sound (i.e., echo) included in the collected sound signal Ra. That is, a collected sound signal Rb in which the near-end sound included in the collected sound signal Ra is emphasized is generated.

[0029] 4 is a block diagram illustrating a specific configuration of the echo suppressor 31. The echo suppressor 31 of the first embodiment includes an adaptive filter 311 and a subtraction processor 312. The adaptive filter 311 generates a pseudo echo signal E from the acoustic signal X. The pseudo echo signal E is an acoustic signal that approximates a feedback sound that reaches the sound collector 14 from the sound emitting device 15. The subtraction processor 312 generates a collected sound signal Rb by subtracting the pseudo echo signal E from the collected sound signal Ra. As can be understood from the above description, the echo suppression process executed by the echo suppressor 31 includes an adaptive filter process that generates a pseudo echo signal E from the acoustic signal X, and a subtraction process that subtracts the pseudo echo signal E from the collected sound signal Ra.

[0030] The adaptive filter 311 of the first embodiment is a finite impulse response (FIR) filter including a plurality of (N) adjustment units 315_1 to 315_N and one adder 316. The n-th (n=1 to N) adjustment unit 315_n is supplied with an acoustic signal X delayed by (n-1) delay units 317. The adjustment unit 315_n adjusts the volume of the acoustic signal X according to a coefficient Cn. Specifically, the adjustment unit 315_n is a multiplier that multiplies the acoustic signal X by a coefficient Cn. The adder 316 generates a pseudo echo signal E by adding the N acoustic signals X adjusted by the N adjustment units 315_1 to 315_N. The N coefficients C1 to CN are controlled according to the picked-up sound signal Rb so that the pseudo echo signal E approximates a feedback sound. Note that the specific configuration of the adaptive filter 311 is not limited to the example shown in FIG. 4. Any known configuration may be adopted for adaptive filter 311 as long as it is capable of executing adaptive filter processing in which response characteristics change according to N coefficients C1 to CN.

[0031] The noise suppression unit 32 in FIG. 3 generates a collected signal Rc by executing noise suppression processing on the collected signal Rb. The noise suppression processing is signal processing that suppresses noise components contained in the collected signal Rb. The noise components contained in the collected signal Rb are, for example, stationary environmental noise such as the operating sound of an air conditioning device. The noise suppression processing is, for example, spectral subtraction (SS) that subtracts a frequency spectrum of the noise components (hereinafter referred to as "noise spectrum") Q from the frequency spectrum of the collected signal Rb in the frequency domain. Specifically, the noise suppression processing includes frequency analysis that calculates the frequency spectrum of the collected signal Rb, subtraction processing that subtracts the noise spectrum Q from the frequency spectrum, and waveform synthesis that converts the frequency spectrum after subtraction into a collected signal Rc in the time domain. The noise spectrum Q is a parameter that represents the noise components contained in the collected signal Rb.

[0032] 3 executes a volume adjustment process on the picked-up signal Rc to generate an acoustic signal Y. The acoustic adjustment process is signal processing (AGC: Auto Gain Control) that amplifies the picked-up signal Rc by a gain G according to the volume of the picked-up signal Rc.

[0033] As can be understood from the above description, the acoustic processing performed by the acoustic processing unit 30 of the first embodiment includes echo suppression processing, noise suppression processing, and volume adjustment processing. Processing parameters are applied to the acoustic processing. The processing parameters of the first embodiment include N coefficients C1 to CN applied to the echo suppression processing, a noise spectrum Q applied to the noise suppression processing, and a gain G applied to the volume adjustment processing. The order of the processes included in the acoustic processing is not limited to the above example. For example, the order of the noise suppression processing and the volume adjustment processing may be reversed.

[0034] The update processing unit 40 updates the processing parameters applied to the acoustic processing by the acoustic processing unit 30 in response to the acoustic signal X or the picked-up sound signal R (Ra to Rc). The update processing unit 40 updates the processing parameters at a predetermined cycle. The update processing unit 40 of the first embodiment includes a setting unit 41, a setting unit 42, and a setting unit 43.

[0035] The setting unit 41 updates N coefficients C1 to CN applied to the echo suppression process. Specifically, the setting unit 41 iteratively updates each of the N coefficients C1 to CN in accordance with the acoustic signal X and the picked-up sound signals Ra and Rb so that the pseudo echo signal E approximates the feedback sound.

[0036] The setting unit 42 repeatedly updates the noise spectrum Q applied to the noise suppression process according to the picked-up sound signal Rb. Specifically, the setting unit 42 estimates the frequency spectrum of the picked-up sound signal Rb during a period in which both the near-end sound and the far-end sound are silent as the noise spectrum Q. The setting unit 42 may update the noise spectrum Q according to the picked-up sound signal Ra.

[0037] The setting unit 43 repeatedly updates the gain G applied to the volume adjustment process according to the volume of the picked-up sound signal Rc. Specifically, the setting unit 43 sets the gain G to a smaller value as the volume of the picked-up sound signal Rc increases. The setting unit 43 may update the gain G according to the volume of the picked-up sound signal Ra or the picked-up sound signal Rb.

[0038] The determination processing unit 50 in FIG. 3 analyzes the state of pronunciation by the user Ua and the user Ub. Specifically, the determination processing unit 50 determines whether each of the far-end sound represented by the audio signal X and the near-end sound represented by the collected sound signal R (Ra, Rb, or Rc) corresponds to (1) a silent state, (2) a state including a performance sound, or (3) a state including a speech sound. The silent state of the near-end sound is a state in which the volume of the near-end sound is below a predetermined threshold. The state in which the near-end sound includes a performance sound is a state in which the near-end sound includes only a performance sound and does not include a speech sound, or a state in which the near-end sound includes both a performance sound and a speech sound but the volume of the performance sound exceeds the volume of the speech sound. Similarly, the state in which the near-end sound includes a speech sound is a state in which the near-end sound includes a speech sound but does not include a performance sound, or a state in which the near-end sound includes both a performance sound and a speech sound but the volume of the speech sound exceeds the volume of the performance sound. In the above explanation, the state of the near-end sound has been focused on, but the state of the far-end sound is defined in the same way. Also, when the volume of the performance sound and the volume of the speech sound are equivalent in the far-end sound or near-end sound, the determination processing unit 50 determines that the far-end sound or near-end sound includes the performance sound.

[0039] The determination processing unit 50 determines the type of far-end sound (silence / performance sound / speech sound) by analyzing the audio signal X. The determination processing unit 50 also determines the type of near-end sound (silence / performance sound / speech sound) by analyzing the picked-up sound signal R. For the determination regarding the near-end sound, any of the picked-up sound signal Ra, the picked-up sound signal Rb, and the picked-up sound signal Rc is used.

[0040] 5 is a flowchart illustrating a specific procedure of the operation (hereinafter referred to as "determination process") Sa of the determination processor 50. The determination process Sa by the determination processor 50 is repeated, for example, at a predetermined period. Note that in the following description, the determination process Sa for the near-end sound represented by the picked-up sound signal R is illustrated for convenience, but the determination process Sa is also executed for the far-end sound represented by the acoustic signal X in the same manner.

[0041] When the judgment process Sa is started, the judgment processing unit 50 calculates the volume of the near-end sound represented by the picked-up signal R (Sa1), and judges whether the volume of the near-end sound exceeds a predetermined threshold value (Sa2). If the volume of the near-end sound is below the threshold value (Sa2: NO), the judgment processing unit 50 sets the judgment data of the near-end sound to a numerical value representing silence (Sa3). The judgment data is data representing the result of the judgment by the judgment processing unit 50, and is stored in the storage device 12 for each of the near-end sound and the far-end sound.

[0042] The threshold applied to the determination of silence is experimentally or statistically set to be higher than the volume of stationary noise such as the operating sound of an air conditioning system and lower than the volume of significant performance sounds or speech sounds. As can be understood from the above explanation, the state in which the near-end sound or the far-end sound is silent includes not only a completely silent state in which not even noise is present, but also a state in which noise is present.

[0043] On the other hand, if the volume of the near-end sound exceeds the threshold (Sa2: YES), the judgment processing unit 50 judges whether or not the near-end sound includes a performance sound (Sa4). If it is judged that the near-end sound includes a performance sound (Sa4: YES), the judgment processing unit 50 sets the judgment data of the near-end sound to a numerical value representing a performance sound (Sa5). On the other hand, if it is judged that the near-end sound does not include a performance sound (Sa4: NO), the judgment processing unit 50 sets the judgment data of the near-end sound to a numerical value representing a speech sound (Sa6). In other words, if the volume of the near-end sound exceeds the threshold and the near-end sound does not include a performance sound, the near-end sound is judged to include a speech sound.

[0044] The determination process Sa illustrated in Fig. 5 is also executed for the far-end sound represented by the audio signal X. For example, if the volume of the far-end sound is below the threshold (Sa2: NO), the determination data of the far-end sound is set to a value representing silence (Sa3). If the far-end sound includes a performance sound (Sa4: YES), the determination data of the far-end sound is set to a value representing a performance sound (Sa5). Also, if the far-end sound does not include a performance sound (Sa4: NO), the determination data of the far-end sound is set to a value representing a speech sound (Sa6).

[0045] The process in which the determination processing unit 50 determines whether the near-end sound includes a performance sound or a speech sound utilizes an estimation model 51 illustrated in Fig. 6. The estimation model 51 is a statistical estimation model that generates output data D2 from input data D1. Specifically, the estimation model 51 is a deep neural network that has learned the relationship between the input data D1 and the output data D2. For example, a deep neural network of any type, such as a convolutional neural network (CNN) or a recurrent neural network (RNN), is utilized as the estimation model 51.

[0046] The input data D1 is data corresponding to the audio signal X or the collected sound signal R. Specifically, a feature amount related to the acoustic characteristics of the far-end sound represented by the audio signal X or the near-end sound represented by the collected sound signal R is supplied to the estimation model 51 as the input data D1. The feature amount of the far-end sound or the near-end sound is, for example, MFCC (Mel-Frequency Cepstrum Coefficients) representing the characteristics of the tone color. However, a frequency spectrum calculated from the audio signal X or the collected sound signal R may be supplied to the estimation model 51 as the input data D1. Also, a time series of samples constituting the audio signal X or the collected sound signal R may be supplied to the estimation model 51 as the input data D1. The output data D2 is data specifying either a performance sound or a speech sound. The estimation model 51 may output the output data D2 representing the probability that the near-end sound corresponds to a performance sound and the probability that the near-end sound corresponds to a speech sound.

[0047] The estimation model 51 is realized by a combination of a program that causes the control device 11 to execute a calculation to output output data D2 in response to input of input data D1, and a plurality of variables (e.g., weights and biases) that are applied to the calculation. The plurality of variables that define the estimation model 51 are set by supervised machine learning using a plurality of training data. Each of the plurality of training data is composed of a combination of known input data D1 and known output data D2. In the machine learning of the estimation model 51, the plurality of variables of the estimation model 51 are iteratively updated so that an error between output data D2 when input data D1 of each training data is input to the provisional estimation model 51 and the output data D2 of the training data is reduced. Therefore, the estimation model 51 outputs output data D2 that is statistically appropriate for unknown input data D1 under a tendency that exists between the input data D1 and the output data D2 in the plurality of training data.

[0048] The determination processing unit 50 generates output data D2 indicating whether the far-end sound represented by the sound signal X corresponds to a performance sound or a speech sound by supplying input data D1 corresponding to the sound signal X to the estimation model 51. The determination processing unit 50 also generates output data D2 indicating whether the near-end sound represented by the sound pickup signal R corresponds to a performance sound or a speech sound by supplying input data D1 corresponding to the sound pickup signal R to the estimation model 51.

[0049] The method for determining whether the near-end sound and the far-end sound include a performance sound or a speech sound is not limited to the above example. For example, the determination processing unit 50 may compare the feature of the pickup signal R with the feature of the performance sound and the feature of the speech sound, and determine that the performance sound and the speech sound, whichever has a feature similar to the near-end sound, is included in the near-end sound. Similarly, the determination processing unit 50 may compare the feature of the audio signal X with the feature of the performance sound and the feature of the speech sound, and determine that the performance sound and the speech sound, which has a feature similar to the far-end sound, is included in the far-end sound. In addition, in the configuration using the estimation model 51, the estimation model 51 is not limited to a deep neural network. For example, a statistical estimation model such as HMM (Hidden Markov Model) or SVM (Support Vector Machine) may be used as the estimation model 51.

[0050] The operation control unit 60 in Fig. 3 controls the updating of the processing parameters by the update processing unit 40. Specifically, the operation control unit 60 controls the operation of the update processing unit 40 according to the result of the determination by the determination processing unit 50. The operation control unit 60 in the first embodiment controls the continuation / stop of the operation of each element (setting unit 41, setting unit 42, and setting unit 43) of the update processing unit 40 to repeatedly update the processing parameters according to the result of the determination by the determination processing unit 50. The operation control unit 60 recognizes the result of the determination by the determination processing unit 50 by referring to the determination data stored in the storage device 12 for each of the near-end sound and the far-end sound.

[0051] 7 is an explanatory diagram of the operation of the operation control unit 60 in the first embodiment. Specifically, FIG. 7 illustrates an example of the relationship between the result of the determination by the determination processing unit 50 and the execution / stop of updating by the update processing unit 40.

[0052] In state A1 where both the near-end sound and the far-end sound are silent, the operation control unit 60 causes the setting unit 42 to update the noise spectrum Q. Also, in state A1, the operation control unit 60 stops the setting unit 41 from updating the coefficients Cn and the setting unit 43 from updating the gain G. The picked-up sound signal Ra in state A1 predominantly contains stationary environmental noise such as the operating sound of an air conditioning facility. Therefore, by updating the noise spectrum Q in state A1, it is possible to generate a noise spectrum Q that represents the actual noise with high accuracy.

[0053] In state A2 where the near-end sound is silent and the far-end sound includes a speech sound, the operation control unit 60 causes the setting unit 41 to update each coefficient Cn. Also, in state A2, the operation control unit 60 stops the setting unit 42 from updating the noise spectrum Q and the setting unit 43 from updating the gain G. Through the above processing, a pseudo echo signal E that closely approximates the feedback sound with high accuracy is generated.

[0054] In state A4 where the near-end sound includes a speech sound and the far-end sound is silent, the operation control unit 60 causes the setting unit 43 to update the gain G. Also, in state A4, the operation control unit 60 stops the setting unit 41 from updating each coefficient Cn and the setting unit 42 from updating the noise spectrum Q. Through the above processing, the gain G is updated to a value that appropriately adjusts the volume of the speech sound by the near-end user Ua.

[0055] In a state where one or both of the near-end sound and the far-end sound include a performance sound (states A3, A6-A9), and in a state A5 where both the near-end sound and the far-end sound include a speech sound, the operation control unit 60 stops the updating of the coefficients Cn by the setting unit 41, the updating of the noise spectrum Q by the setting unit 42, and the updating of the gain G by the setting unit 43. That is, the updating of all processing parameters is stopped. In a state where the updating of the processing parameters is stopped, sound processing is executed using processing parameters maintained at the values ​​after the previous update (i.e., the last update before the stop).

[0056] 8 is a flowchart illustrating a specific procedure of an operation (hereinafter referred to as a "control process") Sb in which the operation control unit 60 controls the update processing unit 40. For example, the control process Sb is started in response to an interrupt that occurs at a predetermined period.

[0057] When the control process Sb is started, the operation control unit 60 judges whether or not the state corresponds to state A1 in which both the near-end sound and the far-end sound are silent (Sb11). If the state corresponds to state A1 (Sb11: YES), the operation control unit 60 causes the setting unit 42 to update the noise spectrum Q, and stops the setting unit 41 from updating each coefficient Cn and the setting unit 43 from updating the gain G (Sb12).

[0058] If the state does not correspond to A1 (Sb11: NO), the operation control unit 60 judges whether the state corresponds to A2 in which the near-end sound is silent and the far-end sound includes a speech sound (Sb13). If the state corresponds to A2 (Sb13: YES), the operation control unit 60 causes the setting unit 41 to update each coefficient Cn, and stops the setting unit 42 from updating the noise spectrum Q and the setting unit 43 from updating the gain G (Sb14).

[0059] If the state does not correspond to A2 (Sb13: NO), the operation control unit 60 judges whether the state corresponds to A4 in which the near-end sound contains a speech sound and the far-end sound is silent (Sb15). If the state corresponds to A4 (Sb15: YES), the operation control unit 60 causes the setting unit 43 to update the gain G, and stops the setting unit 41 from updating the coefficients Cn and the setting unit 42 from updating the noise spectrum Q (Sb16).

[0060] If the state does not correspond to state A4, it means that one or both of the near-end sound and the far-end sound include a performance sound (states A3, A6-A9), or it means that the state is A5 in which both the near-end sound and the far-end sound include a speech sound. If the state does not correspond to state A4 (Sb15: NO), the operation control unit 60 stops the update of each coefficient Cn by the setting unit 41, the update of the noise spectrum Q by the setting unit 42, and the update of the gain G by the setting unit 43 (Sb17). In other words, if at least one of the near-end sound and the far-end sound includes a performance sound, the update of the processing parameters by the update processing unit 40 is stopped.

[0061] As described above, in the first embodiment, when at least one of the near-end sound and the far-end sound includes a performance sound, the update of the processing parameters is stopped. Therefore, it is possible to reduce the possibility that inappropriate acoustic processing is performed on the collected sound signal R (Ra, Rb, Rc). The effects of the first embodiment are described in detail below.

[0062] If the processing parameters fluctuate in a state in which the near-end sound includes a performance sound (state A7-A9), the acoustic characteristics of the performance sound will fluctuate, and there is a possibility that the performance expression (e.g., intonation) intentionally given to the performance sound by the user Ua will be diminished by the acoustic processing. In consideration of the above circumstances, in the first embodiment, the updating of the processing parameters is stopped in a state in which the near-end sound includes a performance sound (state A7-A9). With the above configuration, it is possible to reduce the possibility that the performance expression intended by the user Ua will be diminished by the acoustic processing. In other words, the performance expression intended by the user Ua can be accurately conveyed to the user Ub.

[0063] On the other hand, in a state where only the far-end sound includes a performance sound (where the near-end sound does not include a performance sound), even if the processing parameters are changed, the above-mentioned problem of the performance expression of the near-end sound being diminished does not occur. However, for the following reasons, in the first embodiment, even when only the far-end sound includes a performance sound, the update of the processing parameters is stopped.

[0064] Each coefficient Cn of the adaptive filter 311 is updated on the premise that the acoustic signal X and the collected sound signal Ra are not mutually correlated. Therefore, when the acoustic signal X and the collected sound signal Ra are correlated, it is difficult to appropriately update each Cn so that the pseudo echo signal E can be estimated with high accuracy. On the other hand, when both the near-end sound and the far-end sound include performance sounds, it is highly likely that the users Ua and Ub are playing a common piece of music in parallel (i.e., playing together). For example, a situation is assumed in which the users Ua and Ub play different performance parts that make up one piece of music, or a situation in which the users Ua and Ub play a common performance part of one piece of music. When the users Ua and Ub are playing a common piece of music, the far-end sound (the performance sound of the instrument 300b by the user Ub) and the near-end sound (the performance sound of the instrument 300a by the user Ua) are musically in harmony with each other, so that the acoustic signal X and the collected sound signal Ra are mutually correlated. From the above viewpoint, in a state where both the near-end sound and the far-end sound include performance sounds (state A9), the updating of the processing parameters (particularly the coefficients Cn) should be stopped.

[0065] In addition, in a state where the far-end sound is included in the performance sound, the performance sound is fed back from the sound emitting device 15 to the sound collecting device 14, resulting in the collected sound signal Ra including the performance sound of the far-end sound. Therefore, in a configuration where the determination processing unit 50 analyzes the acoustic signal X and the collected sound signal R, it is possible to determine with high accuracy that one or both of the near-end sound and the far-end sound include the performance sound, but it is difficult to determine with high accuracy a state where one of the near-end sound and the far-end sound includes the performance sound and the other does not include the performance sound (states A3, A6-A8). That is, it is actually difficult to specify with high accuracy which of the near-end sound and the far-end sound includes the performance sound (to distinguish state A3 and states A6-A8 from state A9). In consideration of the above circumstances, in the first embodiment, in addition to the case where both the near-end sound and the far-end sound include the performance sound (state A9), the update of the processing parameters is also stopped when only one of the near-end sound and the far-end sound includes the performance sound (states A3, A6-A8). According to the above configuration, the possibility that the processing parameters are updated to inappropriate values ​​due to the correlation between the near-end sound and the far-end sound is reduced.

[0066] B: Second embodiment A second embodiment will be described. Note that, in each of the following exemplary embodiments, for elements whose functions are similar to those of the first embodiment, the reference numerals used in the description of the first embodiment will be used and detailed descriptions of each will be omitted as appropriate.

[0067] 9 is a block diagram illustrating a functional configuration of a sound processing system 100a in the second embodiment. The sound collection device 14 in the second embodiment is a microphone array including a plurality (M pieces) of sound collection units 14_1 to 14_M. The M pieces of sound collection units 14_1 to 14_M are arranged in a line or a matrix at intervals from one another. The m-th (m=1 to M) sound collection unit 14_m is a microphone that generates a collected sound signal Ra_m by collecting surrounding sound. Specifically, each sound collection unit 14_m collects the performance sound of a musical instrument 300a or the speech sound of a user Ua as a near-end sound.

[0068] In the acoustic processing unit 30 of the second embodiment, the echo suppression unit 31 of the first embodiment is replaced with a beam forming unit 34. The configurations and operations of the noise suppression unit 32 and the volume adjustment unit 33 of the acoustic processing unit 30 are similar to those of the first embodiment. In addition, in the update processing unit 40 of the second embodiment, the setting unit 41 of the first embodiment is replaced with a setting unit 44. The configurations and operations of the setting units 42 and 43 of the update processing unit 40 are similar to those of the first embodiment.

[0069] 9 executes beamforming processing on M systems of collected sound signals Ra_1 to Qa_M generated by different sound collecting sections 14_m to generate a collected sound signal Rb. The beamforming processing is a filtering process to which a plurality of coefficients W are applied.

[0070] Specifically, the beam forming process includes signal processing for forming a sound collection beam directed in the direction from which the near-end sound arrives. The sound collection beam is a localized range with high sound collection sensitivity. That is, the beam forming unit 34 directs the sound collection beam in the direction of the musical instrument 300a or the user Ua to generate a sound collection signal Rb in which the performance sound of the musical instrument 300a or the speech sound by the user Ua is emphasized. The beam forming process of the second embodiment also includes signal processing for forming a sound collection blind spot in the direction from which the far-end sound arrives. The sound collection blind spot is a localized range with low sound collection sensitivity. Specifically, the beam forming unit 34 of the second embodiment directs the sound collection blind spot in the direction of the sound emitting device 15 to generate a sound collection signal Rb in which the feedback sound reaching the sound collecting device 14 from the sound emitting device 15 is suppressed.

[0071] The setting unit 44 of the update processing unit 40 updates multiple coefficients W applied to the beam forming process. Specifically, the setting unit 44 iteratively updates the multiple coefficients W so that the sound collection beam is directed in the direction from which the near-end sound arrives and the sound collection blind spot is directed in the direction from which the far-end sound arrives. The multiple coefficients W applied to the beam forming process are updated on the premise that the acoustic signal X and the sound collection signal Ra are not mutually correlated.

[0072] FIG. 10 is a block diagram illustrating a specific configuration of the setting unit 44. The setting unit 44 includes a first analysis unit 441, a second analysis unit 442, and a coefficient setting unit 443. The first analysis unit 441 estimates the direction θ1 from which the far-end sound arrives (i.e., the direction of the sound emitting device 15 that is the sound source of the far-end sound) by analyzing the acoustic signal X representing the far-end sound and the M-system sound pickup signals Ra_1 to Qa_M representing the near-end sound. The second analysis unit 442 estimates the direction θ2 from which the near-end sound arrives by analyzing the M-system sound pickup signals Ra_1 to Qa_M representing the near-end sound. The estimation of the direction θ1 and the direction θ2 is repeated. That is, the first analysis unit 441 iteratively updates the direction θ1, and the second analysis unit 442 iteratively updates the direction θ2. The coefficient setting unit 443 sets a plurality of coefficients W according to the direction θ1 estimated by the first analysis unit 441 and the direction θ2 estimated by the second analysis unit 442. That is, the coefficient setting unit 443 sets a plurality of coefficients W so that a sound collection blind spot is formed in the direction θ1 of the far-end sound and a sound collection beam is formed in the direction θ2 of the near-end sound.

[0073] Fig. 11 is an explanatory diagram of the operation of the operation control unit 60 in the second embodiment. Specifically, Fig. 11 illustrates an example of the relationship between the result of the determination by the determination processing unit 50 and the execution / stop of updating by the update processing unit 40.

[0074] In state B1 in which both the near-end sound and the far-end sound are silent, the operation control unit 60 causes the setting unit 42 to update the noise spectrum Q, as in the first embodiment. Also, in state B1, the operation control unit 60 stops the setting unit 44 from updating the coefficients W and the setting unit 43 from updating the gain G. Therefore, the noise spectrum Q representing stationary environmental noise, such as the operating sound of an air conditioning facility, is estimated with high accuracy.

[0075] In state B2 where the near-end sound is silent and the far-end sound includes a speech sound, the operation control unit 60 causes the first analysis unit 441 to update the direction θ1 from which the far-end sound arrives. Also, in state B2, the operation control unit 60 stops the second analysis unit 442 from updating the direction θ2, the setting unit 42 from updating the noise spectrum Q, and the setting unit 43 from updating the gain G. Through the above operations, the direction θ1 from which the far-end sound arrives is estimated with high accuracy. A plurality of coefficients W are also updated in conjunction with the update of the direction θ1.

[0076] In state B4, where the near-end sound includes a speech sound and the far-end sound is silent, the operation control unit 60 causes the second analysis unit 442 to update the direction θ2 and the setting unit 43 to update the gain G. Also, in state B4, the operation control unit 60 causes the setting unit 42 to stop updating the noise spectrum Q. Through the above operations, the direction θ2 from which the near-end sound arrives is estimated with high accuracy. In conjunction with the update of the direction θ2, multiple coefficients W are also updated. Also, the gain G is updated to a value that allows the volume of the speech sound by the near-end user Ua to be appropriately adjusted.

[0077] In a state where one or both of the near-end sound and the far-end sound include a performance sound (states B3, B6-B9) and in a state B5 where both of the near-end sound and the far-end sound include a speech sound, the operation control unit 60 stops the updating of the direction θ1 by the first analysis unit 441 and the updating of the direction θ2 by the second analysis unit 442. That is, the updating of the multiple coefficients W by the setting unit 44 is stopped. Furthermore, in the above states (states B3, B5-B9), the operation control unit 60 stops the updating of the noise spectrum Q by the setting unit 42 and the updating of the gain G by the setting unit 43. That is, the updating of all the processing parameters is stopped. In a state where the updating of the processing parameters is stopped, the sound processing is executed by applying the processing parameters maintained at the values ​​after the immediately preceding update.

[0078] 12 is a flowchart illustrating a specific procedure of the control process Sb in the second embodiment. For example, the control process Sb is started in response to an interrupt that occurs at a predetermined period.

[0079] When the control process Sb is started, the operation control unit 60 judges whether or not the state corresponds to state B1 in which both the near-end sound and the far-end sound are silent (Sb21). If the state corresponds to state B1 (Sb21: YES), the operation control unit 60 causes the setting unit 42 to execute the update of the noise spectrum Q, and stops the update of the direction θ1 by the first analysis unit 441, the update of the direction θ2 by the second analysis unit 442, and the update of the gain G by the setting unit 43 (Sb22).

[0080] If the state does not correspond to state B1 (Sb21: NO), the operation control unit 60 judges whether or not the state corresponds to state B2 in which the near-end sound is silent and the far-end sound includes a speech sound (Sb23). If the state corresponds to state B2 (Sb23: YES), the operation control unit 60 causes the first analysis unit 441 to execute updating of the direction θ1 from which the far-end sound arrives, and stops updating of the direction θ2 by the second analysis unit 442, updating of the noise spectrum Q by the setting unit 42, and updating of the gain G by the setting unit 43 (Sb24). The multiple coefficients W are updated in conjunction with the update of the direction θ1 by the first analysis unit 441.

[0081] If the state does not correspond to state B2 (Sb23: NO), the operation control unit 60 judges whether or not the state corresponds to state B4 in which the near-end sound contains a speech sound and the far-end sound is silent (Sb25). If the state corresponds to state B4 (Sb25: YES), the operation control unit 60 executes the update of the direction θ2 by the second analysis unit 442 and the update of the gain G by the setting unit 43, and stops the update of the direction θ1 by the first analysis unit 441 and the update of the noise spectrum Q by the setting unit 42 (Sb26). The multiple coefficients W are updated in conjunction with the update of the direction θ2 by the second analysis unit 442. In addition, the gain G is updated to a value that can appropriately adjust the volume of the speech sound by the near-end user Ua.

[0082] If the state does not correspond to state B4, it means that one or both of the near-end sound and the far-end sound include a performance sound (states B3, B6-B9), or state B5 where both the near-end sound and the far-end sound include a speech sound. If the state does not correspond to state B4 (Sb25: NO), the operation control unit 60 stops the update of the direction θ1 by the first analysis unit 441, the update of the direction θ2 by the second analysis unit 442, the update of the noise spectrum Q by the setting unit 42, and the update of the gain G by the setting unit 43 (Sb27). Therefore, the update of the multiple coefficients W is stopped. In other words, if at least one of the near-end sound and the far-end sound includes a performance sound, the update of the processing parameters by the update processing unit 40 is stopped.

[0083] As described above, in the second embodiment, when at least one of the near-end sound and the far-end sound includes a performance sound, the update of the processing parameters is stopped. Therefore, as in the first embodiment, the possibility that inappropriate acoustic processing is performed on the collected sound signal R (Ra, Rb, Rc) can be reduced.

[0084] C: Third embodiment 13 is a block diagram illustrating the configuration of the acoustic processing unit 30 in the third embodiment. The acoustic processing unit 30 in the third embodiment has a configuration in which a nonlinear processing unit 35 is added to the same elements as in the first embodiment (an echo suppression unit 31, a noise suppression unit 32, and a volume adjustment unit 33).

[0085] The nonlinear processing unit 35 generates a collected sound signal Rb2 by performing nonlinear processing on the collected sound signal Rb1 (the collected sound signal Rb in the first embodiment) after processing by the echo suppression unit 31. The nonlinear processing is a signal processing in which a frequency mask consisting of a plurality of gains corresponding to different frequency bands on the frequency axis is multiplied by the frequency spectrum of the collected sound signal Rb1. The frequency mask is iteratively updated according to the acoustic characteristics of the collected sound signal Rb1. Specifically, the frequency mask is a binary mask in which the gain of each frequency band in which feedback sound remains among the plurality of frequency bands is set to a first value (for example, 0), and the gain of each remaining frequency band is set to a second value (for example, 1) that exceeds the first value. As can be understood from the above description, the acoustic components of the feedback sound remaining in the collected sound signal Rb1 after the echo suppression processing are reduced by the nonlinear processing. The configurations and operations of the noise suppression unit 32 and the volume adjustment unit 33 are the same as those of the first embodiment. The order of the nonlinear processing, the noise suppression processing, and the volume adjustment processing is not limited to the example of FIG. 13 and can be changed arbitrarily.

[0086] Moreover, the control device 11 in the sound processing system 100a of the third embodiment executes a program stored in the storage device 12 to realize a delay measurement unit 55 in addition to the same elements as those in the first embodiment (the communication control unit 20, the playback processing unit 25, the sound processing unit 30, the update processing unit 40, the determination processing unit 50, and the operation control unit 60). The delay measurement unit 55 measures the communication delay L between the sound processing system 100a and the sound processing system 100b. The communication delay L is, for example, the time required for a signal transmitted from one of the sound processing system 100a and the sound processing system 100b to be received by the other. Any known technique may be adopted to measure the communication delay L.

[0087] The operation control unit 60 of the fourth embodiment controls the operation of the sound processing unit 30 according to the communication delay L. Specifically, the operation control unit 60 controls the response speed Z1 and the response speed Z2 according to the communication delay L. The response speed Z1 is an index of the speed at which the N coefficients C1 to CN applied to the adaptive filter 311 are linked to changes in the sound signal X and the collected sound signal R. Specifically, the higher the response speed Z1, the more the N coefficients C1 to CN are updated so as to follow changes in the sound characteristics of the sound signal X and the collected sound signal R more sensitively. On the other hand, the response speed Z2 is an index of the speed at which the frequency mask applied to the nonlinear processing is linked to changes in the collected sound signal Rb1. Specifically, the higher the response speed Z2, the more the frequency mask is updated so as to follow changes in the sound characteristics of the collected sound signal Rb1 more sensitively.

[0088] In a situation where the communication delay L is sufficiently small, the feedback sound reaching the sound collection device 14 from the sound emission device 15 does not pose a particular problem for the user Ub to hear the near-end sound. In consideration of the above circumstances, the operation control unit 60 decreases the response speed Z1 and the response speed Z2 as the communication delay L becomes smaller. In other words, in a situation where the communication delay L is small, the change over time of the N coefficients C1 to CN and the frequency mask is suppressed. Specifically, the change of each coefficient Cn and the frequency mask in response to the change of the acoustic characteristics of the acoustic signal X or the collected sound signal Rb1 is suppressed.

[0089] On the other hand, feedback noise tends to become evident in a situation where the communication delay L is large. In consideration of the above circumstances, the operation control unit 60 increases the response speed Z1 and the response speed Z2 as the communication delay L increases. In other words, in a situation where the communication delay L is large, the coefficients Cn and the frequency mask change sensitively and quickly in response to changes in the acoustic characteristics of the sound signal X or the picked-up sound signal Rb1.

[0090] As described above, in the third embodiment, the response speed Z1 of the N coefficients C1 to CN applied to the adaptive filter 311 is controlled in response to the communication delay L. Therefore, appropriate echo suppression processing can be performed on the collected sound signal Ra to reduce the feedback sound collected by the sound collection device 14. Also, in the third embodiment, the response speed Z2 of the frequency mask applied to the nonlinear processing is controlled in response to the communication delay L. Therefore, appropriate nonlinear processing can be performed on the collected sound signal Rb1 to reduce the feedback sound collected by the sound collection device 14.

[0091] In addition, in FIG. 13, an example is shown of a configuration based on the configuration of the first embodiment having an echo suppression unit 31, but the configuration of the third embodiment can also be applied to the configuration of the second embodiment having a beam forming unit 34.

[0092] D: Variation Specific modified embodiments added to each of the above-mentioned embodiments are exemplified below. Two or more embodiments selected from the following examples may be appropriately combined as long as they are not mutually contradictory.

[0093] (1) The specific configuration of the acoustic processing unit 30 is not limited to the configurations exemplified in the above-mentioned embodiments. For example, some of the elements (echo suppression unit 31, noise suppression unit 32, volume adjustment unit 33, beam formation unit 34, and nonlinear processing unit 35) included in the acoustic processing unit 30 in the above-mentioned embodiments may be omitted.

[0094] (2) In each of the above-mentioned embodiments, the update of the processing parameters is stopped when at least one of the near-end sound and the far-end sound includes a performance sound, but it is not essential to stop the update of the processing parameters. For example, the operation control unit 60 may control the update speed of the processing parameters (hereinafter referred to as the "update speed") depending on the presence or absence of a performance sound. The update speed is an index related to the speed at which the processing parameters are updated, and specifically includes the update frequency and the update rate. The update frequency means the number of times the processing parameters are updated within a unit time. An example of the update frequency is also referred to as the update period of the processing parameters. Note that each of the above-mentioned embodiments may also be expressed as a configuration in which the update frequency of the processing parameters is set to zero when at least one of the near-end sound and the far-end sound includes a performance sound.

[0095] On the other hand, the update ratio is an index of the degree to which the numerical value of the processing parameter changes with each update by the update processing unit 40. For example, assume that the update processing unit 40 calculates the updated processing parameter Pnext by the calculation (i.e., exponential moving average) of the following formula (1) using the latest numerical value Pnew of the processing parameter and a past (e.g. immediately preceding) processing parameter Pold. The symbol α is a predetermined coefficient and is set to a non-negative value equal to or less than 1 (0≦α≦1). Pnext=(1-α)·Pold+α·Pnew (1) The larger the coefficient α, the greater the influence of the latest numerical value Pnew on the updated processing parameter Pnext, and the smaller the coefficient α, the greater the influence of the past processing parameter Pold on the updated processing parameter Pnext. In other words, the larger the coefficient α, the more sensitive the updated processing parameter Pnext is to changes in the acoustic signal X or the picked-up sound signal R (Ra to Rc). As can be understood from the above explanation, the coefficient α is an index that represents the update ratio of the processing parameter Pnext (i.e., the degree of change in the processing parameter with respect to changes in the acoustic signal X or the picked-up sound signal R).

[0096] The operation control unit 60 makes the update speed different between when at least one of the near-end sound and the far-end sound includes a performance sound and when neither the near-end sound nor the far-end sound includes a performance sound. Specifically, when at least one of the near-end sound and the far-end sound includes a performance sound, the operation control unit 60 reduces the update speed of the processing parameters compared to when neither of the near-end sound and the far-end sound includes a performance sound. For example, the operation control unit 60 sets the update frequency when at least one of the near-end sound and the far-end sound includes a performance sound to a numerical value smaller than the update frequency when neither of the near-end sound and the far-end sound includes a performance sound. In addition, the operation control unit 60 sets the update ratio (for example, coefficient α) when at least one of the near-end sound and the far-end sound includes a performance sound to a numerical value smaller than the update ratio when neither of the near-end sound and the far-end sound includes a performance sound. Even with the above configuration, the expected effect of being able to appropriately control the processing parameters applied to the sound processing is realized compared to a configuration in which the processing parameters are updated without distinguishing whether or not at least one of the near-end sound and the far-end sound includes a performance sound. In the above description, the update speed when at least one of the near-end sound and the far-end sound includes a performance sound is lower than the update speed when no performance sound is included. However, a configuration in which the update speed when at least one of the near-end sound and the far-end sound includes a performance sound exceeds the update speed when no performance sound is included is also assumed.

[0097] (3) In the first embodiment, the acoustic processing unit 30 including the echo suppression unit 31 is illustrated, and in the second embodiment, the acoustic processing unit 30 including the beam formation unit 34 is illustrated, but a configuration in which the acoustic processing unit 30 includes both the echo suppression unit 31 and the beam formation unit 34 is also assumed. For example, an echo suppression unit 31 is provided for each of the M sound collection units 14_1 to 14_M. The beam formation unit 34 generates a picked-up sound signal Rb from M systems of picked-up sound signals Ra_1 to Ra_M generated by the different echo suppression units 31.

[0098] (4) In the above-mentioned embodiments, the sound processing system 100a of the user Ua communicates with the sound processing system 100b of the user Ub. However, the above-mentioned embodiments are also applicable to a situation where the sound processing system 100a communicates with a plurality of sound processing systems 100b. For example, a situation is assumed where one user Ua who is an instructor instructs a plurality of users Ub. In the above-mentioned situation, the sound signal X representing a mixture of performance sounds or speech sounds produced by the plurality of users Ub is received by the communication device 13 of the sound processing system 100a. In the above-mentioned configuration, as in the above-mentioned embodiments, when both the far-end sound of the sound signal X and the near-end sound of the picked-up signal Ra include performance sounds, the update of the processing parameters applied to the sound processing is stopped.

[0099] (5) As described above, the functions of the sound processing system 100a exemplified above are realized by the cooperation of one or more processors constituting the control device 11 and the program stored in the storage device 12. The program according to the present disclosure can be provided in a form stored in a computer-readable recording medium and installed in a computer. The recording medium is, for example, a non-transitory recording medium, and a good example is an optical recording medium (optical disk) such as a CD-ROM, but also includes any known type of recording medium such as a semiconductor recording medium or a magnetic recording medium. Note that the non-transitory recording medium includes any recording medium except for a transient propagating signal, and does not exclude volatile recording media. In addition, in a configuration in which a distribution device distributes a program via a communication network, a storage device that stores the program in the distribution device corresponds to the non-transitory recording medium described above.

[0100] E: Notes From the above-described exemplary embodiments, the following configurations can be understood, for example.

[0101] A sound processing method according to one aspect (aspect 1) of the present disclosure includes receiving a first sound signal representing a far-end sound produced by a first user from a far-end device, emitting the far-end sound represented by the first sound signal by a sound emitting device, performing sound processing to which a processing parameter is applied to a sound collection signal generated by a sound collection device by collecting sound including a near-end sound produced by a second user at the near end, generating a second sound signal, transmitting the second sound signal to the far-end device, updating the processing parameter according to the first sound signal or the collected sound signal, and stopping the update of the processing parameter when at least one of the near-end sound and the far-end sound includes a performance sound. In the above aspect, the update of the processing parameter is stopped when at least one of the near-end sound and the far-end sound includes a performance sound. Therefore, it is possible to reduce the possibility that inappropriate sound processing is performed on the collected sound signal.

[0102] "Near-end sound" is the sound to be transmitted to the far-end device, and includes speech sounds or performance sounds uttered by the user. Speech sounds are sounds that express language. A typical example of speech sounds is, for example, the sounds of a conversation with another user, but speech sounds also include sounds that are uttered unilaterally without forming part of a conversation (for example, instructional sounds in music lessons). Performance sounds refer to sounds that express music. A typical example of performance sounds is, for example, instrument sounds produced from an instrument played by a user, but singing sounds produced by a user singing are also included in the concept of performance sounds in the sense of musical sounds. In other words, in this specification, "performance" includes not only playing an instrument (playing in the narrow sense), but also singing of a piece of music.

[0103] The case where the near-end sound "includes a performance sound" includes a case where the near-end sound includes only a performance sound (a case where the near-end sound does not include a speech sound), and a case where the near-end sound includes both a speech sound and a performance sound but the volume of the performance sound exceeds the volume of the speech sound. The same applies to the far-end sound. In other words, the case where the far-end sound "includes a performance sound" includes a case where the far-end sound includes only a performance sound (a case where the far-end sound does not include a speech sound), and a case where the far-end sound includes both a speech sound and a performance sound but the volume of the performance sound exceeds the volume of the speech sound.

[0104] A sound processing method according to another aspect (aspect 2) of the present disclosure includes receiving a first sound signal representing a far-end sound produced by a first user from a far-end device, emitting the far-end sound represented by the first sound signal by a sound emitting device, and generating a second sound signal by performing sound processing to which a processing parameter is applied on a sound collection signal generated by a sound collection device by collecting sound including a near-end sound produced by a second user at the near end, transmitting the second sound signal to the far-end device, updating the processing parameter according to the first sound signal or the collected sound signal, and controlling the update of the processing parameter so that the update speed of the processing parameter when at least one of the near-end sound and the far-end sound includes a performance sound is different from the update speed of the processing parameter when the performance sound is not included. In the above aspect, the update speed of the processing parameter differs between when at least one of the near-end sound and the far-end sound includes a performance sound and when the performance sound is not included. Therefore, it is possible to reduce the possibility that inappropriate sound processing is performed on the collected sound signal.

[0105] The update rate refers to the rate at which the value of a processing parameter changes due to an update. For example, the update frequency and the update rate are included in the concept of update rate. The update frequency refers to the number of updates of a processing parameter within a unit time. On the other hand, the update rate refers to the degree to which a processing parameter changes with each update.

[0106] In a specific example (aspect 3) of aspect 2, the update speed is an update frequency that is the number of updates within a unit time, and in controlling the update of the processing parameters, when at least one of the near-end sound and the far-end sound includes a performance sound, the update frequency of the processing parameters is reduced compared to when the performance sound is not included. With the above configuration, it is possible to reduce the possibility that inappropriate acoustic processing is performed on the collected sound signal.

[0107] In a specific example (aspect 4) of aspect 2, the update speed is an update rate that is a degree at which the processing parameter changes for each update, and in controlling the update of the processing parameter, when at least one of the near-end sound and the far-end sound includes a performance sound, the update rate of the processing parameter is reduced compared to when the performance sound is not included. With the above configuration, it is possible to reduce the possibility that inappropriate acoustic processing is performed on the collected sound signal.

[0108] In a specific example (Aspect 5) of any one of Aspects 1 to 4, the acoustic processing includes an echo suppression process for suppressing, from the collected sound signal, a pseudo echo signal that approximates a feedback sound that reaches the sound collection device from the sound emitting device. According to the above aspect, the second acoustic signal in which the influence of the feedback sound that reaches the sound collection device from the sound emitting device is reduced can be transmitted to the far-end device.

[0109] In a specific example (aspect 6) of aspect 5, the echo suppression process includes an adaptive filter process that generates the pseudo echo signal from the first acoustic signal, and a subtraction process that subtracts the pseudo echo signal from the picked-up sound signal, and the processing parameters include a plurality of coefficients that are applied to the adaptive filter process. In the above aspect, when at least one of the near-end sound and the far-end sound includes a performance sound, updating of the plurality of coefficients that are applied to the adaptive filter process is stopped. Therefore, the feedback sound included in the picked-up sound signal can be appropriately suppressed.

[0110] In a specific example (aspect 7) of any one of aspects 1 to 6, the acoustic processing includes a beam forming process for forming a sound collection beam directed in a direction from which the near-end sound arrives, and the processing parameters include a plurality of coefficients applied to the sound collection beam formation. In the above aspects, when at least one of the near-end sound and the far-end sound includes a performance sound, updating of the plurality of coefficients applied to the beam forming process is stopped. Therefore, feedback sound included in the collected sound signal can be appropriately suppressed.

[0111] In a specific example (Aspect 8) of any one of Aspects 1 to 7, the acoustic processing includes a volume adjustment process for amplifying the picked-up signal by a gain corresponding to the volume of the picked-up signal, and the processing parameters include the gain. In the above aspects, when at least one of the near-end sound and the far-end sound includes a performance sound, updating of the gain applied to the volume adjustment process is stopped. Therefore, the volume of the picked-up signal can be appropriately adjusted.

[0112] In a specific example (aspect 9) of any one of aspects 1 to 8, the acoustic processing includes a noise suppression process for suppressing a noise component included in the picked-up signal, and the processing parameters include a parameter representing the noise component. In the above aspects, when at least one of the near-end sound and the far-end sound includes a performance sound, updating of the parameter representing the noise component to be suppressed from the picked-up signal in the noise suppression process is stopped. Therefore, the noise component of the picked-up signal can be appropriately suppressed.

[0113] An acoustic processing system according to one aspect (aspect 10) of the present disclosure is an acoustic processing system that receives a first acoustic signal representing a far-end sound produced by a first user from a far-end device and emits the far-end sound represented by the first acoustic signal by an audio output device, and includes an acoustic processing unit that generates a second acoustic signal by applying acoustic processing parameters to a picked-up signal generated by the sound input device by collecting sound including a near-end sound produced by a second user at the near-end, a communication control unit that transmits the second acoustic signal to the far-end device, an update processing unit that updates the processing parameters in accordance with the first acoustic signal or the picked-up signal, and an operation control unit that stops updating the processing parameters when at least one of the near-end sound and the far-end sound includes a performance sound.

[0114] An acoustic processing system according to another aspect (aspect 11) of the present disclosure is an acoustic processing system that receives a first acoustic signal representing a far-end sound produced by a first user from a far-end device and emits the far-end sound represented by the first acoustic signal by an acoustic emission device, and includes an acoustic processing unit that generates a second acoustic signal by applying acoustic processing parameters to a picked-up signal generated by the sound pickup device by collecting sound including a near-end sound produced by a second user at the near-end, a communication control unit that transmits the second acoustic signal to the far-end device, an update processing unit that updates the processing parameters in accordance with the first acoustic signal or the picked-up signal, and an operation control unit that controls the update of the processing parameters so that the update speed of the processing parameter when at least one of the near-end sound and the far-end sound includes a performance sound is different from the update speed of the processing parameter when the sound does not include the performance sound. [Explanation of symbols]

[0115] 1...communication system, 100a, 100b...acoustic processing system, 200...communication network, 300a, 300b...musical instrument, 11...control device, 12...storage device, 13...communication device, 14...sound collection device, 15...sound emission device, 20...communication control unit, 25...playback processing unit, 30...acoustic processing unit, 31...echo suppression unit, 311...adaptive filter, 312...subtraction processing unit, 32...noise suppression unit, 33...volume adjustment unit, 34...beam forming unit, 35...nonlinear processing unit, 40...update processing unit, 41-44...setting unit, 50...judgment processing unit, 55...delay measurement unit, 60...operation control unit.

Claims

1. receiving a first acoustic signal from a far-end device representing a far-end sound produced by a first user; The far-end sound represented by the first acoustic signal is emitted by a sound emitting device; generating a second acoustic signal by performing acoustic processing using a processing parameter on a collected signal generated by a sound collection device by collecting sound including a near-end sound uttered by a second user at the near-end; transmitting the second acoustic signal to the far-end device; The processing parameters are controlled so that the processing parameters when at least one of the near-end sound and the far-end sound includes a performance sound are different from the processing parameters when the near-end sound and the far-end sound does not include the performance sound. A computer-implemented acoustic processing method.

2. The acoustic processing includes an echo suppression process for suppressing, from the collected sound signal, a pseudo echo signal that approximates a feedback sound that reaches the sound collecting device from the sound emitting device. The acoustic processing method according to claim 1.

3. The echo suppression process includes: an adaptive filter process for generating the pseudo echo signal from the first acoustic signal; a subtraction process of subtracting the pseudo echo signal from the picked-up sound signal; Including, The processing parameters include a number of coefficients that are applied to the adaptive filter processing. The acoustic processing method according to claim 2.

4. The acoustic processing includes a beam forming process for forming a sound collection beam directed in a direction from which the near-end sound arrives, The processing parameters include a number of coefficients that are applied to the sound pickup beamforming. The acoustic processing method according to claim 1.

5. the acoustic processing includes a volume adjustment process of amplifying the picked-up signal by a gain corresponding to the volume of the picked-up signal, The processing parameters include the gain. The acoustic processing method according to claim 1.

6. The acoustic processing includes a noise suppression process for suppressing a noise component included in the picked-up sound signal, The processing parameters include parameters representing the noise components. The acoustic processing method according to claim 1.

7. A sound processing system that receives a first sound signal representing a far-end sound produced by a first user from a far-end device, and emits the far-end sound represented by the first sound signal from a sound emitting device, an acoustic processing unit that generates a second acoustic signal by performing acoustic processing using a processing parameter on a collected signal generated by a sound collecting device by collecting a near-end sound produced by a second user at the near end; a communication control unit that transmits the second acoustic signal to the far-end device; an operation control unit that controls the processing parameters such that the processing parameters when at least one of the near-end sound and the far-end sound includes a performance sound are different from the processing parameters when the near-end sound and the far-end sound does not include the performance sound; An audio processing system comprising:

Images