Hearing assistance device and system, and sound source localization device

Abstract

To clearly hear an object voice distinctively from other voices, and to easy achieve that.SOLUTION: A hearing support device 100 includes: a sound environment intelligent map creation part 176 which uses output of an LRF and output from a microphone array to specify positions of sound sources; a selective sound source separation part 190 which separates voices from the sound sources from each other and outputs them in the form of voice signals based upon a voice signal from a smart phone, a voice signal from the microphone array, and the positions of the sound sources; and a user oriented voice reproduction part 154 which reconstitutes a voice signal from a specific sound source selectively based upon a face attitude of a user, the positions of the sound sources, and the voice signal separated for each of the sound sources.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a technique for assisting a user's hearing by using sound source localization and sound source separation techniques. [Background technology]

[0002] It is said that around 10% to 20% of the population in each country has hearing loss or hearing impairment. In a 2009 report titled "Study on the state of the hearing aid supply system" by the Japan Hearing Aid Retailers Association, it was reported that the hearing-impaired population in Japan is 15.7% (19.44 million people). The proportion is 25% to 40% for those aged 65 and over, and 40% to 66% for those aged 75 and over. Hearing loss in the elderly is presbycusis, a phenomenon caused by the aging of nerve cells, etc. In the future, the number of people with hearing loss is expected to increase further as the Japanese population ages.

[0003] However, only about 4 million people in Japan use hearing aids, meaning that only one in five people with hearing loss use them. Many people with hearing loss also stop using their hearing aids halfway through. In many cases, the reason users stop using hearing aids is because they have not chosen a hearing aid that suits them, or because the settings are difficult to adjust and they are being used incorrectly. However, even if these are appropriate, there is a limit to the comfort (ease of hearing) that a hearing aid can provide on its own.

[0004] One of the reasons why hearing aids are not widely used is that they have a fundamental problem: the microphone embedded in them amplifies surrounding noise. This microphone also causes feedback (high-pitched noise) which can be distressing to users. Some hearing aids have signal processing to prevent feedback, but this means that the volume of the hearing aid must be reduced, and the volume cannot be adequate for people with severe hearing loss. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP2017-092732A Summary of the Invention [Problem to be solved by the invention]

[0006] Conventional hearing aids and other hearing support devices have the above problems as well as other problems. For example, consider a situation where multiple people are talking in a place with multiple people. Normally, even in such a situation, humans can select only the target sound and distinguish it from the surrounding sounds. However, there is a problem that it is difficult to selectively hear sounds in this way just by using conventional hearing aids.

[0007] One proposal for solving these problems is disclosed in the above-mentioned Patent Document 1. The entire disclosure of Patent Document 1 is incorporated herein by reference.

[0008] 20, a hearing assistance system 900 disclosed in Patent Document 1 includes: a microphone array group 910; an LRF (Laser Rangefinder) group 912 for measuring the distance to a target (speaker); a sound source localization device 914 for determining the position of a sound source (speaker) present in the vicinity based on the output of the microphone array group 910 and the output of the LRF group 912 and outputting the information; a headset 920 having a head posture sensor 918 worn by a user 916; a spatial sense synthesis unit 924 for synthesizing an audio signal for the sound of a sound source (speaker) in a direction in which the user is paying attention based on the output of the sound source localization device 914 and the output of the head posture sensor 918 so that the sound signal is heard by the user 916 as a sound coming from that direction in three-dimensional space, and outputting the synthesized audio signal to the headset 920; and a display unit 922 for displaying the position of each sound source two-dimensionally, and for responding to a user's input specifying a sound source whose audio should be reconstructed and a sound source whose audio should be suppressed from among the sound sources, and for providing the information to the spatial sense synthesis unit 924.

[0009] The spatial sensation synthesis unit 924 includes a face pose estimation unit 954 that estimates the direction of the face of the user 916 based on the output of the head pose sensor 918, a database 952 that stores head-related transfer functions in various situations, a volume control unit 950 that corrects the amplitude of the audio signal (sound source signal) from each sound source output from the sound source localization device 914 based on the distance between the microphone array in the microphone array group 910 and the sound source, and a sound space reconstruction unit 956 that selects and applies appropriate head-related transfer functions for the left and right ears of the user 916 from the database 952 based on the direction of the face of the user 916 and the angle between the sound sources estimated by the face pose estimation unit 954 to the sound source signal whose volume is controlled by the volume control unit 950, thereby generating an audio signal that reconstructs an audio space according to the direction of the face of the user 916.

[0010] At this time, the respective sounds are adjusted according to the direction of the user's 916 face so that sounds from sound sources within a predetermined range in front of the face are emphasized and sounds from sound sources outside the predetermined range are attenuated. Therefore, the user can hear the sound from the sound source he or she is paying attention to in a more emphasized form than other sounds. Since the sound source to be emphasized can be identified based on the user's facial posture, the user can hear the voice of the speaker he or she is paying attention to in a natural way.

[0011] However, the hearing assistance system disclosed in Patent Document 1 requires multiple microphone arrays and multiple LRFs. For example, when a user goes out or attends a meeting in a room that does not have the necessary equipment, it is not realistic for the user to carry all of these with him or her. Therefore, there is a demand for a hearing assistance system that can more easily achieve the effects of the hearing assistance system disclosed in Patent Document 1.

[0012] Therefore, an object of the present invention is to provide a hearing assistance device and system, a sound source localization device, an input device for a hearing assistance system, a computer program, and a microphone array with integrated distance detection device, which enable a target sound to be distinguished from other sounds and be clearly heard, and which can be easily implemented. [Means for solving the problem]

[0013] According to a first aspect of the present invention, a hearing assistance device for assisting a user's hearing includes: a sound source localization means for identifying and outputting the position of a sound source using the output of a position detection device that detects the position of an object and the output from a microphone array; a sound source separation means for separating sounds from the sound sources from each other and outputting them as audio signals based on an audio signal from a terminal capable of transmitting an audio signal representing surrounding audio, an audio signal from the microphone array, and the position of the sound source identified by the sound source localization means; and an audio signal reconstruction means for selectively reconstructing an audio signal from a specific sound source based on the user's facial posture, the position of the sound source identified by the sound source localization means, and the audio signal separated and output for each sound source by the sound source separation means.

[0014] Preferably, the hearing assistance device further includes an anti-target exclusion means for excluding the audio signal from the terminal from being subject to reconstruction of the audio signal by the audio signal reconstruction means in response to the audio signal from the terminal being marked with a predetermined anti-target flag.

[0015] More preferably, the hearing assistance device further includes a terminal management means for managing terminals, a terminal designation means for designating any terminal among the terminals managed by the terminal management means as a target or an anti-target, and a target selection / exclusion means for selecting or excluding, in accordance with the designation, an audio signal from the terminal designated by the terminal designation means as a target for audio signal reconstruction by the audio signal reconstruction means.

[0016] More preferably, the terminal designation means includes a symbol display means for displaying on a display device a symbol representing a terminal managed by the terminal management means and a symbol representing a sound source separated by the sound source separation means as a two-dimensional display indicating the position of each symbol, a target / anti-target input means for receiving user input regarding the setting of one of the terminal symbols displayed on the display device as a target or anti-target by selecting it through dialogue with the user, and a target flag storage means for storing a target flag representing the designation for a terminal set as a target or anti-target by the target / anti-target input means among the terminals managed by the terminal management means.

[0017] Preferably, the terminal management means includes a position storage means for determining and storing an initial position of a terminal in response to a participation request from the terminal requesting participation in hearing support by the hearing support device, and a position update means for updating the position of the specific terminal stored in the position storage means based on the display position of a specific symbol representing the specific terminal displayed on the display device in response to a user input that changes the display position on the display device of the specific symbol.

[0018] More preferably, the terminal management means further includes an identification information transmitting means for transmitting identification information identifying the hearing support device to the new terminal in response to a request from the new terminal for information regarding hearing support that can be participated in, and a record generating means for generating a new record in the position storage means for storing information regarding the new terminal in response to a participation request from the new terminal specifying the identification information in response to the identification information.

[0019] More preferably, the terminal is a communication terminal having the function of performing wireless communication according to a specified communication standard, the hearing assistance device further includes a wireless communication device that performs wireless communication in accordance with the specified communication standard, and the sound source separation means determines the position of the sound source using an audio signal received by the wireless communication device from the terminal.

[0020] Preferably, the communication terminal is a smartphone.

[0021] A hearing assistance system according to a second aspect of the present invention is a hearing assistance system for assisting a user's hearing, and includes: a position detection device that detects the position of an object; at least one microphone array; sound source localization means that identifies and outputs the position of a sound source using an output of the position detection device and an output from the microphone array; sound source separation means that separates sounds from each sound source from each other and outputs them as audio signals based on an audio signal from a terminal capable of transmitting an audio signal representing surrounding sounds, an audio signal from the microphone array, and the position of the sound source identified by the sound source localization means; and audio signal reconstruction means that selectively reconstructs an audio signal from a specific sound source based on the user's facial posture, the position of the sound source identified by the sound source localization means, and the audio signal separated and output for each sound source by the sound source separation means.

[0022] A sound source localization device according to a third aspect of the present invention is a sound source localization device used in a hearing assistance system for assisting a user's hearing, and includes: a sound source localization means for identifying and outputting the position of a sound source using an output from a position detection device that detects the position of an object and an output from a microphone array; a sound source separation means for separating sounds from each sound source from each other and outputting them as audio signals based on an audio signal from a terminal capable of transmitting an audio signal representing surrounding sounds, an audio signal from the microphone array, and the position of the sound source identified by the sound source localization means; and an output means for outputting sound source position information indicating the position of the sound source identified by the sound source localization means and the audio signals separated for each sound source by the sound source separation means in association with each other.

[0023] An input device according to a fourth aspect of the present invention is an input device used in a hearing assistance system for assisting a user's hearing, and includes: terminal management means for managing at least terminals used in the hearing assistance system; symbol display means for displaying on a display device symbols representing terminals managed by the terminal management means and symbols representing sound sources within the hearing assistance system as two-dimensional displays indicating the position of each terminal; target / anti-target input means for receiving user input regarding settings for selecting one of the terminal symbols displayed on the display device through dialogue with the user and making it a target or anti-target; and setting output means for outputting a terminal set by the target / anti-target input means in association with the setting for that terminal.

[0024] Preferably, the input device further includes a location updating means for updating, for each of the terminals managed by the terminal management means, the location of the terminal through interactive processing with a user using the display device.

[0025] A computer program according to a fifth aspect of the present invention is a computer program for causing a communication terminal including a microphone, a wireless communication device, and a computer connected to the microphone and the wireless communication device to function as an audio input device in a hearing assistance system for assisting a user's hearing, and causes the computer to function as: a setting input means for setting a user input regarding whether or not to set the communication terminal as an anti-target in the hearing assistance system; a transmitting means for transmitting an audio signal input from the microphone to a sound source localization device in the hearing assistance system via the wireless communication device; and a labeling means for labeling the audio signal transmitted by the transmitting means with an anti-target flag in accordance with the setting input by the setting input means.

[0026] A microphone array integrated with a distance detection device according to a sixth aspect of the present invention includes a housing, a microphone array provided in the housing, a distance detection device provided in the housing for detecting the distance to an object present around the housing, and a wireless communication device for transmitting an output of the microphone array and an output of the distance detection device to the outside.

[0027] Preferably, the distance detection device includes a plurality of laser range finders, each of which is provided in a housing and detects the distance to an object present within a predetermined angular range around the housing.

[0028] More preferably, the plurality of laser range finders are arranged so that the plurality of laser range finders can detect distances within a full angular range around the housing.

[0029] The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the invention taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0030] [Figure 1] FIG. 1 is a schematic diagram for explaining a state in which the first embodiment is used. [Diagram 2] FIG. 2 is a schematic diagram for explaining a usage state of the first embodiment. [Diagram 3] FIG. 3 is a conceptual diagram showing a schematic overall configuration of the hearing support system according to the first embodiment. [Figure 4] FIG. 4 is a block diagram showing the configuration of the hearing assistance system. [Diagram 5] FIG. 5 is a diagram showing an example of a screen display of a computer that realizes an operation I / F (interface). [Figure 6] FIG. 6 is a block diagram of the user-specific sound source separation unit of the first embodiment. [Figure 7] FIG. 7 is a block diagram of the microphone array shown in FIG. [Figure 8]FIG. 8 is a block diagram of the smartphone microphone shown in FIG. [Figure 9] FIG. 9 is a block diagram of a gain normalization unit for normalizing the gain of the output of the microphone array shown in FIG. [Figure 10] FIG. 10 is a block diagram showing a more detailed configuration of the selective volume control unit and the audio signal generation unit in the user-oriented audio reproduction unit shown in FIG. [Figure 11] FIG. 11 is a graph showing the relationship between the gain determined by the selective volume control unit shown in FIG. 4 for the audio signal from each sound source based on the user's head posture estimated by the user's head movement tracking unit, and the angle of each sound source as seen by the user. [Figure 12] FIG. 12 is a diagram illustrating an example of a hardware configuration for implementing the first embodiment. [Figure 13] FIG. 13 is a block diagram showing the hardware configuration of the notebook computer shown in FIG. [Figure 14] FIG. 14 is a block diagram showing a hardware configuration of the tablet computer shown in FIG. [Figure 15] FIG. 15 is a block diagram showing a combination of functions required for the smartphone shown in FIG. 12 to participate in the hearing support system according to the first embodiment. [Figure 16] FIG. 16 is a block diagram showing a combination of functions that the tablet computer shown in FIG. 12 has in order to be used in the hearing support system according to the first embodiment. [Figure 17] FIG. 17 is a flowchart showing a control structure of a program executed by the notebook computer shown in FIG. 12 to process audio from a smartphone. [Figure 18] FIG. 18 is a block diagram of a user-specific sound source separation unit according to the second embodiment. [Figure 19] FIG. 19 is a sequence diagram showing a sequence of communication performed between a smartphone and each hearing support device when the smartphone participates in the hearing support system in the first and second embodiments. [Figure 20] FIG. 20 is a schematic block diagram of a conventional hearing assistance system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] In the following description and drawings, the same parts are given the same reference numbers, and therefore detailed descriptions thereof will not be repeated.

[0032] 1. First embodiment 1 Configuration (1) Schematic configuration With reference to FIG. 1, it is assumed that there are a plurality of speakers 54, ..., speaker 64 in a room. Among these speakers, speaker 54 is conversing with speaker 56, speaker 58 with speaker 60, and speaker 62 with speaker 64. In this situation, for example, a case where a hearing-impaired user 50 hears the contents of these conversations is considered. In this example, an LRF-integrated microphone array 52 is used, in which one microphone array in which 16 MEMS (Micro-Electro-Mechanical Systems) microphones are discretely arranged in a hemisphere and two LRFs are integrated, and this LRF-integrated microphone array 52 is placed, for example, in the center of a table in front of the speaker. Since the LRF-integrated microphone array 52 is an integrated type, this LRF-integrated microphone array 52 can be easily carried. In addition, the housing of the LRF-integrated microphone array 52 has the above-mentioned two LRFs provided back to back. Each of these LRFs can measure the distance to an object within a range of 180 degrees in front of it. As a result, the LRF integrated microphone array 52 can measure the distance from the LRF integrated microphone array 52 to the target over its entire circumference. In this example, the MEMS microphones are provided at the center of 16 leaves arranged in a hemisphere to resemble a houseplant.

[0033] In this embodiment, a user 50 wears earphones with built-in microphones in both ears. These earphones are configured to also function as hearing aids under normal circumstances. Here, the earphones are also equipped with a sensor for detecting the head posture of the user 50, and transmit that information to a hearing support device, which is the core of the hearing support system according to this embodiment. Information regarding the head posture is used to estimate the facial posture of the user 50.

[0034] 1, when the user 50 faces the speakers 54 and 56, the hearing support device determines that the user 50 faces the directions of the speakers 54 and 56 based on the output of the head posture sensor. Then, audio signals representing the voices of the speakers 54 and 56 are reconstructed and transmitted to the earphones of the user 50 so that the user 50 can perceive only the voices of the speakers 54 and 56 from among the voices of the speakers 54, ..., speaker 64 as voices from the speakers 54 and 56 in a three-dimensional space. The user 50 can clearly hear the conversation between the speakers 54 and 56 from the voices reproduced by the earphones.

[0035] 2, it is assumed that user 50 faces the direction of speakers 62 and 64. Based on the output of the posture sensor of user 50, a hearing support device (not shown) estimates the direction in which user 50 is looking, and transmits audio signals obtained by reconstructing the voices of speakers 62 and 64 in that direction in the same manner as the voices of speakers 54 and 56 in FIG. 1 to the earphones of user 50. Therefore, in the state shown in FIG. 2, user 50 can clearly hear the voices of speakers 62 and 64.

[0036] Incidentally, the hearing assistance system described in Patent Document 1 is premised on the use of multiple microphone arrays. By using the outputs of the microphone arrays and the outputs of multiple LRFs, it becomes possible to localize multiple sound sources and separate sounds from each sound source.

[0037] However, when only the LRF-integrated microphone array 52 is used as shown in Figures 1 and 2, the voice of a speaker who is far away from the microphone array cannot be acquired sufficiently. Therefore, there is a problem that it is difficult to always clearly reconstruct the voice of a specific speaker, as described in Patent Document 1. If there are multiple microphone arrays, this problem can be easily solved, but in light of the purpose of building a hearing support system on the go as in this embodiment, using multiple microphone arrays is not realistic.

[0038] Therefore, in this embodiment, in addition to the voice acquired by the LRF integrated microphone array 52, the so-called smartphones held by the speakers 54, ..., speaker 64 are used instead of microphones. Since the smartphone has a microphone, it can convert the voice of at least the speaker in the vicinity of the smartphone into an electrical signal. Furthermore, since almost all smartphones have a Wi-Fi communication function, it can transmit voice to the hearing support device via Wi-Fi communication. The hearing support device can reconstruct the voice of a specific speaker in the vicinity of the smartphone from the voice received from the smartphone via Wi-Fi communication. In this case, it is not necessary to use all of the smartphones of each speaker. For example, for speaker 54 and speaker 56, if only the smartphone owned by either one of them is placed between the two, the voices of both speakers can be acquired with one smartphone.

[0039] FIG. 3 shows a conceptual schematic configuration of a hearing assistance system 100 according to this embodiment. Referring to FIG. 3, the hearing assistance system 100 includes the LRF-integrated microphone array 52 shown in FIG. 1 and FIG. 2. The hearing assistance system 100 can further have an additional LRF 104 and a microphone array 106 as an input device in addition to the LRF-integrated microphone array 52. ​​However, when such an additional LRF and microphone array are used, it is not necessary to use the speaker's smartphone, and the hearing assistance system has substantially the same configuration as the hearing assistance device disclosed in Patent Document 1. Therefore, in the following description, the schematic configuration of the hearing assistance system 100 will be described assuming that neither the LRF 104 nor the microphone array 106 is used, and only the LRF-integrated microphone array 52 is used.

[0040] The LRF integrated microphone array 52 includes a microphone array 134, and LRFs 130 and 132. The LRFs 130 and 132 are arranged back to back in the housing of the LRF integrated microphone array 52. ​​The LRFs 130 and 132 can each detect the distance to an object present within a range of 180 degrees in front. Therefore, the LRFs 130 and 132 can measure the distance from the LRF integrated microphone array 52 to the object for the entire circumference of the LRF integrated microphone array 52.

[0041] The hearing assistance system 100 further includes a human position detection and tracking unit 112 that detects and tracks the position of a person based on outputs from the LRF 130 and LRF 132 of the LRF-integrated microphone array 52 and outputs the information, an operation I / F 122 for receiving instructions from a user, and an attention-directed and selective hearing assistance device 114 that reconstructs a sound space by emphasizing the voices of speakers present within the range that the user is paying attention to, based on voice from the microphone array 134 of the human position detection and tracking unit 112, the output of the human position detection and tracking unit 112, a user microphone 118 and a head posture sensor 120 worn by the user, user instructions regarding an anti-target (a speaker whose voice is to be suppressed) received via the operation I / F 122, and audio signals from smartphones 108, 110, etc. used by speakers participating in this hearing assistance system 100 and the user microphone 118, and transmits the audio signals to stereo earphones 116 worn by the user.

[0042] The attention-directed / selective hearing assistance device 114 includes an audio tracking unit 136 that separates and tracks the audio of each sound source based on the output of the human position detection and tracking unit 112 and the output of the microphone array 134, and an HRTF processing unit 138 that reconstructs and synthesizes left and right audio signals to be played back by the stereo earphones 116 using head-related transfer functions (HRTFs) corresponding to the left and right channels of the user's stereo earphones 116, based on the position of each sound source output by the human position detection and tracking unit 112, the audio signals from each sound source tracked by the audio tracking unit 136, and the direction of the user's face (facial posture) estimated from the output of the head posture sensor 120.

[0043] (2) Detailed configuration A. Hearing assistance system 100 The specific configuration of the hearing assistance system 100 shown in Fig. 3 will be described with reference to Fig. 4 and subsequent figures. The hearing assistance system 100 includes an environmental sensor network 150, a sound environment processing device 152, a user-oriented sound reproduction unit 154, and a user-worn device 156.

[0044] The environmental sensor network 150 includes the LRF integrated microphone array 52, the microphone array 106, the LRF 104, the smartphone 108, and the smartphone 110 shown in FIG. 3. Note that in FIG. 4, the environmental sensor network 150 includes a microphone array group and a distance sensor group. However, in the following description, only the microphone array 134 shown in FIG. 3 is used as the microphone array group, and only the LRF 130 and the LRF 132 shown in FIG. 3 are used as the distance sensor group. This is because this embodiment assumes that only the LRF integrated microphone array 52 can be used. When using more microphone arrays and distance sensors, it is considered that the configuration and operation of the hearing assistance system 100 will be substantially similar to the configuration similar to the processing according to Patent Document 1.

[0045] Referring to FIG. 4, the user-worn device 156 worn by the user includes left and right microphones 220, a left earphone 222 and a right earphone 224, and a head posture sensor 120.

[0046] The human position detection and tracking unit 112 and the attention-directed and selective hearing assistance device 114 in FIG. 3 are realized by the sound environment processing device 152 and the user-oriented sound reproduction unit 154 in FIG.

[0047] The sound environment processing device 152 includes a three-dimensional space sound source direction estimation unit 170 that estimates a three-dimensional sound source direction centered on the position of the LRF integrated microphone array 52 based on the output of the microphone array group, a human position detection unit 172 that detects the position of each speaker at each measurement time point based on the output of the distance sensor group, a sound environment intelligence map creation unit 176 that creates a sound environment intelligence map showing the three-dimensional sound environment (position of the sound source) based on the output of the three-dimensional space sound source direction estimation unit 170 and the output of the human position detection unit 172, a sound environment intelligence map storage unit 174 for storing the sound environment intelligence map created by the sound environment intelligence map creation unit 176, and user-specific sound source separation units 178, 180, and 182 that selectively separate sound sources for each user based on the audio signals output by the microphone array group and the smartphone group of the environmental sensor network 150 and the arrangement (position) of the sound source according to the sound environment intelligence map created by the sound environment intelligence map creation unit 176. For example, the user-specific sound source separation unit 178 includes a selective sound source separation unit 190 that selectively separates sounds from a sound source that the first user pays attention to from the audio signal from the environmental sensor network 150 and outputs the sounds together with information on the positions of the sound sources (position information) in order to support the first user. The user-specific sound source separation units 180 and 182 are configured in a similar manner.

[0048] The sound environment intelligence map creation unit 176 estimates the position of the sound source as follows. The 3D space sound source direction estimation unit 170 estimates the direction of arrival of the sound arriving at the microphone array. Information on the position of the object detected by the distance sensor group is obtained based on the output of the human position detection unit 172 and the position of the LRF integrated microphone array 52. ​​If any of the objects detected by the distance sensor group is on any of the half lines drawn from the microphone array in the direction of the sound source, or within a predetermined range from the half lines, the object is determined to be the sound source of the voice, and the position detected by the distance sensor group is determined to be the position of the sound source. In the following explanation, it is assumed that there are J speakers other than the user.

[0049] Note that the "user" referred to here refers to a user who receives support from the hearing assistance system, and is different from a "speaker" who simply participates in a conversation within the hearing assistance system. This hearing assistance system can operate only one hearing assistance system and support multiple users within it, or can operate multiple hearing assistance systems and support the users of each system separately. In the following, for the sake of simplicity, it is assumed that there is only one user to be supported, for example, the first user corresponding to the user-specific sound source separation unit 178.

[0050] In addition, the selective sound source separation unit 190 has a function of suppressing the user's voice in each audio signal upon receiving the output of the left and right microphones 220 so as to prevent the user's voice collected by the left and right microphones 220 from being heard by the user with a slight delay via the left earphone 222 and the right earphone 224 when performing sound source separation.

[0051] The user-oriented sound reproduction unit 154 comprises a user head movement tracking unit 206 for tracking the user's head movement (posture) based on the output of the head posture sensor 120, a sound selection interface 200 which displays on a display device the positions of each sound source, LRF-integrated microphone array 52, and each smartphone received from the selective sound source separation unit 190, and the position of the user receiving support from the hearing assistance system 100, and receives the position of the smartphone and the designation of a target (speaker whose voice is to be heard) or an anti-target through interactive processing with the user through operation by the user as described below, and displays the updated situation, and a voice signal of each sound source received from the selective sound source separation unit 190, the position of each sound source and each smartphone, the designation of a target / anti-target, and the user's head movement (posture) received from the sound selection interface 200. the selective sound source separation unit 190 and the user's head posture received from the user's head motion tracking unit 206, and the direction and position information of each sound source received from the selective sound source separation unit 190 and the estimated direction of the user's face, and reconstructing the position of the sound source according to the head posture of the user received from the user's head motion tracking unit 206 and the direction and position information of each sound source received from the selective sound source separation unit 190 and the estimated direction of the user's face, and generating accurate head-related transfer functions corresponding to the left and right channels for both ears of the user, respectively; and an audio signal generation unit 208 using the head-related transfer functions reconstructed by the sound source spatial transformation unit 204 to generate audio signals for both ears of the user for the left and right channels, respectively, and output them to the left earphone 222 and the right earphone 224.

[0052] The user-worn device 156 is used not only to play back audio signals for hearing assistance using the hearing assistance system 100, but also as a hearing aid on its own. Therefore, the user-worn device 156 is provided with a volume control mechanism (not shown) for feeding back audio from the left and right microphones 220 to the left earphone 222 and the right earphone 224 with different volume balances when used to receive assistance from the hearing assistance system 100 and when used as a hearing aid.

[0053] Referring to FIG. 5, in the sound selection interface 200, the LRF integrated microphone array 52, each speaker (speaker 54, ..., speaker 64), and user 50 are displayed with symbols at the corresponding positions according to the sound environment intelligence map created by the sound environment intelligence map creation unit 176. The LRF integrated microphone array 52, the speaker, and the user 50 are displayed with different symbols, and each speaker has the same symbol, but for example, different numbers (not shown) are superimposed on each speaker. In addition, the smartphones 250, ..., smartphones 254 participating in this system apart from the sound source are also displayed with different symbols. By default, the LRF integrated microphone array 52 is placed in the center of the screen. In the case of FIG. 5, this figure is drawn so that the front of the LRF integrated microphone array 52 is on the right, but it can also be rotated and displayed so that the user 50 is at the bottom center of the screen.

[0054] For example, the user-oriented sound reproduction unit 154 is realized by a tablet computer held by the user 50, and the screen shown in FIG. 5 is displayed on the display. When the user designates a particular sound source as a target or anti-target, the user 50 taps the symbol corresponding to the sound source on the display. A predetermined dialogue appears on the display, and the user designates the sound source as a target or anti-target. In this embodiment, as for the smartphones 250, ..., 254, when each smartphone joins the hearing support system 100 as described later, the smartphone transmits location information obtained by its GPS (Global Positioning System) function to the tablet computer. The tablet computer that receives this location information displays a symbol of the smartphone at that position. Although the position is not necessarily accurate, the user 50 drags and drops the symbol to the correct position, so that the sound selection interface 200 of the user-oriented sound reproduction unit 154 can correctly determine the location of the smartphone.

[0055] 6, the selective sound source separation unit 190 shown in FIG. 4 includes an inter-channel suppression unit 290 for receiving sound source signals 292, ..., 294 obtained by separating the sounds of each sound source from the microphone array 134, a voice signal of the user 50 from the user microphone 118, and a voice signal of each smartphone from the smartphone microphone 286, ..., 288, and performing processing to suppress leakage of interference signals (interference sounds) between channels after synchronizing the signals from each device. The inter-channel suppression unit 290 outputs sound source signals 310, ..., 312 after inter-channel suppression for the sound source signals 292, ..., 294 from the microphone array 134, respectively, and sound source signals 314, ..., 316 after inter-channel suppression for the smartphone microphones 286, ..., 288, respectively. The inter-channel suppression unit 290 only uses the sound from the user microphone 118 when performing inter-channel suppression, and does not output it.

[0056] The selective sound source separation unit 190 further calculates a distance vector r 1 , 288 to each speaker. p1 , …, r pn , 306 which normalize the output audio signals by distance using

[0057] Note that, here, the case where there is only one microphone array, microphone array 134, is shown. However, it goes without saying that there may be two or more microphone arrays, as described in Patent Document 1. However, as described above, this invention is particularly meaningful when there is only one microphone array. When there are multiple microphone arrays, there is no need to use a smartphone, so the configuration and operation can be considered to be basically the same as in Patent Document 1. Therefore, here, the description will be given assuming that there are no microphone arrays other than microphone array 134.

[0058] Also, as a pre-processing step in Fig. 6, although not shown in Fig. 6, there is a process that refers to the target / anti-target flag contained in the signal from the smartphone, and when the value of this flag is "1", the gain for that signal is set to 0. This process will be described later with reference to Fig. 17.

[0059] The inter-channel suppression unit 290 is a process for suppressing leakage of interference signals (jamming signals) between channels when sufficient sound source separation cannot be performed by beamforming using the microphone array 134. For this process, for example, a Wiener filter as described in Patent Document 1 may be used.

[0060] 7, it is assumed that the microphone array 134 shown in FIG. 6 has N1 microphones. These are the microphones in FIG. 1 1, ..., 1 N1The microphone array 134 includes a noise processor 320 for suppressing noise from each of the audio signals from these N1 microphones, and a beamformer generator 322 for separating the audio signals from which noise has been suppressed by the noise processor 320 through beamforming using information about the position of each sound source (speaker) estimated by the sound environment intelligence map creator 176 shown in Fig. 4, and outputting the separated audio signals (sound source signal 292, ..., sound source signal 294). As described above, it is assumed here that there are a total of J speakers other than the user.

[0061] The noise processor 320 is a first microphone of the microphone array 134. 1 a noise estimation unit 340 for estimating noise in the speech signal from the microphone 1; 1 The noise processing unit 320 similarly performs the noise suppression process on the N1-th microphone of the microphone array 134. 1 N1 a noise estimation unit 342 for estimating noise in a speech signal from a microphone; 1 N1 The noise to be suppressed here is stationary noise, which is estimated as an average spectrum in a section where no audio signal from the target sound source exists.

[0062] The beamformer generation unit 322 includes a DS beamformer 360 that performs DS (Delay-Sum) beamforming using outputs from noise estimation units 342, ..., 346 included in the noise processing unit 320 based on the position of the first speaker estimated by the sound environment intelligence map creation unit 176, and outputs a sound source signal 292 in which the voice from the first speaker is separated and emphasized. Similarly, the beamformer generation unit 322 includes a total of J DS beamformers up to DS beamformer 362 that outputs a signal in which the voice from the Jth speaker is separated and emphasized, assuming that there are a total of J speakers.

[0063] 8, smartphone microphone 286 shown in Fig. 6 includes a noise estimation unit 380 and a noise suppression unit 382 that removes noise from the audio signal of smartphone microphone 286 based on the output of noise estimation unit 380. This configuration is basically the same as that for each microphone of noise processing unit 320 shown in Fig. 7. Smartphone microphone 288 shown in Fig. 6 also has a similar configuration to smartphone microphone 286.

[0064] 9, the gain normalization unit 300 shown in FIG. 6 calculates the gain of the noise-suppressed sound source signals 310, ..., 312 output from the inter-channel suppression unit 290 by subtracting the distances r1, ..., r2 from the LRF-integrated microphone array 52 to the respective speakers. J Using g=1 / r i , 402 for performing gain normalization by multiplying the signals by gains indicated by (i=1, . . . , J) and outputting the normalized excitation signal group 330.

[0065] With reference to Fig. 10, the selective volume control unit 202 and the audio signal generating unit 208 shown in Fig. 4 have the following configurations. Note that in this embodiment, as described above, it is assumed that there is one microphone array. Therefore, in Fig. 10 and subsequent figures, the microphone array 134 and the gain normalization unit 300 shown in Fig. 6 are described, but it should be noted that other configurations related to the microphone array are not shown.

[0066] The selective volume control unit 202 includes a target / anti-target selection unit 420 that receives normalized sound source signals 330 (sound source signals 414, ..., sound source signal 416) from the gain normalization unit 300 and sound source signals 334, ..., 336 from the gain normalization units 304, ..., 306, and sets a gain of 1 for signals from a sound source designated by the user as a target among the sound source signals, and sets a gain of 0 for signals from a sound source designated by the user or the owner of the smartphone as an anti-target, among the sound source signals, and an automatic attention target selection unit 422 that selectively emphasizes sounds from sound sources according to the user's attention by applying a gain of 3 dB to target sound sources in the direction to which the user is paying attention and a gain of -3 dB to other directions based on the estimated angle 230 of the user's face estimated by the user's head movement tracking unit 206 shown in Figure 4 among each sound source signal output by the target / anti-target selection unit 420.

[0067] More specifically, in this embodiment, the automatic attention target selection unit 422 multiplies the audio signal of each direction by a gain as shown by the adder 510 in FIG. 11 based on the angle with respect to the direction of the user's face. In this embodiment, the automatic attention target selection unit 422 automatically adjusts the gain to be -3 dB to 3 dB in a cosine wave for beams in the range of -30 degrees to 30 degrees relative to the direction of the face being 0 degrees. This gain is called attention gain. This attention gain selectively emphasizes the sound from a sound source that exists within 30 degrees on either side of the direction in which the user's face is facing, compared to other sounds. The gain is maximum in the direction in which the user's face is facing, decreases as it moves away from that direction, and becomes a constant gain when it is 30 degrees or more away from the direction in which the user's face is facing.

[0068] The audio signal generation unit 208 includes a sound image reconstruction unit 450 which is provided for the sound source signal from the microphone array output by the automatic attention object selection unit 422 and the sound source signal from the smartphone, and which generates and outputs left and right audio signals to be provided to the user based on appropriate head-related transfer functions for each of the sound source signals, based on the angle between the direction of the user's face and the direction from the user to the sound source.

[0069] The sound image reconstruction unit 450 includes a plurality of HRTF processing units 470, ..., 478 that are provided for each sound source and apply a filter based on the head-related transfer function calculated for each sound source to the sound source signal from each sound source, and output left and right audio signals. All of these HRTF processing units 470, ..., 478 have the same configuration.

[0070] For example, the HRTF processing unit 470 processes the sound source signal of the first speaker from the microphone array 134. The HRTF processing unit 470 includes a target angle calculation unit 480 for calculating the relative angle between the direction of the user's face and the direction from the user to the first speaker from the direction of the user's face, the position of the user, and the position of the first speaker, a left ear HRTF filter 482 for selecting an appropriate head-related transfer function based on the calculated target angle, and passing the sound source signal of the first speaker from the automatic attention target selection unit 422 through a filter for the left ear based on this head-related transfer function to generate an audio signal for the left ear, and a right ear HRTF filter 484 for similarly generating an audio signal for the right ear. The configuration of the HRTF processing units 472, ..., 476 is also the same as that of the HRTF processing unit 470. Therefore, the sound image reconstruction unit 450 outputs audio signals in the number of speakers x 2 + number of smartphones x 2.

[0071] The audio signal generation unit 208 further includes a left audio synthesis unit 490 that adds up all of the audio signals for the left ear among the outputs of the sound image reconstruction unit 450 to generate an audio signal for the left ear, and a right audio synthesis unit 492 that similarly adds up all of the audio signals for the right ear among the outputs of the sound image reconstruction unit 450 to generate an audio signal for the right ear.

[0072] The left sound synthesis unit 490 includes an adder 500 that adds all of the sound signals for the left ear from the HRTF processing units 470, ..., 476, a feedback gain setting unit 502 that sets, with respect to the output of the adder 500, a volume balance between when the user-worn device 156 is used alone and when the hearing assistance system 100 is used, and an adder 504 that adjusts the output of the adder 500 in accordance with the setting by the feedback gain setting unit 502.

[0073] The right voice synthesis unit 492 has a similar configuration and includes an adder 510 that adds all of the right ear voice signals from the HRTF processing units 470, ..., 476, a feedback gain setting unit 512 that sets, with respect to the output of the adder 510, the volume balance between when the user-worn device 156 is used by itself and when the hearing assistance system 100 is used, and an adder 514 that adjusts the output of the adder 510 in accordance with the setting by the feedback gain setting unit 512.

[0074] 2 operations With reference to FIG. 12, the hearing support system 100 operates as follows. In the following description, the hearing support system 100 according to this embodiment includes a computer system 530 functioning as the sound environment processing device 152 shown in FIG. 4, a smartphone 250, a smartphone 252, and a smartphone 254, a tablet computer 532 held by a user functioning as the user-oriented sound reproduction unit 154 shown in FIG. 4, and a left earphone 222 and a right earphone 224 paired with the tablet computer 532 and functioning as a set of the user-worn device 156 shown in FIG. 4. The left earphone 222 and the right earphone 224 can communicate with the tablet computer 532 wirelessly. Although not shown in FIG. 12, the left and right microphones 220 and the head posture sensor 120 shown in FIG. 4 are respectively incorporated in the left earphone 222 and the right earphone 224. The LRF-integrated microphone array 52, the computer system 530, the smartphones 250, ..., 254, and the tablet computer 532 can all communicate at high speed using a predetermined wireless standard.

[0075] Here, it is assumed that a user has a tablet computer 532, a computer system 530, and an LRF-integrated microphone array 52, and they are set up in a certain room. The user places the LRF-integrated microphone array 52 in a location where sound source localization is easy, such as the center of the room. The user turns on the computer system 530 and the tablet computer 532, and starts a hearing assistance program on the computer system 530 and a hearing assistance client application on the tablet computer 532, and the LRF-integrated microphone array 52, the computer system 530, and the tablet computer 532 cooperate to start operating as the hearing assistance system 100 according to this embodiment.

[0076] For example, a speaker carrying a smartphone 250 is assumed to participate in this system. When the speaker enters a range that can be managed by the LRF integrated microphone array 52, either the LRF 130 or the LRF 132 of the LRF integrated microphone array 52 shown in FIG. 3 detects the speaker's position and provides the position information to the human position detection unit 172 shown in FIG. 4. When the speaker utters some kind of sound, the sound is captured by the microphone array 134 shown in FIG. 3, and the direction of the speaker as seen from the LRF integrated microphone array 52 (angle based on a predetermined direction) is estimated by the three-dimensional space sound source direction estimation unit 170 shown in FIG. 3. The three-dimensional space sound source direction estimation unit 170 provides information indicating this angle to the sound environment intelligence map creation unit 176.

[0077] The sound environment intelligence map creation unit 176 estimates the position of the speaker in the space based on the object position input from the person position detection unit 172 and the sound source direction information provided from the three-dimensional space sound source direction estimation unit 170 as described above, and creates a sound environment intelligence map. This map is stored in the sound environment intelligence map storage unit 174. New speakers are automatically recognized in this way by the sound environment intelligence map creation unit 176. The same applies when the number of speakers decreases. The sound environment intelligence map creation unit 176 updates the sound environment intelligence map storage unit 174 in this way every time the number of speakers increases or decreases.

[0078] A user who has smartphone 250 starts a predetermined hearing assistance application on the smartphone 250, and connects the hearing assistance application to a hearing assistance program running on computer system 530. The procedure and the like will be described later. In this embodiment, by starting the hearing assistance application as described later, the microphone of smartphone 250 functions as smartphone microphone 286 shown in FIG. 6, and an audio signal indicating the surrounding sounds collected by smartphone 250 is wirelessly transmitted to computer system 530.

[0079] In this way, multiple speakers participate in the hearing aid system. However, for example, it is not necessary for all speakers participating in a conversation to connect their smartphones to the computer system 530. Since the microphone of a smartphone collects surrounding sounds, not limited to a specific speaker, for example, when there are two speakers close to each other, it is sufficient for only one of them to connect his or her smartphone to the hearing aid system.

[0080] In this embodiment, for each smartphone participating in the hearing support system, the hearing support app can specify that the smartphone is an anti-target in order to protect the privacy of the participating speakers. A speaker who has a smartphone can freely switch the smartphone between target and anti-target at any time.

[0081] Similarly, the user can designate a target and an anti-target for each smartphone so that talk that the user is not interested in is not collected by the user-worn device 156. This designation is performed by interactive processing using the tablet computer 532 shown in FIG. 12. However, even if the user designates a smartphone as a target, if the smartphone designates an anti-target, the user cannot hear the voice of that smartphone. In this way, sounds that are unnecessary for the user can be suppressed, and the privacy of speakers near the smartphone is protected.

[0082] 4, while the sound environment processing device 152 is updating the sound environment intelligent map in real time in this manner, the audio signals from the microphone array 134 shown in FIG. 3 are input to the selective volume control unit 202 as sound source signals separated for each sound source by the selective sound source separation unit 190. Similarly, audio signals from the smartphones 250, ..., 254 are wirelessly transmitted from these to the sound environment processing device 152 and input to the selective volume control unit 202 via the selective sound source separation unit 190.

[0083] Specifically, microphone array 134 shown in Fig. 3 and Fig. 6 operates as follows. As shown in Fig. 7, noise estimation section 340 in noise processing section 320 of microphone array 134 estimates stationary noise as the average spectrum from the audio signal in a section where no audio signal from the sound source is present, and provides the stationary noise to noise suppression section 344. 1 By subtracting this stationary noise from the audio signal output by microphone 1, 1 The noise of the first audio signal is suppressed and provided to the beamformer generation unit 322. Other noise estimation units 342 and noise suppression units 346 operate in the same manner. As a result, audio signals indicating the surrounding audio collected by each microphone are provided from the noise processing unit 320 to the beamformer generation unit 322 in the number equal to the number of microphones.

[0084] The DS beamformer 360 of the beamformer generation unit 322 is provided with information indicating the position of the first speaker from the sound environment intelligence map creation unit 176 shown in Fig. 3. Based on this information, the DS beamformer 360 performs beamforming for the position of the first speaker using the audio signals from each microphone provided by the noise processing unit 320, selects and emphasizes the voice of the first speaker, and provides it to the inter-channel suppression unit 290 shown in Fig. 6 as a sound source signal 292. Information indicating the position of each corresponding speaker is also provided to the other beamformers. For example, the position information of the Jth speaker is provided to the DS beamformer 362, and the DS beamformer 362 outputs the sound source signal 294 in which the voice of the Jth speaker is selected and emphasized based on the output of the microphone array 134, and provides it to the inter-channel suppression unit 290 (Fig. 6).

[0085] In the smartphone microphone 286 shown in Fig. 8, the noise estimation units 380 and 382 also perform the same processing as the noise estimation units 340 and 344 in Fig. 7 on the audio signal from the smartphone, and provide the processed audio signal to the inter-channel suppression unit 290. The same applies to other smartphone microphones, for example, the smartphone microphone 288.

[0086] The inter-channel suppression unit 290 passes the sound source signals from each sound source thus input through a Wiener filter as described above in order to prevent leakage of interference signals between channels, and provides the signals to the target / anti-target selection unit 420 shown in FIG. 10 as sound source signals 310, ..., 312, 314, ..., 316.

[0087] 9, noise-suppressed excitation signals 310, ..., 312 corresponding to J excitation sources from the first speaker to the J-th speaker are provided to the gain normalization unit 300. The excitation signal 310 is provided to an individual gain normalization unit 400, and the other excitation signals are similarly provided to the corresponding individual gain normalization units. For example, the last excitation signal 312 is provided to the individual gain normalization unit 402.

[0088] The individual gain normalization unit 400 includes a distance r from the position of the microphone array 134 (the position of the LRF-integrated microphone array 52) shown in FIG. 1 1 is given. The individual gain normalization unit 400 calculates this distance r 1 Using 1, 1 / r 1 The gain indicated by 1 is multiplied by the sound source signal 310 to normalize the amplitude according to the distance. The same applies to other individual gain normalization units, for example, the individual gain normalization unit 402. That is, the individual gain normalization unit 402 normalizes the amplitude according to the distance r 1 J The individual gain normalization unit 402 is given as 1 / r 1 JThe amplitude is normalized according to distance by multiplying the sound source signal 312 by a gain represented by: In this way, a normalized sound source signal group 330 consisting of sound source signals 414, ..., 416 is provided to a target / anti-target selection unit 420 of the selective volume control unit 202 shown in FIG.

[0089] Similar processing is performed on the audio signals from the smartphone microphones 286, ..., 288, etc. shown in Fig. 6. The sound source signals 334, ..., 336 that have been subjected to distance normalization for these are also provided to the target / anti-target selector 420 of the selective volume control unit 202 shown in Fig. 10.

[0090] Referring to FIG. 10, the target / anti-target selection unit 420 multiplies the gain of the audio signals from the normalized sound source signal group 330 (sound source signals 414, ..., 416) and the normalized sound source signals 334, ..., 336 from each smartphone, which are designated as anti-targets by the speaker or user, by a gain of 0, and multiplies the gain of the audio signals from smartphones that are not designated as anti-targets by the speaker but which the user designated as targets by a gain of 1, and outputs the multiplied audio signals to the automatic attention target selection unit 422.

[0091] The automatic attention target selection unit 422 multiplies each audio signal of each sound source thus obtained by a gain of a magnitude as shown in the gain waveform 520 in Fig. 11 based on the estimated angle 230 of the user's face and the relative angle between the direction of the face and the direction of each sound source and each smartphone, with the direction of the face of the user as a reference. As a result, the audio signal from the sound source that is the attention target of the user is automatically selected and emphasized, and the audio signals from other sound sources are automatically attenuated.

[0092] The sound image reconstruction unit 450 of the sound signal generation unit 208 calculates the relative angle between the direction of the user's face and the corresponding sound source for each sound signal from the sound source and the smartphone, generates a sound signal for the left ear and a sound signal for the right ear using an appropriate head-related transfer function, and provides the signal to the sound synthesis unit 452. The left sound synthesis unit 490 of the sound synthesis unit 452 adds all the sound signals for the left ear output from the sound image reconstruction unit 450, and further adds a feedback gain set by the feedback gain setting unit 502 to generate a sound signal for the left ear of the user, and provides the signal to the left earphone 222. The sound synthesis unit 452 similarly generates a sound signal for the right ear, and provides the signal to the right earphone 224.

[0093] As a result, the sound that the user pays attention to is selectively emphasized by the sounds reproduced from the left earphone 222 and the right earphone 224, and the acoustic space is reconstructed for the user, so that the user can experience hearing the sound of the speaker that the user pays attention to clearly from the appropriate direction. Since a smartphone can be used instead of a microphone array, multiple microphone arrays are not required, and a hearing assistance system can be easily constructed and operated using only the LRF-integrated microphone array 52.

[0094] Even when there are multiple users using this hearing support system, the computer system 530 can share the acoustic intelligence map and run multiple processes for them, and each user can use the tablet computer 532 or smartphone as the voice signal generator 208. In addition, the owner of each smartphone can designate that smartphone as an anti-target, so the privacy of each speaker can be protected.

[0095] 3. Computer implementation 13, a computer system 530 constituting a part of the hearing assistance system 100 according to the above embodiment of the present invention and its components are realized by computer hardware, a program executed by the computer hardware, and data stored in the computer hardware. FIG. 13 shows the hardware configuration of the computer system 530.

[0096] Referring to FIG. 13, computer system 530 includes a computer 540 connectable to a network 556, a mouse 546 and a keyboard 544 which are input devices connected to computer 540, a monitor 542 which is an output device connected to computer 540, a speaker 550 and a microphone 552 which are connected to monitor 542, and a USB (Universal Serial Bus) memory 554 which is detachable from monitor 542 and can be read and rewritten at any time.

[0097] The computer 540 includes a CPU (Central Processing Unit) 560, a bus 580 to which the CPU 560 is connected, a GPU (Graphics Processing Unit) 562 connected to the bus 580 and capable of executing a large number of multiplication and addition operations in parallel and at high speed, a ROM (Read-Only Memory) 566 connected to the bus 580 and storing a boot-up program, etc. of the computer system 530, a RAM (Random Access Memory) 568 connected to the bus 580 and storing program instructions, system programs, working data, etc., which can be rewritten and read at any time, an SSD (Solid State Drive) 570 which is a non-volatile memory connected to the bus 580, and a DVD drive 572 connected to the bus 580 and into which a DVD (Digital Versatile Disc) 548 can be attached.

[0098] Computer 540 further includes a network I / F 578 connected to bus 580 and providing a connection to network 556, a USB port 576 connected to bus 580 and to which USB memory 554 is removably attached, and an audio I / F 574 connected to bus 580 and outputs audio signals to speaker 550 and digitizes audio signals from microphone 552 so that CPU 560 can process them.

[0099] In this embodiment, the sound environment intelligence map storage unit 174 shown in FIG. 4, the target / anti-target information set by the sound selection interface 200, the data specifying the computer system 520 shown in FIG. 11, and other data generated during program execution are basically stored in the RAM 568 when the program is executed, and are written out to the SSD 570 as necessary, and are read out from the SSD 570 and loaded into the RAM 568.

[0100] Basically, the computer program executed by the computer 540 is generated as an object program linked with a part of a library preinstalled in the system to realize necessary processing after compiling a source program written in a high-level language with a compiler and converting it into a machine language that the CPU 560 can interpret and execute. The computer program is stored in the DVD 548 or the USB memory 554 and transferred to the SSD 570 via the DVD drive 572 or the USB port 576. Alternatively, the computer program may be sent to the computer 540 through the network 556 and stored in the SSD 570 from the network I / F 578. The program is loaded into the RAM 568 at the time of execution, and dynamically linked with routines in the library preinstalled in the system as necessary during execution. Alternatively, the program may be loaded directly into the RAM 568 from a storage device connected to another terminal or network via the DVD 548 via the DVD drive 572, the USB memory 554 via the USB port 576, or the network 556 and the network I / F 578.

[0101] This program includes a number of instructions that cause the computer 540 to function as the sound environment processing device 152, which is a part of the hearing support system 100 of this embodiment. Some of the basic functions required to perform this operation are linked from a predetermined library by a linker after compiling the source program. Some of the other basic functions are provided through dynamic linking when the program is executed by an operating system (OS) or a third-party program running on the computer 540, or by modules of various tool kits installed on the computer 540. Therefore, this program does not necessarily include all of the functions required to realize the system and method of this embodiment. Among the instructions, this program may include only instructions that execute the operations of the computer system 530 and its components described above by calling appropriate functions or machine language routines in a controlled manner to obtain the desired results. The operation of the computer system 530 is well known, so it will not be repeated here.

[0102] Fig. 14 is a block diagram showing a hardware configuration of the tablet computer 532. Referring to Fig. 14, the tablet computer 532 includes an integrated circuit 638 having a processor 660, a peripheral I / F 664, and a memory controller 662. The processor 660, the peripheral I / F 664, and the memory controller 662 can communicate with each other via a bus 666.

[0103] A memory 628 is connected to the memory controller 662 of the integrated circuit 638. In this embodiment, the memory 628 includes a combination of volatile memory and nonvolatile memory. The nonvolatile memory includes a rewritable flash memory, and stores, for example, a basic program for operating the processor 660, an application program, and the like. The volatile memory is used as a working memory, a working memory when a program is executed.

[0104] The tablet computer 532 further includes a wireless communication device 620 capable of mutual communication with the processor 660 via a peripheral I / F 664, an audio circuit 622, a group of sensors 624, a display controller 632, and a camera controller 636, and an external port 626 connected to the integrated circuit 638 and to which a portable memory, a portable hard disk, etc. (not shown) can be connected. A touch panel display 630 is further connected to the display controller 632. A camera 634 is similarly connected to the camera controller 636.

[0105] The operation of the tablet computer 532 is basically the same as that of the computer system 530 shown in FIG. 13. However, in the case of the tablet computer 532, the program is stored in a non-volatile memory in the memory 628, and is loaded into a volatile memory in the memory 628 at the time of execution. The wireless communication device 620 can directly communicate with other devices at short distances in addition to so-called Wi-Fi communication. It may have a function that allows communication with other devices via a mobile phone line. The configuration of a smartphone is similar to that shown in FIG. 14. However, in the case of a smartphone, the wireless communication device 620 can also perform wireless communication and calls via a mobile phone line. The audio circuit 622 includes a microphone, a speaker, and an earphone output circuit, as well as a digital and analog conversion circuit, and after digitally converting the sound collected by the microphone, the sound is further processed for communication by the processor 660 and transmitted to the tablet computer 532 as a packet via the wireless communication device 620.

[0106] Fig. 15 shows functions realized by cooperation between a computer program and hardware as shown in Fig. 14 in a smartphone (e.g., smartphone 250) in order to process smartphone 250 as an anti-target. With reference to Fig. 15, smartphone 250 functions as an anti-target information input unit 604 that receives input of anti-target setting and non-setting by interactive processing with a user using touch panel display 630 shown in Fig. 14 by a program executed by a processor such as processor 660, an anti-target information storage unit 606 for storing the setting related to the anti-target inputted by anti-target information input unit 604, a voice acquisition unit 608 that digitizes surrounding voice collected by microphone 602, and a packet assembly unit 610 for assembling the voice digitized by voice acquisition unit 608 into a format suitable for communication and then assembling it into a series of packets of a predetermined length.

[0107] The smartphone 250 further functions as a packet header setting unit 612 that sets a flag indicating anti-target information stored in the anti-target information storage unit 606 at a predetermined position in the packet header of each of the series of packets assembled by the packet assembly unit 610, and a packet transmitting unit 614 for transmitting each packet output by the packet header setting unit 612 to the computer system 530 via the wireless communication device 616.

[0108] When a speaker with the smartphone 250 does not want the user to hear their conversation, the speaker designates the smartphone 250 as an anti-target via the user interface 600. This setting is provided from an anti-target information input unit 604 to an anti-target information storage unit 606 and stored therein.

[0109] Meanwhile, the voice acquisition unit 608 digitizes the surrounding voice collected by the microphone 602 and provides it to the packet assembly unit 610. The packet assembly unit 610 encodes the voice so that it is suitable for communication, assembles a data packet including a header including a destination address and the like and a body consisting of data, and provides it sequentially to the packet header setting unit 612. The packet header setting unit 612 sets a flag indicating target / anti-target stored in the anti-target information storage unit 606 to the information portion indicating target / anti-target in the header of these data packets, and provides it to the packet transmission unit 614. The packet transmission unit 614 transmits this packet to the computer system 530 via the wireless communication device 616. Here, the flag indicating target / anti-target indicates an anti-target when it is 1, and indicates a target when it is not 1.

[0110] FIG. 16 is a block diagram showing functions realized by the tablet computer 532 shown in FIGS. 12 and 13 through cooperation of a program and the hardware shown in FIG. 14 to receive settings regarding targets and anti-targets from a user as part of the sound selection interface 200 shown in FIG. 4.

[0111] 16, the tablet computer 532 functions as a target / anti-target information input unit 642 for receiving target / anti-target settings for an arbitrary smartphone from a user by using the touch panel display 630 and the display controller 632 shown in FIG. 14 and a program, a storage device 644 for receiving and storing position information of each sound source and each smartphone from the selective sound source separation unit 190 shown in FIG. 4 and storing the target / anti-target information set by the target / anti-target information input unit 642 for a smartphone specified by the target / anti-target information input unit 642, a sound environment intelligence map display unit 646 for displaying a sound source arrangement as shown in FIG. 5 on the touch panel display 630 shown in FIG. 14 as symbols representing each sound source and each smartphone based on the position information of each sound source and each smartphone stored in the storage device 644, and a smartphone position setting unit 648 for changing the position of the smartphone stored in the storage device 644 in response to a process in which a user changes the display position of the smartphone symbol displayed on the sound environment intelligence map display unit 646 via the user interface 640. The display by the sound environment intelligence map display unit 646 reflects changes in the smartphone's position in real time in response to operations by the user interface 640 and changes in the contents stored in the storage device 644.

[0112] The tablet computer 532 further includes a communication control unit 650 for transmitting, in response to a change in the location of the smartphone stored in the storage device 644, information identifying the smartphone and its new location to the sound environment intelligent map creation unit 176 shown in FIG. 4 via the wireless communication device 652.

[0113] A program for receiving audio data from each smartphone runs on computer system 530. Its control structure is shown in Fig. 17. Referring to Fig. 17, this program is started every time a data packet is received, reads the packet header (step 680), determines whether the value of the target / anti-target flag is 1, and branches the control flow according to the determination result (step 682). If the determination in step 682 is positive, the gain for the audio signal from that smartphone is set to 0 (step 684), and processing for this data packet is terminated.

[0114] If the determination in step 682 is negative, the target / anti-target flag set by the tablet computer 532 for the corresponding smartphone is read (step 686). It is then determined whether the target / anti-target flag is 1 (step 688), and the control flow branches according to the determination result. If the determination in step 688 is positive, control proceeds to step 684, where the gain for the audio signal from the smartphone is set to 0, and execution of the program is terminated. If the determination in step 688 is negative, control proceeds to step 690, where the gain for the audio signal from the smartphone is set to 1, and execution of the program is terminated.

[0115] By implementing such a function, when a speaker near a smartphone designates the smartphone as an anti-target, the designation is immediately transmitted to and stored in the computer system 530 and the tablet computer 532. The gain in the selective volume control unit 202 for the audio signal from the smartphone is immediately set to 0. Therefore, the user cannot hear the conversation voice from the smartphone. When the smartphone is designated as a target, the designation is also immediately transmitted to and stored in the computer system 530 and the tablet computer 532. Therefore, the user can immediately hear the voice from the smartphone.

[0116] Similarly, when the user designates a particular smartphone as an anti-target using the tablet computer 532, the gain of the audio signal from that smartphone becomes 0. When the user designates the smartphone as a target, the gain of the audio signal from that smartphone becomes 1.

[0117] As a result of the above processing, the user can clearly hear the voice they want to hear while respecting the privacy of the smartphone user and preventing the user from hearing unnecessary sounds. Hearing support for users can be easily provided using the smartphone carried by the user without the need for multiple microphone arrays.

[0118] Second Second Embodiment 1 Configuration In the first embodiment, as shown in Fig. 6, gain normalization of each sound source signal is performed after the inter-channel suppression unit 290. However, the present invention is not limited to such an embodiment. In the second embodiment, gain normalization is provided before the inter-channel suppression unit 290.

[0119] 18, a user-specific sound source separation unit 700 according to the second embodiment can be used in place of the selective sound source separation unit 190 of the first embodiment shown in Fig. 6. The user-specific sound source separation unit 700 uses the microphone array 134, the user microphone 118, the smartphone microphones 286, ..., 288 similar to those in Fig. 6, and a distance vector r 1 9, and a gain normalization unit 720 having a configuration similar to that of the individual gain normalization unit 400 shown in FIG. U and the distance r U , 288 and the distance r between the smartphone microphones 286, . . . , 288 and the user's microphones. p1 , …, r pn, 734 which perform amplitude normalization based on distance using the above-mentioned gain normalization units 304, . . . , 306 shown in FIG.

[0120] The excitation signal group output from gain normalization section 720 and the excitation signals output from gain normalization sections 730, . . . , 732 are all input to inter-channel suppression section 290, where interference signals between channels are suppressed.

[0121] In other respects, the hearing assistance system according to this embodiment has the same configuration as the hearing assistance system 100 according to the first embodiment, and operates in the same manner.

[0122] By arranging the gain normalization section before the inter-channel suppression section 290 in this manner, it is possible to correct the difference in gain between devices and prevent excessive or insufficient suppression in the inter-channel suppression processing.

[0123] 3. Participation sequence for hearing support system 19, the communication sequence between the hearing support system and the smartphone when the smartphone is allowed to participate in the hearing support system is as follows. Here, it is assumed that two hearing support devices 762 and 764 provide services of separate hearing support systems in the same or adjacent spaces. As can be seen from the description of the above embodiment, hearing support device 762 and hearing support device 764 can form separate hearing support systems even in overlapping spaces.

[0124] For example, assume that smartphone 760 joins hearing support device 762. Smartphone 760 wirelessly broadcasts a request for the identifiers of the hearing support devices to nearby hearing support devices to join the nearby support system (step 780). In response to this request, hearing support devices 762 and 764 each return their own location (or the location of the LRF-integrated microphone array they are using) and the identifiers of the hearing support systems they operate to smartphone 760 (steps 782 and 784).

[0125] The smartphone 760 displays these locations two-dimensionally using symbols such as icons with identifiers. The user of the smartphone 760 selects the hearing support device represented by a symbol (e.g., hearing support device 762) by clicking on one of the symbols (step 786). The smartphone 760 notifies the hearing support device 762 of a connection request to the hearing support device 762 together with the location information of the smartphone 760 (step 788). Upon receiving this notification, the hearing support device 762 registers the smartphone 760 in the sound environment intelligence map (step 790) and starts a service for processing audio signals from the smartphone 760 (step 792).

[0126] On the other hand, if there is no participation notification from smartphone 760 within a predetermined time after hearing assistance device 764 sends the notification from hearing assistance device 764, hearing assistance device 764 determines that smartphone 760 is not participating and stops subsequent processing related to smartphone 760 (step 794).

[0127] By joining the smartphone to the hearing support system using the above procedure, even when multiple hearing support systems are operated separately in a relatively small area, the smartphone can be joined to an appropriate hearing support system. As a result, even when various groups are talking in a place with many people, the user can select a desired talk by a desired group and listen clearly.

[0128] In the above embodiment, the direction of the user's face is detected by a head posture sensor worn by the user. In the above embodiment, noise suppression is performed before the beamformer, as shown in Figure 7. However, the invention is not limited to such an embodiment, and noise suppression may be performed after the beamformer.

[0129] In the above embodiment, the direction of the user's face is detected by a head posture sensor worn by the user. However, the present invention is not limited to such an embodiment. For example, a camera that captures an image of the upper body of the user may be provided in the LRF integrated microphone array 52, and the direction of the user's face may be detected from the image of the user captured by this camera.

[0130] The embodiments disclosed herein are merely illustrative, and the present invention is not limited to the above-described embodiments. The scope of the present invention is defined by the claims of the appended claims, taking into consideration the detailed description of the invention, and includes all modifications within the scope and meaning equivalent to the words described therein. [Explanation of symbols]

[0131] 50 users 52 LRF Integrated Microphone Array 54, 56, 58, 60, 62, 64 Speakers 100, 900 Assistive Listening System 104, 130, 132 LRF 106, 134 microphone array 108, 110, 250, 252, 254, 760 Smartphone 112 Person Location Detection and Tracking Unit 114 Attention-directed and selective hearing aid 116 Stereo Earphones 118 User Microphone 120, 918 Head posture sensor 122 Operation I / F 136 Audio Tracking Unit 138 HRTF Processing Unit 150 Environmental Sensor Network 152 Sound environment processing device 154 User-oriented audio playback section 156 User-worn devices 170 3D spatial sound source direction estimation unit 172 Person position detection unit 174 Sound Environment Intelligence Map Memory Unit 178, 180, 182, 700 User-specific sound source separation unit 190 Selective source separation unit 202 Selective volume control 204 Spatial transformation of sound source 206 User Head Movement Tracking Unit 208 Audio signal generator 290 Inter-channel suppression section 292, 294, 314, 316, 414, 416 Sound source signal 300, 302, 304, 306, 720, 730, 732, 734 Gain normalization section 310, 312 Sound source signal group 420 Target / Anti-Target Selection Unit 422 Automatic Selection of Attention Targets 450 Sound image reconstruction section 452 Voice synthesis unit 480 Target Angle Calculation Unit 604 Anti-target information input section 606 Anti-Target Information Storage Unit 608 Voice Acquisition Unit 612 Packet Header Setting Section 614 Packet Transmitter 642 Target / Anti-Target Information Input Section 644 Storage device 646 Sound Environment Intelligence Map Display Unit 648 Smartphone position setting unit 660 Processor 762, 764 Hearing assistive devices

Claims

1. A hearing assistance device for assisting a user's hearing, comprising: a sound source localization means for identifying and outputting the position of a sound source using an output from a position detection device that detects the position of an object and an output from the microphone array; a sound source separation means for separating sounds from the sound sources based on a sound signal from a terminal capable of transmitting a sound signal representing surrounding sounds, a sound signal from the microphone array, and the position of the sound source identified by the sound source localization means, and outputting the separated sounds as a sound signal; a sound signal reconstructing means for selectively reconstructing a sound signal from a specific sound source based on the face posture of the user, the position of the sound source identified by the sound source localization means, and the sound signal separated and output for each sound source by the sound source separation means; A terminal management means for managing the terminal; a terminal designation means for designating any one of the terminals managed by the terminal management means as a target or an anti-target; a target selection / exclusion means for selecting or excluding a voice signal from a terminal designated by the terminal designation means as a target for reconstructing the voice signal by the voice signal reconstruction means in accordance with the designation, The terminal designation means a symbol display means for displaying, on a display device, symbols representing terminals managed by said terminal management means and symbols representing sound sources separated by said sound source separation means as two-dimensional displays showing the positions of each symbol; a target / anti-target input means for receiving a user input regarding designation of one of the symbols of the terminals displayed on the display device as a target or an anti-target by selecting the symbol through a dialogue with the user; A hearing support device including: a target flag storage means for storing a target flag representing a terminal designated as a target or an anti-target by the target / anti-target input means among the terminals managed by the terminal management means.

2. The hearing support device of claim 1, further comprising an anti-target exclusion means for excluding the audio signal from the terminal from the target of audio signal reconstruction by the audio signal reconstruction means in response to the audio signal from the terminal being marked with a predetermined anti-target flag.

3. The terminal management means a position storage means for determining and storing an initial position of a terminal in response to a participation request from the terminal requesting participation in the hearing support provided by the hearing support device; and a position updating means for updating the position of the specific terminal stored in the position storage means based on the display position of a specific symbol representing a specific terminal displayed on the display device in response to a user input that changes the display position of the specific symbol on the display device.

4. The terminal management means further comprises: an identification information transmitting means for transmitting, in response to a request from a new terminal for information on a hearing assistance device that can be used, identification information for identifying the hearing assistance device to the new terminal; and record generating means for generating a new record in said position storage means for storing information about said new terminal in response to a participation request from said new terminal specifying said identification information in response to said identification information.

5. the terminal is a communication terminal having a function of performing wireless communication according to a predetermined communication standard, the hearing assistance device further includes a wireless communication device that performs wireless communication in accordance with the predetermined communication standard, The hearing assistance device according to claim 1 , wherein the sound source separation means determines the position of the sound source using an audio signal received by the wireless communication device from the terminal.

6. The hearing assistance device according to claim 5 , wherein the communication terminal is a smartphone.

7. A hearing assistance system for assisting a user's hearing, comprising: A position detection device for detecting a position of an object; at least one microphone array; a sound source localization means for identifying and outputting the position of a sound source using an output from the position detection device and an output from the microphone array; a sound source separation means for separating sounds from each of the sound sources based on a sound signal from a terminal capable of transmitting a sound signal representing surrounding sounds, a sound signal from the microphone array, and the position of the sound source identified by the sound source localization means, and outputting the separated sounds as a sound signal; a sound signal reconstructing means for selectively reconstructing a sound signal from a specific sound source based on the face posture of the user, the position of the sound source identified by the sound source localization means, and the sound signal separated and output for each sound source by the sound source separation means; A terminal management means for managing the terminal; a terminal designation means for designating any one of the terminals managed by the terminal management means as a target or an anti-target; a target selection / exclusion means for selecting or excluding a voice signal from a terminal designated by the terminal designation means as a target for reconstructing the voice signal by the voice signal reconstruction means in accordance with the designation, The terminal designation means a symbol display means for displaying, on a display device, symbols representing terminals managed by said terminal management means and symbols representing sound sources separated by said sound source separation means as two-dimensional displays showing the positions of each symbol; a target / anti-target input means for receiving a user input regarding designation of one of the symbols of the terminals displayed on the display device as a target or an anti-target by selecting the symbol through a dialogue with the user; a target flag storage means for storing a target flag representing a terminal designated as a target or an anti-target by the target / anti-target input means among the terminals managed by the terminal management means, the hearing assistance system including:

8. A sound source localization device for use in a hearing assistance system for assisting a user's hearing, comprising: a sound source localization means for identifying and outputting the position of a sound source using an output from a position detection device that detects the position of an object and an output from the microphone array; a sound source separation means for separating sounds from each of the sound sources based on a sound signal from a terminal capable of transmitting a sound signal representing surrounding sounds, a sound signal from the microphone array, and the position of the sound source identified by the sound source localization means, and outputting the separated sounds as a sound signal; an output means for outputting sound source position information indicating the position of the sound source identified by the sound source localization means and the audio signals separated for each sound source by the sound source separation means in association with each other; A terminal management means for managing the terminal; a terminal designation means for designating any one of the terminals managed by the terminal management means as a target or an anti-target; a target selection / exclusion means for selecting or excluding a voice signal from a terminal designated by the terminal designation means as a target for output of the voice signal by the output means in accordance with the designation; The terminal designation means a symbol display means for displaying, on a display device, symbols representing terminals managed by said terminal management means and symbols representing sound sources separated by said sound source separation means as two-dimensional displays showing the positions of each symbol; a target / anti-target input means for receiving a user input regarding designation of one of the symbols of the terminals displayed on the display device as a target or an anti-target by selecting the symbol through a dialogue with the user; A sound source localization device including: a target flag storage means for storing a target flag representing a terminal designated as a target or anti-target by the target / anti-target input means among the terminals managed by the terminal management means.

Images