Hearing aid device, hearing aid method, and program

Abstract

To assist a target individual with a hearing problem in hearing.SOLUTION: A hearing aid device monitors a conversation between a target person with a hearing problem and a speaker on the basis of a video of the target person, and determines whether the target person can hear the speech uttered by the speaker on the basis of one or more of the target person's appearance and behavior and the content of the target person's response utterance. The hearing aid device then extracts audio signal features of the speech determined to be audible or inaudible.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a technology for assisting a target person in listening. [Background technology]

[0002] Elderly people often have difficulty hearing during conversations due to a decline in their hearing. Traditionally, people with hearing problems, such as the elderly, have often worn hearing aids, which are medical devices that enable them to hear clearly. Hearing aids require periodic adjustments to adapt sounds to the user's hearing ability and make them easier to hear. Patent Document 1 describes an auditory characteristics detection device that shortens the time required to detect the volume and frequency that a target person can hear, while improving detection accuracy and making appropriate adjustments. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-080509 Summary of the Invention [Problem to be solved by the invention]

[0004] There are individual differences in the volume and frequency range that are easy to hear. Hearing aids are used after adjusting for these individual differences, but elderly users with dementia or other illnesses have difficulty recognizing that their hearing aid settings are not appropriate or communicating this to a third party through conversation. This has made it difficult to resolve each individual's specific hearing difficulties through hearing aid adjustments.

[0005] One of the goals of this disclosure is to assist subjects with hearing problems in hearing. [Means for solving the problem]

[0006] In order to solve the above problem, in one aspect of the present invention, a hearing aid support device includes: a determination means for monitoring a conversation between a target person with hearing problems and a speaker based on a video of the target person, and determining whether the target person can hear the voice uttered by the speaker based on at least one of the target person's appearance and behavior and the content of the target person's response; and an extraction means for extracting audio signal features of the audio determined to be audible or inaudible.

[0007] In another aspect of the present invention, a hearing assistance method executed by a hearing assistance device includes: Based on a video of a target person with hearing problems, a conversation between the target person and a speaker is monitored, and whether the target person can hear the sound emitted by the speaker is determined based on one or more of the target person's appearance and behavior and the content of the target person's response utterance; The audio signal features of the audio determined to be audible or inaudible are extracted.

[0008] In yet another aspect of the invention, a program includes: Based on a video of a target person with hearing problems, a conversation between the target person and a speaker is monitored, and whether the target person can hear the sound emitted by the speaker is determined based on one or more of the target person's appearance and behavior and the content of the target person's response utterance; The computer is caused to execute a process of extracting audio signal features of the audio determined to be audible or inaudible. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to provide assistance to a target person with hearing problems in hearing comprehension. [Brief explanation of the drawings]

[0010] [Figure 1] 1 shows an example of a schematic configuration of a hearing aid support system. [Figure 2] FIG. 1 is a diagram schematically illustrating a dialogue between a speaker and a target person. [Figure 3] 1 shows an example of the hardware configuration of a hearing aid support device. [Figure 4] 1 is a block diagram showing an example of the functional configuration of a hearing assistance device and a hearing aid. [Figure 5] 10 is a flowchart of a feature extraction process. [Figure 6] 10 is a flowchart of a value derivation process. [Figure 7] 10 shows an example of a schematic configuration of a hearing aid support system according to a second embodiment. [Figure 8] FIG. 1 is a diagram schematically illustrating a situation in which multiple target persons are listening to an audio broadcast. [Figure 9] FIG. 10 is a block diagram showing an example of the functional configuration of a hearing aid support device according to a second embodiment. [Figure 10] 10 is a flowchart of an audio output process. [Figure 11] FIG. 1 is a diagram schematically illustrating a target person walking outdoors. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. [First embodiment] (composition) 1 shows an example of the schematic configuration of a hearing aid support system 100 that applies a hearing aid support device 1 according to a first embodiment of the present disclosure. The hearing aid support system 100 is a system that supports hearing by extracting, based on a video of a person with hearing problems who is the target of hearing support (hereinafter simply referred to as the "target person"), audio signal features that the target person was able to hear or was unable to hear.

[0012] Specifically, hearing aid system 100 is a system that uses an audio playback device and adjusts the adjustment parameters of the audio playback device to appropriate values ​​based on the audio signal characteristics of the target person, thereby allowing the target person to hear audio that is easy to hear. The target person is someone who has difficulty hearing or cannot hear conversations well, and one example is an elderly person with hearing concerns.

[0013] Here, the audio playback device is a device that outputs received audio, such as a hearing aid or a speaker. In the first embodiment, the audio playback device is a hearing aid worn by the target person. A hearing aid is a medical device that supports the user's hearing by converting audio received from a microphone into easy-to-listen audio that is tailored to the user's hearing ability and outputting the audio from earphones.

[0014] In hearing aid support system 100, hearing aid support device 1 and hearing aid 3 used by a subject are communicatively connected via network 5 such as the Internet. Hearing aid support device 1 is an information processing device that processes, stores, and transmits and receives various data, and acquires and analyzes video of the subject, extracts audio signal features that the subject was able to hear or not hear, and stores and manages them in feature DB 31. In addition, hearing aid support device 1 derives adjustment parameter values ​​for hearing aid 3 based on the audio signal features of the subject, and transmits the derived values ​​to hearing aid 3, thereby adjusting hearing aid 3 to match the hearing ability of the subject.

[0015] Here, the video may be, for example, a surveillance camera image capturing a conversation between a target person who is a listener and a speaker who is a speaker. FIG. 2 is a diagram schematically illustrating a conversation between the speaker and the target person. As shown in the figure, in this embodiment, the target person is wearing a hearing aid 3. It is desirable for the video to show both the target person and the speaker, but this is not limited thereto. Any video of the target person during the conversation can be used as long as it allows confirmation of the target person's appearance, behavior, and utterances in response to the voice uttered by the speaker. The video may be acquired via any method, such as from a surveillance camera or a smartphone via a network 5, or from an external storage device such as a USB (Universal Serial Bus) memory.

[0016] The hearing aid 3 is adjusted to suit the hearing ability of the target person by receiving adjustment parameter values ​​from the hearing assistance device 1 via the network 5. Once adjusted, the hearing aid 3 processes the sound coming in from the microphone into easier-to-hear sound and outputs it from the earphones, thereby assisting the target person in hearing.

[0017] 3 is a block diagram showing an example of the hardware configuration of the hearing aid support device 1. As shown in the figure, the hearing aid support device 1 includes an interface 11, a processor 12, a memory 13, a recording medium 14, a display unit 15, and an input unit 16. These components and the feature DB 31 are interconnected via a bus.

[0018] The interface 11 exchanges data with the hearing aid 3. The interface 11 is used when transmitting the derived adjustment parameter values ​​to the hearing aid 3. The interface 11 is also used when the hearing aid support device 1 exchanges data with a specific device connected by wire or wirelessly.

[0019] The processor 12 is a computer such as a CPU (Central Processing Unit), and executes a pre-prepared program to control the entire hearing aid support device 1. The processor 12 may be a CPU, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating Point number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination of these.

[0020] The memory 13 is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 13 stores programs executed by the processor 12. The memory 13 is also used as a working memory while the processor 12 is executing various processes.

[0021] Recording medium 14 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from hearing aid support device 1. Recording medium 14 records various programs executed by processor 12. When hearing aid support device 1 executes predetermined processes such as feature extraction processing and value derivation processing, the programs recorded on recording medium 14 are loaded into memory 13 and executed by processor 12.

[0022] The display unit 15 displays a predetermined image on, for example, an LCD (Liquid Crystal Display), etc. The input unit 16 includes a keyboard, mouse, touch panel, etc., and is used by an operator who manages the hearing aid support device 1, etc.

[0023] The feature DB 31 stores and manages a person ID for identifying a target person and an audio signal feature that the target person can hear or cannot hear, in association with each other.

[0024] 4 is a block diagram showing an example of the functional configuration of the hearing aid support device 1 and the hearing aid 3. Functionally, the hearing aid support device 1 comprises a determination unit 41, an extraction unit 42, and a management unit 43. The hearing aid 3 comprises a conversion unit 51. The determination unit 41, extraction unit 42, and management unit 43 are realized by the processor 12 executing a program.

[0025] The determination unit 41 monitors the conversation between the target person and the speaker based on the video and determines whether the target person can hear the audio uttered by the speaker based on one or more of the target person's appearance, behavior, and the content of the target person's response utterance. Specifically, the determination unit 41 compares the content of the speaker's utterance with one or more of the target person's appearance, behavior, such as nodding, facial expressions, and gestures, and the content of the response utterance, such as asking again or asking "huh?", based on the monitored video, to determine which audio the target person was able to hear well and which audio they were unable to hear on a phrase-by-phrase basis. For example, if the target person nods after the audio uttered by the speaker, the determination unit 41 determines that the audio was "intelligible." On the other hand, if the target person's face is expressionless or confused after the audio uttered by the speaker, the determination unit 41 determines that the audio was "inaudible."

[0026] Here, a phrase is a unit consisting of a collection of words or sentences uttered by a speaker in one round trip of speech from the speaker and a response speech from the target person. The determination unit 41 determines whether the target person can hear the speech uttered by the speaker on a phrase-by-phrase basis based on the audio in the video. The method of dividing the units for determination is not limited to phrase-by-phrase, and can be set arbitrarily, for example, by dividing when silence continues for more than a threshold, or by dividing after a predetermined time has elapsed.

[0027] The extraction unit 42 extracts audio signal features of audio that has been determined by the determination unit 41 to be audible or inaudible. Audio signal features include frequency, amplitude characteristics, signal-to-noise ratio (hereinafter also referred to as "S / N value"), and audio speed extracted from the audio waveform. Frequency is a numerical representation of the number of times air vibrates per unit time, and corresponds to the pitch of the voice. Amplitude characteristics include the maximum displacement of the audio waveform, and correspond to the volume of the voice. The S / N value represents the ratio between the electrical signal used to transmit audio (signal) and other surrounding electrical signals (noise).

[0028] Furthermore, based on the audio signal features determined to be audible or inaudible, the extraction unit 42 derives adjustment parameter values ​​for the hearing aid 3 tailored to the hearing ability of the target person, and transmits the adjustment parameters to the hearing aid 3. The adjustment parameters are parameters that adjust the frequency, amplitude characteristics, S / N value, speech speed, etc., for converting the audio input from the microphone of the hearing aid 3. As one example, based on the audio signal features, the extraction unit 42 derives a frequency value so that if high-pitched sounds are difficult to hear, they are converted into low-pitched sounds that the target person can hear. As another example, if low-pitched sounds are difficult to hear, the extraction unit 42 derives an amplitude characteristic value so that they are converted into sounds that the target person can hear. As another example, if the speaker speaks too quickly and is difficult to hear, the extraction unit 42 derives a speech speed value to slow them down.

[0029] The management unit 43 stores and manages the person ID of the target person and the audible or inaudible voice signal features of the target person in association with each other in the feature DB 31.

[0030] The feature DB 31 may store the target person ID, the target person's voice signal features, and the date and time when the voice signal features were stored in association with each other. In this case, the extraction unit 42 uses the voice signal features from a preset date and time onward when deriving the adjustment parameter values ​​or converting the voice, which will be described later. This makes it possible to provide appropriate support tailored to the target person's hearing ability, even if the target person's hearing ability deteriorates over time.

[0031] The conversion unit 51 of the hearing aid 3 makes adjustments based on the values ​​of the adjustment parameters received from the hearing aid support device 1, processes the sound coming in from the microphone into easy-to-listen sound that matches the hearing ability of the target person and outputs it from the earphones, thereby assisting the target person in hearing.

[0032] In the above configuration, the determination unit 41 and management unit 43 of the hearing aid support device 1 are examples of the determination means and management means of the present disclosure, respectively. Also, the extraction unit 42 is an example of the extraction means, derivation means, and output means.

[0033] (Feature extraction processing) Next, we will explain the feature extraction process performed by the hearing aid support device 1. Figure 5 is a flowchart of the feature extraction process performed by the hearing aid support device 1. This process is realized by the processor 12 shown in Figure 3 executing a program prepared in advance.

[0034] First, the hearing aid support device 1 acquires video of the target person and monitors the conversation between the target person and the speaker (step S101). Based on the acquired video, the hearing aid support device 1 determines whether the target person can hear the speech uttered by the speaker, phrase by phrase, based on one or more of the target person's appearance and behavior and the content of the target person's response utterance (step S102). Next, the hearing aid support device 1 extracts audio signal features of speech that is determined to be audible or inaudible (step S103). Next, the hearing aid support device 1 associates the target person's person ID with the audio signal features that the target person can hear or cannot hear, and stores and manages them in the feature DB 31 (step S104). This completes the feature extraction process.

[0035] (Value derivation process) Next, we will explain the value derivation process performed by the hearing aid support device 1. Figure 6 is a flowchart of the value derivation process performed by the hearing aid support device 1. This process is realized by the processor 12 shown in Figure 3 executing a program prepared in advance.

[0036] First, the hearing aid support device 1 identifies the person wearing the hearing aid 3 to be adjusted (step S201). Any method can be used to identify the person, such as entering the person ID of a person who has received a hearing aid adjustment request from a family member or the like, or selecting a person wearing a hearing aid whose hearing aid adjustment date is managed in advance and that has not been adjusted for a predetermined period of time.

[0037] The hearing aid support device 1 acquires audio signal features corresponding to the person ID of the identified target person from the feature DB 31 (step S202). Next, the hearing aid support device 1 derives adjustment parameter values ​​for the hearing aid 3 based on the acquired audio signal features (step S203). The hearing aid support device 1 transmits the derived values ​​to the hearing aid 3 (step S204). The hearing aid 3 is adjusted based on the adjustment parameter values ​​received from the hearing aid support device 1, and processes the audio input from the microphone into easy-to-listen audio tailored to the target person's hearing ability and outputs it from the earphones, thereby assisting the target person in hearing. This completes the value derivation process.

[0038] In this embodiment, the hearing aid support device 1 determines whether the target person can hear based on the target person's appearance, behavior, and response utterance content, but this is not limited to this, and the target person's ability to hear may also be determined based on the speaker's appearance, behavior, and response utterance content. In this case, the video used should be one that allows the speaker's appearance, behavior, and response utterance content to be confirmed during the conversation.

[0039] Furthermore, in this embodiment, the hearing aid 3 is automatically adjusted when it receives the value of the adjustment parameter from the hearing aid support device 1, but this is not limited to this. The hearing aid support device 1 may transmit the value of the adjustment parameter to a specified terminal device used by a store clerk who adjusts the hearing aid 3 or a family member of the person in question, and the receiving store clerk or family member may then use the value to adjust the hearing aid 3.

[0040] In addition, the hearing aid support device 1 of this embodiment stores and manages audio signal features in the feature DB 31 together with either of the two values ​​"audible" or "inaudible", but this is not limited to this and the features may also be stored and managed together with any of the continuous values, for example, 0 to 4, which represent the degree of ease or difficulty of hearing.

[0041] Furthermore, in this embodiment, the subject wears a hearing aid 3, but this is not limited to this and the present invention can also be applied to a subject who does not wear a hearing aid 3. In this case, the hearing aid device 1 is communicably connected via network 5 to a personal speaker or the like installed for the subject, and transmits adjustment parameter values ​​to the personal speaker. The personal speaker has a conversion unit and is adjusted based on the adjustment parameter values ​​received from the hearing aid device 1. After adjustment, the personal speaker processes the sound coming from the microphone into easy-to-listen sound tailored to the hearing ability of the subject and outputs it, thereby assisting the subject in hearing.

[0042] In this way, the hearing aid support device 1 manages the target person in association with the audio signal features that the target person was able to hear or could not hear, and can use the audio signal features to set the adjustment parameters of the hearing aid 3 or personal speaker to values ​​that match the target person's hearing ability, thereby providing support for the target person's hearing.

[0043] Elderly people suffering from dementia or other illnesses have difficulty realizing that the settings on their hearing aid 3 are not correct or informing a third party of this, but the hearing aid support device 1 allows the elderly to easily adjust the hearing aid without even realizing it. In other words, the specific hearing difficulties of each individual can be fully reflected in changes to the hearing aid settings, allowing for appropriate hearing support for the elderly.

[0044] Next, a second embodiment will be described. [Second embodiment] (composition) 7 shows an example of the schematic configuration of a hearing aid support system 200 to which a hearing aid support device 1x according to the second embodiment of the present disclosure is applied. Note that, for the sake of convenience, a description of the same parts as those in the first embodiment will be omitted.

[0045] Hearing aid support system 200 is a system that supports the hearing of a target person by converting a predetermined sound into an easy-to-listen sound that matches the hearing ability of the target person based on sound signal features that the target person can hear or cannot hear, and outputting the converted sound. Specifically, hearing aid support system 200 is a system that uses a sound playback device, transmits sound converted based on the sound signal features of the target person to the sound playback device, and causes the sound playback device to play back an easy-to-listen sound that matches the hearing ability of the target person.

[0046] Here, the audio playback device is a device that outputs audio by playing back audio data, and examples thereof include a hearing aid, a speaker, etc. In the second embodiment, the audio playback device is a speaker installed in a position where the target person can hear the audio.

[0047] It is assumed that the feature DB31 of the hearing aid support system 200 stores and manages in advance, by the feature extraction process of the first embodiment, the person ID of the target person and the audio signal features that the target person can hear or cannot hear, in association with each other.

[0048] In the hearing aid support system 200, a hearing aid support device 1x and a speaker 4 are communicatively connected via a network 5 such as the Internet. The hearing aid support device 1x is an information processing device that processes, stores, and transmits / receives various data. It analyzes a video of a target person, extracts audio signal features that the target person can or cannot hear, and stores and manages them in a feature DB 31. When the hearing aid support device 1x acquires a predetermined sound, it converts the sound into an easy-to-listen sound tailored to the target person's hearing ability based on the target person's audio signal features, and transmits the converted sound (hereinafter also referred to as "converted sound") to the speaker 4. The speaker 4 is installed, for example, in a nursing home where the target person lives, and plays back the converted sound received from the hearing aid support device 1x to allow the target person to hear an easy-to-listen sound. Note that in this embodiment, the target person is not limited to one person, and may be multiple people.

[0049] In this embodiment, the microphone of the speaker and the hearing aid support device 1x are communicatively connected via a network 5, and the hearing aid support device 1x can acquire the speech of the speaker in real time. However, the present disclosure is not limited to this, and the hearing aid support device 1 may acquire predetermined speech at any time via the network, or may acquire predetermined speech using external storage such as a USB memory.

[0050] 8 is a diagram showing a schematic diagram of a situation in which multiple target persons are listening to an audio broadcast. As shown in the figure, target persons A to D are present in an audio transmission area where they can hear the audio output from speaker 4. The speaker may be in a different room or a remote location from target persons A to D, or may be present in the same audio transmission area as target persons A to D.

[0051] The hearing aid support device 1x first identifies target persons present in the sound transmission area. Any method can be used to identify target persons, such as having a staff member at a nursing home input the person ID of a target person present in the sound transmission area through a predetermined operation, or having the hearing aid support device 1 identify the target person using facial recognition or other methods based on video captured of the sound transmission area. Next, the hearing aid support device 1 obtains audio signal features for each target person from the feature DB 31 based on the person ID of the identified target person.

[0052] For example, when a speaker in a remote location speaks, the spoken voice is transmitted in real time to the hearing aid device 1x via the network 5. At this time, the hearing aid device 1x acquires voice signal features from each of the target persons present in the voice transmission area and generates voice signal features that make the voice easier for everyone to hear. Next, the hearing aid device 1x uses the generated voice signal features to convert the voice spoken by the speaker so that it is easier for everyone to hear, and transmits the converted voice to the speaker 4. The speaker 4 plays back the received converted voice, and makes the speaker's speech heard in a voice that is easy for all target persons to hear.

[0053] The hardware configuration of the hearing aid support device 1x is the same as that of the first embodiment described above, and therefore a description thereof will be omitted for the sake of simplicity.

[0054] 9 is a block diagram showing an example of the functional configuration of hearing aid support device 1x. Functionally, hearing aid support device 1x includes a determination unit 41, an extraction unit 42, a management unit 43, and a conversion unit 44. Note that the determination unit 41, extraction unit 42, management unit 43, and conversion unit 44 are realized by processor 12 executing a program.

[0055] The determination unit 41, extraction unit 42, and management unit 43 are the same as those in the first embodiment, and therefore descriptions thereof will be omitted for the sake of brevity. Note that in the second embodiment, the extraction unit 42 does not need to derive values ​​of adjustment parameters for the hearing aid 3 or the personal speaker.

[0056] The conversion unit 44 acquires the speech of the speaker from the speaker's microphone in real time. Next, the conversion unit 44 converts the speech of the speaker into easy-to-listen speech tailored to the hearing ability of the speaker in real time by changing the frequency, amplitude characteristics, S / N value, speech speed, etc. based on the speech signal characteristics of the speaker. If there are two or more speakers, the conversion unit 44 compares the individual audibility characteristics based on the speech signal characteristics of each speaker and converts the speech so that it is easy to hear for all speakers. The conversion unit 44 also transmits the converted speech to the speaker 4. The speaker 4 plays back the converted speech received from the hearing aid support device 1x and outputs easy-to-listen speech in real time, thereby assisting the speaker in hearing.

[0057] In the above configuration, the conversion unit 44 of the hearing aid support device 1 is an example of the conversion means of the present disclosure.

[0058] (Audio output processing) Next, we will explain the audio output processing by the hearing aid support device 1. Fig. 10 is a flowchart of the audio output processing by the hearing aid support device 1x. This processing is realized by the processor 12 shown in Fig. 3 executing a program prepared in advance.

[0059] First, the hearing aid support device 1x acquires the speech uttered by the speaker (step S301). Next, the hearing aid support device 1x identifies the target person present in the speech transmission area (step S302). Next, the hearing aid support device 1x acquires the speech signal features of the target person from the feature DB 31 based on the person ID of the identified target person (step S303). Next, the hearing aid support device 1x converts the speech uttered by the speaker into easy-to-listen speech that matches the hearing ability of the target person based on the acquired speech signal features (step S304). Next, the hearing aid support device 1x transmits the converted speech to the speaker 4 (step S305). The speaker 4 plays back the converted speech received from the hearing aid support device 1x and outputs easy-to-listen speech in real time, thereby assisting the target person in hearing. This completes the speech output process.

[0060] In this embodiment, the target person is not wearing a hearing aid 3, and the speaker 4 outputs the converted audio received from the hearing aid support device 1x, but this is not limited to this. If the target person is wearing a hearing aid 3, the hearing aid 3 may receive the converted audio from the hearing aid support device 1x and output it as easy-to-listen audio that is tailored to the target person's hearing ability.

[0061] In this embodiment, the speaker 4 is installed on the ceiling of the elderly care facility, but the present invention is not limited to this and the speaker 4 can be installed anywhere, such as an event hall, as long as the target person can hear the speaker. The speaker 4 may be installed one per target person, and may be, for example, a speaker installed in each room of the elderly care facility or a portable personal speaker.

[0062] Furthermore, in this embodiment, the audio acquired from the microphone is converted so that it is easier for the target person to hear, and the converted audio is output in real time from the speaker 4, but this is not limited to this, and the timing at which the converted audio is transmitted to the speaker 4 and the timing at which the converted audio is played back and output from the speaker 4 can be set arbitrarily. For example, the audio of the video used in the feature extraction process in the first embodiment above may be stored, and at an arbitrary timing, the audio may be converted into audio that is easier to hear based on the audio signal features of the target person, and the converted audio may be transmitted to a hearing aid or personal speaker worn by the target person so that the target person can hear it.

[0063] Furthermore, in this embodiment, when there are two or more target persons, the conversion unit 44 converts the voice so that it is easy for all of the target persons to hear. However, this is not limited to this, and the hearing aid support device 1x may set priorities in advance for people who may be target persons, and use the voice signal features of the target person with the highest priority among multiple target persons present in the voice transmission area to convert the voice so that it is easy for the target person with the highest priority to hear.

[0064] Furthermore, in this embodiment, the feature DB31 stores the target person ID and the voice signal feature in association with each other, but this is not limited to this, and the target person ID may also be stored in association with the algorithm used for voice conversion based on the voice signal feature.

[0065] The hearing aid support device 1x of this embodiment can convert speech that the target person could not hear into speech that is easy to hear, based on the target person's speech signal characteristics. When there are multiple target people, the speech that is difficult to hear will each have different speech signal characteristics, but the hearing aid support device 1x can output speech that has been converted to be easy to hear for all multiple target people. In this way, by associating and managing target people with speech signal characteristics that the target people can or cannot hear, the hearing aid support device 1x can convert a specific speech into easy-to-hear speech that matches the target person's hearing ability and output it via a hearing aid or speaker. This can support the target person's hearing.

[0066] [Variations] FIG. 11 is a schematic diagram of a target person walking outdoors. When the hearing aid support device 1x acquires a video of the target person's surroundings from a surveillance camera, smartphone, or the like, it detects the target person's surrounding environment from the video. The conversion unit 44 may then reflect environmental information about the detected environment when converting a predetermined sound into an easy-to-listen sound. Here, the detected environment may be, for example, the target person's surroundings, such as outdoors, crossing an intersection, or the presence of vehicular traffic, as shown in FIG. 11 . The environmental information may be audio signal characteristics of environmental sounds, such as music playing when crossing a crosswalk, or the sound of a car horn or approaching vehicle. The predetermined sound to be converted into an easy-to-listen sound may be acquired, for example, from the target person's hearing aid, personal speaker, smartphone, or the like via a network 5.

[0067] In this way, when detecting an environment where the target person's life is in danger, such as an intersection, the conversion unit 44 performs a predetermined conversion of the audio signal characteristics of the environmental sound. Specifically, the conversion unit 44 converts the audio so that, as an exception, it prioritizes the music playing when crossing the crosswalk or the sound of an approaching car over the audio spoken by the speaker. In other words, the conversion unit 44 does not prioritize the audibility of the speaker's audio, even to the point of muting the audio played when crossing the crosswalk. The converted audio is transmitted in real time to the hearing aid or personal speaker of the person crossing the intersection, so that the target person can hear it. This can also amplify the volume of the sound of a car approaching the target person, allowing the target person to safely receive hearing assistance from the hearing aid support device 1x.

[0068] Furthermore, when the hearing aid support device 1 detects an environment where the target person's life is in danger, such as an intersection, it may derive adjustment parameter values ​​for the hearing aid or personal speaker based on the audio signal features of a sound that should be given priority over the sound spoken by the speaker, and transmit the derived values ​​for adjustment. Here, the sound that should be given priority is not limited to outdoor sounds, but can be any sound, such as an earthquake early warning.

[0069] In addition, some or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following supplementary notes.

[0070] (Appendix 1) a determination means for monitoring a conversation between a target person with hearing problems and a speaker based on a video of the target person, and determining whether the target person can hear the voice uttered by the speaker based on at least one of the target person's appearance and behavior and the content of the target person's response; extraction means for extracting audio signal features of the audio determined to be audible or inaudible; A hearing aid device comprising:

[0071] (Appendix 2) the hearing aid assistive device is communicably connected to an audio playback device; a derivation means for deriving values ​​of adjustment parameters for the audio reproduction device based on the audio signal features extracted by the extraction means; output means for transmitting the value to the audio playback device; 2. A hearing aid support device as described in Supplementary Note 1, comprising:

[0072] (Appendix 3) 3. The hearing aid support device according to claim 2, wherein the audio playback device is a hearing aid worn by the subject person.

[0073] (Appendix 4) 2. A hearing aid support device according to claim 1, further comprising a conversion means for converting and outputting audio using the audio signal features extracted by the extraction means.

[0074] (Appendix 5) The voice signal characteristics of the target person are managed in association with the identification information of the target person, A hearing aid support device as described in Appendix 4, wherein the conversion means identifies a target person present in a sound transmission area and converts the sound using sound signal features corresponding to the identification information of the identified target person.

[0075] (Appendix 6) The hearing aid support device described in Appendix 5, wherein the conversion means generates audio signal features that are easy for multiple target persons to hear based on audio signal features corresponding to each of the multiple target persons present in the audio transmission area, and converts the audio using the generated audio signal features.

[0076] (Appendix 7) Priorities are set for the target persons, A hearing aid support device as described in Appendix 5, wherein the conversion means converts the voice using the voice signal features of the target person with the highest priority among multiple target people present in the voice transmission area.

[0077] (Appendix 8) an environment detection means for detecting an environment in which the target person exists from a peripheral image of the target person; 5. The hearing aid support device according to claim 4, wherein the conversion means converts the voice using voice signal features corresponding to the target person and information about the environment.

[0078] (Appendix 9) A hearing aid support method executed by a hearing aid support device, comprising: Based on a video of a target person with hearing problems, a conversation between the target person and a speaker is monitored, and whether the target person can hear the sound emitted by the speaker is determined based on one or more of the target person's appearance and behavior and the content of the target person's response utterance; A hearing aid support method for extracting audio signal features of the audio determined to be audible or inaudible.

[0079] (Appendix 10) Based on a video of a target person with hearing problems, a conversation between the target person and a speaker is monitored, and whether the target person can hear the sound emitted by the speaker is determined based on one or more of the target person's appearance and behavior and the content of the target person's response utterance; A program that causes a computer to execute a process of extracting audio signal features of the audio that has been determined to be audible or inaudible.

[0080] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. Various modifications that would be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. In other words, the present disclosure naturally includes various modifications and alterations that would be possible for a person skilled in the art in accordance with the entire disclosure, including the claims, and the technical concept. [Explanation of symbols]

[0081] 1. 1x hearing aid device 3. Hearing aids 4 speakers 11 Interface 12 processors 13. Memory 14 Recording media 15 Display section 16 Input section 31 Feature DB 41 Judgment section 42 Extraction part 43 Management Department 44, 51 Conversion unit 100, 200 hearing aid system

Claims

1. a determination means for monitoring a conversation between a target person with hearing problems and a speaker based on a video of the target person, and determining whether the target person can hear the voice uttered by the speaker based on at least one of the target person's appearance and behavior and the content of the target person's response; extraction means for extracting audio signal features of the audio determined to be audible or inaudible; A hearing aid device comprising:

2. the hearing aid assistive device is communicably connected to an audio playback device; a derivation means for deriving values ​​of adjustment parameters for the audio reproduction device based on the audio signal features extracted by the extraction means; output means for transmitting the value to the audio playback device; The hearing aid device according to claim 1 , comprising:

3. The hearing aid device according to claim 2 , wherein the sound reproduction device is a hearing aid worn by the subject.

4. 2. The hearing aid support device according to claim 1, further comprising a conversion means for converting and outputting a voice using the voice signal features extracted by the extraction means.

5. The voice signal characteristics of the target person are managed in association with the identification information of the target person, 5. The hearing aid support device according to claim 4, wherein the conversion means identifies a target person present in a sound transmission area and converts the sound using a sound signal feature corresponding to identification information of the identified target person.

6. The hearing aid support device of claim 5, wherein the conversion means generates audio signal features that are easy for multiple target persons to hear based on audio signal features corresponding to each of the multiple target persons present in the audio transmission area, and converts the audio using the generated audio signal features.

7. Priorities are set for the target persons, 6. The hearing aid support device according to claim 5, wherein the conversion means converts the voice using voice signal features of the target person with the highest priority among a plurality of target people present in the voice transmission area.

8. an environment detection means for detecting an environment in which the target person exists from a peripheral image of the target person; The hearing aid support device according to claim 4 , wherein the conversion means converts the voice using voice signal features corresponding to the target person and information about the environment.

9. A hearing aid support method executed by a hearing aid support device, comprising: Based on a video of a target person with hearing problems, a conversation between the target person and a speaker is monitored, and whether the target person can hear the sound emitted by the speaker is determined based on one or more of the target person's appearance and behavior and the content of the target person's response utterance; A hearing aid support method for extracting audio signal features of the audio determined to be audible or inaudible.

10. Based on a video of a target person with hearing problems, a conversation between the target person and a speaker is monitored, and whether the target person can hear the sound emitted by the speaker is determined based on one or more of the target person's appearance and behavior and the content of the target person's response utterance; A program that causes a computer to execute a process of extracting audio signal features of the audio that has been determined to be audible or inaudible.

Images