Listening assisting device and method integrating image and audio positioning and omnibearing radio array and having warning function

Abstract

The invention discloses an auxiliary hearing device and method integrating image and audio positioning and an omnibearing radio array and having a warning function. The auxiliary hearing device comprises an auxiliary hearing device body, a 360-degree radio unit, a photographing unit, a warning indicator and a control unit. The auxiliary hearing device body is provided with a neck hanging part and a playing unit, the neck hanging part is used for a user to wear, and the playing unit is an earphone. The 360-degree sound receiving unit is used for capturing 360-degree audio at the periphery of the auxiliary hearing device body, and the audio comprises environment sound and voice; the photographing unit is used for photographing a human face image at the periphery of the auxiliary hearing device body; the control unit stores at least one preset face image, at least one preset voice and special environment sound audio, receives the audio and the face image and separates and enhances the voice corresponding to the face image through the computer vision technology, and then the playing unit plays the voice. The control unit controls the warning indicator to send out warning information when the environment sound belongs to the special environment sound audio.

Description

Technical Field

[0001] The present invention relates to an auxiliary listening device and method with warning function that integrates image, audio positioning and omnidirectional sound array, and in particular, refers to an auxiliary listening device and method that can lock the voice of the pronunciation object during the listening stage through 360-degree sound collection and computer vision, and identify special environmental audio such as ambulance alarms in visual blind spots that the user cannot notice and provide warnings. Background Art

[0002] Studies have shown that hearing loss affects diseases related to the nervous system. For example, hearing loss is believed to be positively correlated with the occurrence of Alzheimer's disease. In addition to hearing loss caused by injuries, hearing also deteriorates as the body ages. For people over 60 years old, hearing loss often causes trouble in life. Once the hearing of the elderly deteriorates, they speak less and less. The use of hearing aids can improve hearing loss problems or delay hearing loss. Currently, smart hearing aids can lock and amplify specific human voices, and can also perform effective environmental noise reduction. However, there are no smart hearing aids that can fully lock the voice of the object of pronunciation during listening and timely remind of environmental sounds that may cause danger.

[0003] U.S. Patent No. US20210160624A1 "HEARING AID WITH VOICE OR IMAGE RECOGNITION" discloses that a hearing aid uses a beamforming microphone array to intelligently collect sound, and uses lip tracking to help distinguish and separate voice signals, and notifies the user of the image of the recognized pronunciation target through a display, etc., and can selectively amplify the sound signal associated with the voice of an identified individual. The case also provides a variety of implementation scenarios, such as glasses combined with image sensors, which can be used to capture real-time image data of the user's field of view. The image sensor can be a camera, a mobile phone or a watch, etc. The glasses can also be belt buckles or other clips that can be clipped on vests, pockets, collars, hats, etc.; the case specification paragraph 0196 mentions that the invention can also provide a camera-based directional hearing aid for selectively amplifying sound based on the user's gaze direction, wherein the user's gaze direction is achieved by tracking the optical axis of the image sensor. As for selectively amplifying sound, it includes one or more microphones for capturing sound from the user's environment, wherein the microphone may include one or more directional microphones to be more sensitive to picking up sounds in certain directions, and audio classification technology is used to classify the sounds captured from the user's environment, such as music, tones, laughter, screams, etc. In paragraph 0216 of the case specification, it is mentioned that the hearing aid system can store the voice features and / or facial features of the identified person to help with recognition and selective amplification. For example, when an individual enters the field of view of the device, the individual may be identified as an individual who has been introduced to the device, or an individual who may have interacted with the user in the past (e.g., a friend, colleague, relative, former acquaintance, etc.), and the audio signal associated with the identified individual's voice may be isolated and / or selectively amplified relative to other sounds in the user's environment. However, if the target leaves the field of view during the conversation, the processor mentioned in paragraph 0228 of the invention specification can analyze the target's voice and enable the hearing aid to continue to selectively amplify the target's voice by determining whether the detected voice matches the individual's voiceprint in the database.

[0004] The above-mentioned U.S. Patent No. US20210160624A1 provides a variety of implementation scenarios, and only generally describes the use of neural networks, convolutional neural networks and other technologies for speech separation, but the technical details of the execution steps are not disclosed at all, such as how to configure the array microphone to perform effective sound collection? In addition, how to effectively "all-round" lock the target image and "all-round" lock the target sound? The case also does not provide a technical solution. Furthermore, the patent uses audio classification to classify ambient sounds such as music, tones, laughter, screams, etc., but it fails to issue warning signals in real time to remind users of dangerous ambient sounds.

[0005] In addition, the inventor has filed Taiwan Invention Patent Application No. 110119518, "Method, device, composite microphone of the device, computer program and computer-readable medium for automatically or freely selecting independent pronunciation targets". The case uses high-order beamforming (second order or above) to focus the sound reception and enhance the speech target (preferably the main lobe of the sound reception is within 20 degrees), while capturing the target image of the speech target, and using an intelligent learning algorithm to separate the independent speech corresponding to each target image, especially being able to identify overlapping target images, so as to select different target images to play the corresponding independent speech.

[0006] At present, most of the hearing aids combined with image recognition use visual tracking combined with beamforming for recognition. However, the image and sound capture are targeted at the front of the user, and the ambient sound is usually not processed but directly filtered out. As a result, the images and sounds of the user in other directions (such as the back) are excluded. For example, when the target is not within the user's line of sight, the hearing aid cannot continuously lock onto the target's voice. And because the ambient sound is usually filtered out, when there is an emergency sound behind the user (such as the sound of an ambulance, a fire truck, the horn of a vehicle behind, etc.), the user cannot avoid it in time. Summary of the invention

[0007] Therefore, an object of the present invention is to provide a hearing aid device with warning function that integrates image, audio positioning and omnidirectional sound array.

[0008] The present invention provides a hearing aid device with warning function integrating image, audio positioning and omnidirectional sound array, comprising:

[0009] A hearing aid body is provided with a playback unit; a 360-degree sound receiving unit is provided on the hearing aid body, the 360-degree sound receiving unit includes an array of six sound receiving microphone chips, each sound receiving microphone chip has a sound receiving area, and the connection line between any two of the aforementioned sound receiving areas is respectively located on a sound receiving connection line of 0 degrees, 60 degrees, 90 degrees, 120 degrees, 150 degrees, 180 degrees, 210 degrees, 240 degrees, 270 degrees, 300 degrees and 380 degrees, so as to capture an audio of 360 degrees around the hearing aid body, the audio including an ambient sound and a voice a camera unit, arranged on the hearing aid body, for photographing a face image of a person around the hearing aid body; a warning unit, arranged on the hearing aid body; a control unit, storing at least one preset face image, at least one preset voice and at least one special ambient sound audio, the control unit signal is connected to the playback unit, the 360-degree sound receiving unit, the camera unit and the warning unit, the control unit receives the audio and the face image, and when the face image matches any of the aforementioned preset face images, the control unit controls the camera unit to lock the face image with a second instruction The control unit determines the face image, and controls the 360-degree sound receiving unit to continuously capture the voice located at the shooting direction according to the shooting direction, and the playback unit plays the voice; when there is no face image, or the face image does not match the preset face image, if the voice matches any of the preset voices, the control unit controls the playback unit to play the voice that matches the preset voice with a third instruction; when there is no face image, or the face image does not match the preset face image, and the voice does not match the preset voice, the control unit controls the playback unit to play the voice that matches the preset voice according to the pointing direction. , controlling the 360-degree sound receiving unit to continuously capture the ambient sound and / or the voice at the pointing direction, and the playing unit plays the ambient sound and / or the voice; the control unit also controls the 360-degree sound receiving unit to capture the ambient sound at any position around the hearing aid body at any time, and the control unit compares whether the ambient sound belongs to the special ambient sound audio, and when the ambient sound belongs to the special ambient sound audio, the control unit gives priority to playing the ambient sound, interrupting or reducing the aforementioned voice, and the control unit simultaneously controls the warning unit to issue a warning message in real time.

[0010] Furthermore, the 360-degree sound receiving unit includes six sound receiving microphone chips, and the six sound receiving areas are arranged in three rows on the 360-degree sound receiving unit. The first row includes a first sound receiving area and a second sound receiving area arranged in parallel, the second row includes a third sound receiving area, a fourth sound receiving area and a fifth sound receiving area arranged alternately, and the third row includes a sixth sound receiving area; wherein the sound receiving line formed by the fifth sound receiving area and the sixth sound receiving area is located at the azimuths of 0 degrees and 180 degrees, and the sound receiving line formed by the second sound receiving area and the fourth sound receiving area is located at the azimuths of 180 degrees and 0 degrees. The sound connection line is located at 30 degrees and 210 degrees, the sound connection line formed by the third sound receiving area and the fourth sound receiving area is located at 60 degrees and 240 degrees, the sound connection line formed by the first sound receiving area and the second sound receiving area is located at 90 degrees and 270 degrees, the sound connection line formed by the fourth sound receiving area and the fifth sound receiving area is located at 120 degrees and 300 degrees, and the sound connection line formed by the first sound receiving area and the fourth sound receiving area is located at 150 degrees and 330 degrees.

[0011] Furthermore, when the face image cannot continuously lock onto the voice due to angle deviation, the control unit uses an advanced adaptive combined differential array (DMA) algorithm to lock onto the loudest human voice as the voice, and separates the voice and the ambient sound from the audio.

[0012] Further, the advanced adaptive combined differential array (DMA) algorithm is as follows: S m (t) represents the audio received by the mth microphone chip, Q(t) = [q(t)…q(tK g +1)] T K is the impulse response of speech to the mth microphone chip, showing a zero-mean signal. g A set of column vectors of samples, h m =[h m,1 …h m,Kg ] is represented by K g Column vector of samples, U m (t) is the ambient sound in the audio, and T is the transpose operator; therefore, the advanced adaptive combined differential array (DMA) algorithm can calculate the mth a The mth microphone chip and the a+1 The pointing pattern of the above-mentioned microphone chip is defined by the zero-point steering angle θ0, which is determined by the distance d between the microphone chips. k and delay time τ k Control, as follows: θ0 = cos -1 (-cτ k / d k ), where c represents the speed of sound, and the directional pattern is a cardioid pattern (θ0 = 180, k = dk / c), and then by the compensation factor h k The frequency response is as follows:

[0013] Furthermore, the photographic unit includes two cameras, which are respectively located at the front and rear sides of the neck hanging part.

[0014] Furthermore, the warning unit includes two vibrators, which are respectively located on the left and right sides of the neck hanging part. The control unit identifies the direction of the ambient sound through the 360-degree sound receiving unit, and the control unit independently controls one of the vibrators corresponding to the direction of the ambient sound to vibrate.

[0015] Furthermore, the hearing aid body includes two ear hooks and a neck-mounted part, the 360-degree sound receiving unit and the camera unit are arranged on the neck-mounted part, and the warning unit and the playback unit are arranged on the two ear hooks.

[0016] Furthermore, the hearing aid body includes two ear hooks and a head-mounted part, the 360-degree sound receiving unit and the photography unit are arranged on the head-mounted part, and the warning unit and the playback unit are arranged on the two ear hooks.

[0017] Furthermore, the hearing aid body includes two ear hooks and a hanging part, the hanging part is provided with a detachable hanging rope, wherein the 360-degree sound receiving unit and the photographic unit are arranged on the hanging part, and the warning unit and the playback unit are arranged on the two ear hooks, wherein the hanging part and the two ear hooks are wirelessly connected.

[0018] The present invention also provides an assisted listening method with a warning function that integrates image, audio positioning and an omnidirectional sound array, comprising the following steps: storing a plurality of special ambient sound audios; storing a plurality of preset facial images; storing a plurality of preset voices; using the 360-degree sound receiving unit to continuously collect an ambient sound and a voice, the photographic unit photographs a facial image with a variable shooting orientation or points to a target with a pointing orientation; the control unit also compares the ambient sound with the aforementioned special ambient sound audio, and generates a first instruction when the comparison is consistent; the control unit compares the facial image with the aforementioned preset facial image, and generates a second instruction when the comparison is consistent; the control unit compares the voice with the aforementioned preset voice, and generates a third instruction when the comparison is consistent; the photographic unit activates the pointing orientation and generates a fourth instruction; the first instruction is to command the playback unit to play the ambient sound at that time and generate a warning message; the second instruction is to command the photographic unit to lock the facial image, and command the playback unit to play the voice located at the shooting orientation; the The third instruction is to command the playback unit to play the voice that matches the aforementioned preset voice; the fourth instruction is to command the playback unit to play the ambient sound and / or the voice located in the pointed direction; the first instruction, the second instruction, the third instruction and the fourth instruction are executed according to an execution order or a mobile order, wherein the execution order is that the first instruction takes precedence over the second instruction, the second instruction takes precedence over the third instruction, and the third instruction takes precedence over the fourth instruction; the mobile order is that the first instruction has the highest priority, and the fourth instruction takes precedence over the second instruction and the third instruction; when at least two of the first instruction, the second instruction, the third instruction and the fourth instruction exist, the control unit executes the existing one of the first instruction, the second instruction, the third instruction and the fourth instruction according to the aforementioned execution order or the mobile order; when only one of the first instruction, the second instruction, the third instruction and the fourth instruction exists, the control unit only executes the existing one of the first instruction, the second instruction, the third instruction and the fourth instruction.

[0019] The aforementioned warning information causes the warning unit to vibrate.

[0020] Furthermore, when there are two or more facial images that match the aforementioned preset facial images, the second instruction commands the photographic unit to lock the facial image at a shooting position closest to the photographic unit, and commands the playback unit to play the voice at the shooting position closest to the shooting position.

[0021] Furthermore, when the photographic unit fails to lock the face image, the control unit locks the voice with the loudest volume at that time with an advanced adaptive combined differential array (DMA) algorithm, and controls the playback unit to play the voice with the loudest volume.

[0022] Furthermore, when no face image matches the aforementioned preset face image, the second instruction is to instruct the playback unit to play the voice closest to the pointing direction. When two or more voices match the aforementioned preset voices, the second instruction is to instruct the playback unit to play the voice closest to the pointing direction, wherein after the camera unit activates the pointing direction, it further executes a pointing direction priority command, and the control unit changes the execution order to the maneuvering order according to the pointing direction priority command.

[0023] The following effects can be achieved according to the above technical features:

[0024] 1. The present invention can perform 360-degree sound collection through a 360-degree sound collection unit, and in combination with a camera unit to dynamically track the pronunciation object to lock the pronunciation source, so that the face image can be locked within the shooting range of the camera unit through computer vision, and the voice of the pronunciation object can be continuously captured without interruption. Even if the pronunciation object is in a direction that the user cannot see, such as the left or right side or the back, the user does not need to turn his head at all. As long as the pronunciation object makes a sound, the voice of the pronunciation object can be continuously captured through the 360-degree sound collection unit.

[0025] 2. When the user is in a street environment, the present invention can provide real-time warnings by identifying special ambient audio and video (such as ambulance alarms, car horns, etc.) from the sides or rear that the user cannot notice.

[0026] 3. The present invention uses the six sound receiving areas of the array so that at least two sound receiving areas are located on each sound receiving line of each main angle, thereby achieving a 360-degree omnidirectional sound receiving function.

[0027] 4. When the face image of the present invention cannot continuously lock the voice due to certain angle deviations, the control unit of the present invention continuously locks the voice with the loudest volume using an advanced adaptive combined differential array (DMA) algorithm to prevent the voice from being interrupted. The user can also activate the pointing direction of the camera unit at any time to lock the voice in any direction.

[0028] 5. Dangerous ambient sounds are always played first. Facial images and voices of acquaintances are pre-stored, so you can quickly lock on to acquaintances during a conversation. When there are multiple acquaintances in the conversation, you can also select the pronunciation object based on the camera's shooting direction. Even if the acquaintance is outside the shooting angle, such as on the side or back, the 360-degree sound receiving unit can also perform targeted sound reception through comparison with pre-stored voices for the user to have a conversation.

[0029] 6. Using the method provided by the present invention, for example, acquaintances with whom you often talk can pre-store their face images and voices, while ordinary friends can only pre-store their voices. The present invention will determine the priority of the face images captured and the voices recorded in the conversation according to the "execution order", and the order is: environmental dangerous sounds, acquaintances with pre-stored face images, ordinary friends with pre-stored voices, and strangers. Through the "execution order" method, the most frequent interlocutors can be quickly locked in the conversation, increasing the efficiency of the conversation.

[0030] 7. The method provided by the present invention can also enable smooth conversations with strangers. Although the facial images and voices of strangers are not pre-stored, in conversations between acquaintances and strangers, the pointing direction priority command can be activated by the photographic unit, allowing the user to choose to talk to strangers or specific acquaintances first, which can fully meet the needs of actual conversations. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a three-dimensional appearance diagram of the hearing aid of the present invention, wherein the hearing aid body includes a neck-mounted portion and an ear-hook portion.

[0032] Figure 1A It is a schematic diagram of the configuration of the 360-degree sound receiving unit and the circuit board of the present invention.

[0033] Figure 1B It is a three-dimensional appearance diagram of another embodiment of the hearing aid of the present invention, wherein the hearing aid body includes a head-mounted portion and an ear-hook portion.

[0034] Figure 1C It is a three-dimensional appearance diagram of another embodiment of the hearing aid of the present invention, wherein the hearing aid body includes a wearing portion and an ear hook portion.

[0035] Figure 1D It is a schematic diagram of another embodiment of the hearing aid of the present invention, in which the wearing part and the ear hook part of the hearing aid body are separated from each other and used in a classroom.

[0036] Figure 2 It is a schematic diagram of component connections of the hearing aid of the present invention.

[0037] Figure 3 FIG. 4 is a schematic diagram of a user wearing a hearing aid in an embodiment of the present invention.

[0038] Figure 4 It is a schematic diagram of the arrangement array of the sound receiving microphone chips in the 360-degree sound receiving unit of the hearing aid of the present invention.

[0039] Figure 5 It is a schematic diagram of a radio microphone chip using an advanced adaptive combined differential array (DMA) algorithm in an embodiment of the present invention.

[0040] Figure 6 Schematic diagram of testing the sound receiving microphone chip to detect sound sources at different angles in an embodiment of the present invention.

[0041] Fig. 7A for Figure 6 The test results show that the sound source can be effectively detected based on the signal peak at an angle of 30 degrees.

[0042] Figure 7B for Figure 6 The test results show that the sound source can be effectively detected based on the signal peak at an angle of 60 degrees.

[0043] Figure 7C for Figure 6 The test results show that the sound source can be effectively detected based on the signal peak at a 90-degree angle.

[0044] Figure 8 for Figure 6 The test results show the success rate of the microphone chip in detecting sound sources at different angles.

[0045] Fig. 9 This is a test schematic diagram of using an advanced adaptive combined differential array (DMA) algorithm to separate speech and ambient sound at different angles in an embodiment of the present invention.

[0046] Fig.10 for Fig. 9 The test results show the success rate of voice separation when the voice and ambient sound are 330 and 60 degrees apart and at different distances from the microphone chip.

[0047] Fig.11 The success rates of speech separation of the hearing aid of the embodiment of the present invention and the existing hearing aid in different directions.

[0048] Fig.12 It is a schematic diagram of an implementation of the auxiliary listening method with warning function integrating image, audio positioning and omnidirectional sound receiving array of the present invention.

[0049] Explanation of reference numerals: 1, 1A, 1B-body of hearing aid; 10, 10A, 10B-ear hanging part; 11-neck hanging part; 11A-head hanging part; 11B-wearing part; 111B-hanging rope; 12, 12A, 12B-playing unit; 2, 2A, 2B, 2C-360-degree sound receiving unit; 20, 20A, 20B-circuit board; 21-sound receiving microphone chip; 211-sound receiving area; 212-first side; 213-second side; 214-shielded area; 3, 3A, 3B, 3C-photographic unit; 31-camera; 31A, 31B-360-degree panoramic camera; 4, 4A, 4B-warning unit; 41, 41A, 41B-vibrator; 5, 5C-control unit; 6-sound object; 7-noise source; 8-battery; 901C-special environmental audio; 902C-preset face image; 903C-preset voice; 904C-ambient sound; 905C-voice; 906C-shooting direction; 907C-face image; 908C-pointing direction; 909C-target; 910C-object; 911C-object; 912C-object; 913C-first instruction; 914C-second instruction; 915C-third instruction; 916C-fourth instruction; 917C-execution order; 918C-maneuver order. DETAILED DESCRIPTION

[0050] The following embodiments are merely used to assist in explaining the hearing aid device with warning function that integrates image, audio positioning and omnidirectional sound array of the present invention, and are not intended to limit the present invention.

[0051] See also Figure 1 , Figure 1A , Figure 2 and Figure 3 As shown, the hearing aid of this embodiment includes a hearing aid body 1 , a 360-degree sound receiving unit 2 , a camera unit 3 , an alarm unit 4 and a control unit 5 .

[0052] The hearing aid body 1 includes two ear hooks 10, a neck hook 11 and a playback unit 12. The neck hook 11 connects the two ear hooks 10, wherein the neck hook 11 is used to be hung on the user's neck, and the ear hooks 10 are used to be hung on the user's ears respectively. The playback unit 12 and the warning unit 4 of this embodiment are respectively arranged on the ear hooks 10.

[0053] See also Figure 1 , Figure 1A and Figure 4As shown, the 360-degree sound receiving unit 2 is disposed on the hearing aid body 1 to capture an audio signal within 360 degrees of the periphery of the hearing aid body 1, and the audio signal includes an ambient sound and a voice. The 360-degree sound receiving unit 2 includes a circuit board 20, and preferably the circuit board 20 is horizontally configured during implementation, so that the circuit board 20 can receive sound at 360 degrees in a horizontal state, but the configuration of the circuit board 20 is not limited to being completely horizontal. In this embodiment, the 360-degree sound receiving unit 2 is pivotally connected to one of the end positions of the neck hanging portion 11, and the angle between the 360-degree sound receiving unit 2 and the neck hanging portion 11 can be adjusted so that the above-mentioned circuit board 20 can be close to horizontal. The 360-degree sound receiving unit 2 includes six sound receiving microphone chips 21, which are implanted on a circuit board 20. Each sound receiving microphone chip 21 has a sound receiving area 211. The sound receiving microphone chip 21 has a first side 212 and a second side 213 opposite to each other, and the sound receiving area 211 is close to the first side 212. The six sound receiving microphone chips 21 are arranged in three rows, with two sound receiving microphone chips 21 in the first row, three sound receiving microphone chips 21 in the second row, and one sound receiving microphone chip 21 in the third row. ; Two adjacent sound microphone chips 21 in the first row are arranged adjacent to each other with a first side 212; among the adjacent three sound microphone chips 21 in the second row, the central sound microphone chip 21 is adjacent to the sound microphone chip 21 in the first row with a first side 212, the sound microphone chips 21 on the opposite sides are adjacent to the sound microphone chip 21 in the first row with a second side 213, and the sound microphone chip 21 in the third row is aligned with one of the two sound microphone chips 21 in the first row, and is arranged on the opposite side to the aligned sound microphone chip 21 in the first row. Specifically, the six sound receiving areas are arranged in three rows on the circuit board 30, the first row has a first sound receiving area 2111 and a second sound receiving area 2112 arranged in parallel, the second row has a third sound receiving area 2113, a fourth sound receiving area 2114 and a fifth sound receiving area 2115 arranged alternately, and the third row has a sixth sound receiving area 2116; wherein the sound receiving line formed by the fifth sound receiving area 2115 and the sixth sound receiving area 2116 is located at 0 degrees and 180 degrees, and the sound receiving line formed by the second sound receiving area 2112 and the fourth sound receiving area 2114 is located at 30 degrees. The sound reception line formed by the third sound reception area 2113 and the fourth sound reception area 2114 is located at an orientation of 60 degrees and 240 degrees, the sound reception line formed by the first sound reception area 2111 and the second sound reception area 2112 is located at an orientation of 90 degrees and 270 degrees, the sound reception line formed by the fourth sound reception area 2114 and the fifth sound reception area 2115 is located at an orientation of 120 degrees and 300 degrees, and the sound reception line formed by the first sound reception area 2111 and the fourth sound reception area 2114 is located at an orientation of 150 degrees and 330 degrees.The sound receiving line formed by the first sound receiving area 2111 and the fourth sound receiving area 2114 passes through the sixth sound receiving area 2116 at the same time, but whether the sound receiving line passes through the sixth sound receiving area 2116 or not, it does not affect the sound receiving effect of the sound receiving lines at 150 degrees and 330 degrees. Figure 4 The array configuration shown has more directions of sound receiving lines, such as the sound receiving line between the first sound receiving area 2111 and the third sound receiving area 2113, the sound receiving line between the first sound receiving area 2111 and the fifth sound receiving area 2115, the sound receiving line between the third sound receiving area 2113 and the sixth sound receiving area 2116, etc. The above description only lists the sound receiving lines in the main directions, and the directions of the sound receiving lines of the present invention are not limited to the above. Preferably, the 0 degree direction of the above sound receiving lines is the direction in which the user's line of sight is directly looking. In addition, it is preferred that the six microphone chips 21 have the same size, which is conducive to modular manufacturing. By using the above-mentioned matrix arrangement, only six microphone chips 21 are needed, and a very small arrangement area on the circuit board 20 is used to have a full range of sound reception effects, and the main angles such as 0 degrees, 30 degrees, 60 degrees, 90 degrees, 120 degrees, 150 degrees, 180 degrees, 210 degrees, 240 degrees, 270 degrees, 300 degrees, and 330 degrees can include at least two sound reception areas 211. However, the six microphone chips 21 having the same size is not the only embodiment, and microphone chips 21 of different sizes can also be used when necessary.

[0054] See also Figure 1 , Figure 1A , Figure 2 and Figure 3 As shown, the photographic unit 3 is pivotally connected to the hearing aid body 1 for photographing a facial image of a person at the periphery of the hearing aid body 1. In this embodiment, the photographic unit 3 includes two cameras 31, each camera 31 has a wide-angle shooting angle, for example, 180 degrees. The two cameras 31 are pivotally connected to the other end position of the neck hanging part 11 and the center of the neck hanging part 11, respectively, wherein the camera 31 pivotally connected to the other end position of the neck hanging part 11 photographs the facial image of the user in front, and the camera 31 pivotally connected to the center position of the neck hanging part 11 photographs the facial image of the user behind. The angle between the two cameras 31 and the neck hanging part 11 can be adjusted during implementation. The photographic unit 3 of the present invention can also use more than three cameras 31 equidistantly arranged on the neck hanging part 11.

[0055] The warning unit 4 is disposed on the ear hook portion 10 of the hearing aid body 1. In this embodiment, the warning unit 4 includes two vibrators 41, and the two vibrators 41 are respectively located on the two ear hook portions 11. The present invention can also use more than three vibrators 41, and the vibrators 41 are disposed on the ear hook portion 10 and the neck hook portion 11.

[0056] The control unit 5 includes a processing unit and a storage unit, wherein the processing unit is used to process instructions and operations, and the storage unit stores at least one preset face image and at least one preset voice and special environmental audio. The control unit 5 signal of this embodiment is connected to the playback unit 12, the 360-degree sound receiving unit 2, the photographic unit 3 and the warning unit 4, and a battery 8 is used to connect the control unit 5, the playback unit 12, the 360-degree sound receiving unit 2, the photographic unit 3 and the warning unit 4 to provide the necessary power for operation. The control unit 5 and the battery 8 can be arranged together with the aforementioned 360-degree sound receiving unit 2. The photographic unit 3 locks a pronunciation object 6 (displayed on Fig. 9 ), the photographic unit 3 dynamically tracks and photographs the pronunciation object 6, and continues to photograph the facial image, the 360-degree sound receiving unit 2 cooperates with the facial image 6 to continuously capture the voice of the pronunciation object 6, and the playback unit 12 plays it. Specifically, the control unit 5 receives the audio and the facial image and separates and enhances the voice corresponding to the facial image through computer vision technology, and then the playback unit 12 plays the voice to prompt the identity of the pronunciation object. In the above-mentioned embodiment, the 360-degree sound receiving unit 2 captures the voice during the locking period based on the photography unit 3 locking the pronunciation object 6, but in addition to this, based on the pre-stored special ambient sound audio, the 360-degree sound receiving unit 2 further captures the ambient sound "at any time", and the control unit 5 compares whether the ambient sound belongs to the special ambient sound audio, and controls the warning unit 4 to issue a warning message when the ambient sound belongs to the special ambient sound audio. For example, when the special ambient sound audio comes from the left rear of the user, the control unit 5 controls the vibrator 41 on the left to vibrate, and the user can dodge to the right according to the warning information of the vibration on the left and observe the sound source of the special ambient sound audio on the left rear (such as police car sirens, ambulance sirens, etc.); and when the special ambient sound audio comes from the right rear of the user, the control unit 5 controls the vibrator 41 on the right to vibrate, and the user can dodge to the left according to the warning information of the vibration on the right and observe the sound source of the special ambient sound audio on the right rear.

[0057] In different embodiments, the control unit 5 may also be a smart phone, a smart watch or a tablet, etc., which is connected to the playback unit 12, the 360-degree sound receiving unit 2, the camera unit 3 and the warning unit 4 via an APP to wirelessly transmit instructions and information.

[0058] like Figure 1BAs shown, in different embodiments, the hearing aid body 1A includes two ear hooks 10A and a head-mounted portion 11A. The head-mounted portion 11A is for being mounted on the top of the user's head, and the two ear hooks 10A are mounted on the user's ears. The photographic unit 3A and the 360-degree sound receiving unit 2A are connected to the top of the head-mounted portion 11A with a ball joint, and can be adjusted in multiple directions. The vibrator 41A of the playback unit 12A and the warning unit 4A are respectively arranged on the ear hooks 10A. The above-mentioned 360-degree sound receiving unit 2A also includes a horizontally arranged circuit board 20A, and the circuit board 20A also includes six sound receiving microphone chips (not marked with symbols) in the aforementioned array manner. The above-mentioned photographic unit 3A is a 360-degree panoramic camera 31A, which can capture 360-degree panoramic images.

[0059] See also Figure 1C , showing another embodiment of the present invention, the hearing aid body 1B includes two ear hooks 10B and a wearing part 11B, the wearing part 11B is provided with a detachable hanging rope 111B for being mounted on the neck of the user, the wearing part 11B is located in front of the chest of the user after being worn, and the two ear hooks 10B are hung on the ears of the user, and the two ear hooks 10B are connected to the wearing part 11B wirelessly via Bluetooth or WiFi. The photographic unit 3B and the 360-degree sound receiving unit 2B are arranged in the wearing part 11B, and the playback unit 12B and the vibrator 41B of the warning unit 4B are respectively arranged in the ear hooks 10B. The above-mentioned 360-degree sound receiving unit 2B also includes a horizontally arranged circuit board 20B, and the circuit board 20B also includes six sound receiving microphone chips (not marked with symbols) in the aforementioned array manner. The above-mentioned photographic unit 3B is a 360-degree panoramic camera 31B, which can capture 360-degree panoramic images. Please refer to Figure 1D , indicating the situation in which the present invention is used in a classroom. The two ear hooks 10B are still hung on the user's ears, and the wearing part 11B is placed far away from the user, such as a desk. The 360-degree panoramic camera 31B is used to lock the pronunciation object 6, such as the teacher's facial image, and the 360-degree sound receiving unit 2B captures the teacher's voice. Except for dangerous ambient sounds, all sounds other than the teacher will be filtered out. Through the wireless transmission between the two ear hooks 10B and the wearing part 11B, the teacher's voice can be transmitted to the playback unit 12B of the two ear hooks 10B. As the teacher walks in the classroom, the teacher's facial image captured by the 360-degree panoramic camera 31B will change, Figure 1DThe arrow direction indicates the teacher's walking path. Depending on the teacher's position, the side face image and the back head image may not be used to lock the voice, but at this time, the loudest human voice is used as a condition, and the 360-degree sound receiving unit 2B can still continuously capture the teacher's voice. That is, the two ear hooks 10B and the one wearing part 11B of this embodiment are wirelessly connected. In addition to using the wearing rope 111B to wear on the chest for the user to use in daily conversation, the wearing part 11B can also be placed outside the user's body where the wireless connection is accessible, such as in classrooms, lecture halls, etc., to assist in locking the voice of the pronunciation object 6 and avoid interference from noise.

[0060] See also Figure 5 As shown, when the face image cannot continuously lock the voice due to some angle deviation, the control unit 5 can use an advanced adaptive combined differential array (DMA) algorithm to lock the human voice with the loudest volume as the voice, and separate the voice and the ambient sound from the audio. The advanced adaptive combined differential array (DMA) algorithm is as follows: S m (t) represents the audio received by the mth microphone chip 21, Q(t) = [q(t)…q(tK g +1)] T K is the impulse response of the speech to the mth microphone chip 21, and is presented as a zero-mean signal. g A set of column vectors of samples, h m =[h m,1 …h m,Kg ] is represented by K g Column vector of samples, U m (t) is the ambient sound in the audio, and T is the transpose operator; therefore, the advanced adaptive combined differential array (DMA) algorithm can calculate the mth a The mth microphone chip 21 and the mth a+1 The pointing pattern of the above-mentioned microphone chip 21 is defined by the zero-point steering angle θ0, and the angle is defined by the distance d between the microphone chips 21. k and delay time τ k Control, as follows: θ0 = cos -1 (-cτ k / d k ), where c represents the speed of sound, and the directional pattern is a cardioid pattern (θ0 = 180, k = d k / c), and then by the compensation factor h k The frequency response is as follows: The speech is separated from the obtained audio by the above algorithm.

[0061] See also Figures 6 to 8 As shown, the sound receiving microphone chip 21 of this embodiment adopts a micro-electromechanical (MEMS) omnidirectional microphone. Through the special arrangement of six sound receiving microphone chips 21, the function of 360-degree accurate sound receiving can be ensured.

[0062] In the case where the face image cannot be continuously locked to the voice due to some angle deviation, the ability of the 360-degree sound receiving unit 2 of this embodiment to identify the sound source with the loudest volume is tested. The sound source is turned on for 5 seconds at every 30 degrees from 0 degrees in the 360-degree circle at 100 cm, 130 cm and 160 cm away from the 360-degree sound receiving unit 2, and the MATLAB TM The Hilbert transform written in the code processes the sound signal from each angular position to calculate the amplitude of the sound signal and find the peak. 7A to 7C As shown, when the sound source is turned on at an angle of 30 degrees, the peak value between 5 seconds and 10 seconds is greater than the average value of the entire sound wave. When the sound source is turned on at an angle of 60 degrees, the peak value between 10 seconds and 15 seconds is greater than the average value of the entire sound wave. When the sound source is turned on at an angle of 90 degrees, the peak value between 15 seconds and 20 seconds is greater than the average value of the entire sound wave.

[0063] Among them, the detection success rate of sound signals at different angles is calculated as: Among them, N s is each θ t is the number of successful detections at an angle of 30 degrees, N T is the total number of experiments. In this embodiment, N T For 100 experiments, Figure 8 The detection success rate of most angles is above 90%, and although the success rate at 60 degrees, 120 degrees, 240 degrees and 300 degrees drops slightly, it is still above 80%.

[0064] See also Figures 9 and 10 As shown, when the face image cannot continuously lock the voice due to some angle deviation, the performance of the 360-degree sound receiving unit 2 of this embodiment in separating the voice of the pronunciation object 6 and the ambient sound of the noise source 7 from different positions is tested. The pronunciation object 6 is set at the 0-degree angle position of the 360-degree sound receiving unit 2, and the noise source 7 is respectively set at the 60-degree angle position and the 330-degree angle position of the 360-degree sound receiving unit 2, and the pronunciation object 6 and the noise source 7 are respectively at the same distance from the 360-degree sound receiving unit 2. Fig.10The success rate of detecting speech is presented at different distances (100 cm, 120 cm, 140 cm and 160 cm) between the pronunciation object 6 and the noise source 7 and the 360-degree sound receiving unit 2. The success rate of detecting speech at the 60-degree angle position is: 86% at 100 cm, 89% at 120 cm, 93% at 140 cm, and 97% at 160 cm; the success rate of detecting speech at the 330-degree angle position is: 81% at 100 cm, 83% at 120 cm, 88% at 140 cm, and 89% at 160 cm. The above experiment shows that the 360-degree sound receiving unit 2 of this embodiment can effectively separate speech and ambient sound.

[0065] See also Fig.11 As shown, when the hearing aid of this embodiment is compared with the existing hearing aid, when the voice is respectively in front of the user P1, 30 degrees to the left front P2, 30 degrees to the right front P3, 30 degrees to the left rear P4, and 30 degrees to the right rear P5, the hearing aid of this embodiment can effectively lock the voice, while the existing hearing aid cannot lock the voice behind the user.

[0066] See also Fig.12 The present invention is also a method for assisting hearing with a warning function that integrates image, audio positioning and an omnidirectional sound array, comprising the following steps:

[0067] The storage unit is used to store a plurality of special ambient audios 901C, a plurality of preset face images 902C and a plurality of preset voices 903C as comparison samples. The special ambient audios 901C are, for example, the sound of an ambulance, a vehicle horn, a fire truck, and a police car. The preset face image 902C can be a face image of the most frequent interlocutor, and each of the opposite faces can be multiple, each with different shooting angles or different expressions. The preset voices 903C include a pre-recorded voice of each of the most frequent interlocutors and general interlocutors.

[0068] The 360-degree sound receiving unit 2C continuously collects an ambient sound 904C and a voice 905C, and the camera unit 3C uses a variable shooting orientation 906C to shoot a face image 907C or points to a target 909C in a pointing orientation 908C. The shooting orientation 906C refers to the orientation of the camera unit 3C when it is in the shooting stage. The shooting orientation 906C is controlled by the user. The user controls the direction of the camera unit 3C to point the shooting orientation 906C to the object 910C in the conversation. Fig.12There are three objects 910C, 911C, and 912C, wherein object 910C is, for example, a familiar friend, with a facial image and voice pre-stored in the storage unit, object 911C is, for example, a stranger, with no facial image and voice pre-stored in the storage unit, and object 912C is, for example, another familiar friend, with a facial image and voice pre-stored in the storage unit. In this embodiment, the user is having a conversation with two familiar friends and a stranger. The above-mentioned pointing direction 908C refers to the direction to which the shooting unit 3C points during the period when the shooting unit 3C is manually stopped, and the pointing direction 908C is controlled by the user. The above-mentioned target 909C refers to the source of a sound, such as a human voice or an animal sound, etc. The target 909C in this embodiment is the voice 905C of the stranger object 911C.

[0069] The control unit 5C compares the ambient sound 904C with the aforementioned special ambient sound audio 901C, and generates a first instruction 913C when the comparison matches. In this embodiment, a certain ambient sound 904C is the sound of an ambulance, and the 360-degree sound receiving unit 2C captures the sound of the ambulance in real time. The control unit 5C compares the ambient sound 904C with the pre-stored special ambient sound audio 901C. The ambient sound 904C matches one of the many special ambient sound audios 901C, and the control unit 5C generates a first instruction 913C accordingly.

[0070] The control unit 5C compares the face image 907C with the preset face image 902C, and generates a second instruction 914C when the comparison is consistent. In the conversation scenario, the user is talking to three people. The object 910C and the object 912C are both familiar friends, and the preset face image 902C contains the data of the two people. The object 911C is a stranger, and the preset face image 902 does not have the data of the person. During the conversation, the faces of the three objects 910C, 911C, and 912C are all captured by the camera unit 3C, but the second instruction 914C is only generated when the face images 907C of the objects 910C and 912C are captured.

[0071] The control unit 5C compares the voice 905C with the preset voice 903C and generates a third instruction 915C when the comparison matches. In the conversation, there are three subjects 910C, 911C, and 912C. The preset face image 902C stores data of the two subjects 910C and 912C, but does not store data of the subject 911C. Therefore, the 360-degree sound receiving unit 2C generates the third instruction 915C only when the voice 905C of the two subjects 910C and 912C is captured.

[0072] The camera unit 3C activates the pointing direction 908C and generates a fourth instruction 916C. The user can control the camera unit 3C to stop capturing the face image 907C at any time. At this time, the direction of the lens optical axis of the camera unit 3C can still be transmitted to the control unit 5C, for example, the direction sensor built into the camera unit 3C transmits the information of the lens optical axis direction. At that time, the direction of the lens optical axis of the camera unit 3C is the above-mentioned pointing direction 908C. After the control unit 5C grasps the pointing direction 908C of the camera unit 3C, the control unit 5C can filter out the voice 905C of the pointing direction 908C from the 360-degree sound receiving unit 2C.

[0073] The first instruction 913C is to instruct the playback unit to play the ambient sound 904C at that time and generate a warning message. During the conversation, general ambient sounds 904C such as wind noise and noise will be filtered out, so that the user can focus on the voice 905C of the interlocutor. However, if an ambulance siren sounds, the first instruction 913C will be triggered, and the playback unit will immediately play the ambulance siren, and the warning unit will also vibrate to remind the user. At this time, the voice 905C in the conversation will be temporarily interrupted or reduced in volume.

[0074] The second instruction 914C is to instruct the camera unit 3C to lock the face image 907C and instruct the playback unit to play the voice 905C located at the shooting direction 906C. In this embodiment, the shooting direction 906C of the camera unit 3C points to the object 910C, the object 910C is talking to the user, and the face of the object 910C is also facing the user. At this time, the camera unit 3C will lock the face image 907C of the object 910C, and the playback unit will also play the voice 907C of the object 910C. In this stage, even if the other two objects 911C and 912C are talking to each other, the playback unit will not play the voice of the objects 911C and 912C. In addition, if two familiar objects 910C and 912C speak at the same time, since the object 910C is closer to the shooting direction 906C, it means that the user is more concerned about the object 910C at this time, so the object 910C will be selected as the dialogue partner first. During the dialogue phase, if no face image 907C matches the preset face image 902C, the second instruction 914C will instruct the playback unit to play the voice 905C closest to the current pointing direction 908C. If during the dialogue phase, if there are more than two voices 905C that match the preset voice 903C, the second instruction 914C will instruct the playback unit to play the voice closest to the pointing direction 908C.

[0075] The third instruction 915C is to instruct the playback unit to play the voice 905C that matches the preset voice 903C. When the subject 910C is talking to the user, but the camera unit 3C cannot completely capture the face image 907C of the subject 910C, for example, the subject 910C is wearing a mask or facing the user with the side face, etc., because the voice 905C of the subject 910C is matched with the preset voice 903C, the third instruction 915C will instruct the playback unit to play the voice 905C of the subject 910C. Alternatively, if the interlocutor is an ordinary friend, the preset face image 902C does not have the data of the person pre-stored, but the preset voice 903C has the data of the person pre-stored, then the third instruction 915C will instruct the playback unit to play the voice 905C of the friend in the conversation.

[0076] The fourth instruction 916C is to instruct the playback unit to play the ambient sound 904C and / or the voice 905C located at the pointing direction 908C. This method retains the function of the user actively controlling the sound collection direction. The user can temporarily turn off the face shooting of the camera unit 3C at any time, so that the camera unit 3C is temporarily used for pointing. The camera unit 3C points to any target 909C with the pointing direction 908C, so that the playback unit plays the ambient sound 904C, the voice 905C of a stranger, etc. located at the pointing direction 908C through the fourth instruction 916C.

[0077] The first instruction 913C, the second instruction 914C, the third instruction 915C and the fourth instruction 916C are executed according to an execution order 917C or a flexible order 918C. The execution order 917C is that the first instruction 913C takes precedence over the second instruction 914C, the second instruction 914C takes precedence over the third instruction 915C, and the third instruction 915C takes precedence over the fourth instruction 916C. The flexible order 918C is that the first instruction 913C has the highest priority, and the fourth instruction 916C takes precedence over the second instruction 914C and the third instruction 915C.

[0078] When at least two of the first instruction 913C, the second instruction 914C, the third instruction 915C and the fourth instruction 916C exist, the control unit 5C executes the first instruction 913C, the second instruction 914C, the third instruction 915C and the fourth instruction 916C according to the aforementioned execution sequence 917C or the mobile sequence 918C.

[0079] When only one of the first instruction 913C, the second instruction 914C, the third instruction 915C and the fourth instruction 916C exists, the control unit 5C only executes the existing one among the first instruction 913C, the second instruction 914C, the third instruction 915C and the fourth instruction 916C.

[0080] Through the execution sequence 917C, the voice 905C of a familiar friend can be screened and played first, and the voice 905C of a general friend can also be screened and played before the voice 905C of a stranger. Through the mobile sequence 918C, the user can decide to screen and play the voice 905C of any one of the familiar friends, general friends, and strangers first. However, no matter whether the execution sequence 917C or the mobile sequence 918C is used, the dangerous ambient sound 904C can be played with the highest priority.

[0081] The method and device provided by the present invention can provide a more intelligent listening assistance process and give real-time prompts for dangerous sounds in the environment. When used in the classroom, it can also effectively help learners focus only on the teacher's voice without being affected by other noises, which helps to improve learning concentration.

[0082] Based on the description of the above embodiments, the operation, use and effects of the present invention can be fully understood. However, the above embodiments are only preferred embodiments of the present invention and should not be used to limit the scope of the present invention. That is, simple equivalent changes and modifications made according to the claims and the description of the present invention are all within the scope of the present invention.

Claims

1. A hearing aid device with warning function integrating image, audio positioning and omnidirectional sound array, characterized in that: include: A hearing aid body is provided with a playback unit; a 360-degree sound receiving unit, disposed on the hearing aid body, the 360-degree sound receiving unit comprising an array of six sound receiving microphone chips, each of which has a sound receiving area, and a line connecting any two of the aforementioned sound receiving areas is located on a sound receiving line of 0 degrees, 60 degrees, 90 degrees, 120 degrees, 150 degrees, 180 degrees, 210 degrees, 240 degrees, 270 degrees, 300 degrees and 380 degrees, respectively, so as to capture an audio of 360 degrees around the hearing aid body, the audio including an ambient sound and a voice; A camera unit is disposed on the hearing aid body, and is used to capture a face image of a person around the hearing aid body in a shooting direction, or to point to any target around the hearing aid body in a pointing direction; An alarm unit is provided on the hearing aid body; A control unit stores at least one preset face image, at least one preset voice and at least one special ambient sound audio. The control unit is connected to the playback unit, the 360-degree sound receiving unit, the camera unit and the warning unit by signal. The control unit receives the audio and the face image. When the face image matches any of the aforementioned preset face images, the control unit controls the camera unit to lock the face image with a second instruction, and controls the 360-degree sound receiving unit to continuously capture the voice located at the shooting position according to the shooting direction, and the playback unit plays the voice. When there is no face image, or the face image does not match the preset face image, if the voice matches any of the preset voices, the control unit controls the playback unit to play the voice with a third instruction. the voice that matches the preset voice; when there is no facial image, or the facial image does not match the preset facial image, and the voice does not match the preset voice, the control unit controls the 360-degree sound receiving unit to continuously capture the ambient sound and / or the voice located at the pointing direction according to the pointing direction, and the playback unit plays the ambient sound and / or the voice; the control unit also controls the 360-degree sound receiving unit to capture the ambient sound at any position around the hearing aid body at any time, the control unit compares whether the ambient sound belongs to the special ambient sound audio, and when the ambient sound belongs to the special ambient sound audio, the control unit gives priority to playing the ambient sound, interrupting or reducing the aforementioned voice, and the control unit simultaneously controls the warning unit to issue a warning message in real time.

2. The hearing aid device with warning function integrating image, audio positioning and omnidirectional sound array as claimed in claim 1, characterized in that: The six sound receiving areas are arranged in three rows on the 360-degree sound receiving unit, the first row has a first sound receiving area and a second sound receiving area arranged in parallel, the second row has a third sound receiving area, a fourth sound receiving area and a fifth sound receiving area arranged alternately, and the third row has a sixth sound receiving area; wherein the sound receiving line formed by the fifth sound receiving area and the sixth sound receiving area is located at the azimuth of 0 degrees and 180 degrees, and the sound receiving line formed by the second sound receiving area and the fourth sound receiving area is located at the azimuth of 30 degrees and 210 degrees. The sound reception line formed by the third sound reception area and the fourth sound reception area is located at an orientation of 60 degrees and 240 degrees, the sound reception line formed by the first sound reception area and the second sound reception area is located at an orientation of 90 degrees and 270 degrees, the sound reception line formed by the fourth sound reception area and the fifth sound reception area is located at an orientation of 120 degrees and 300 degrees, and the sound reception line formed by the first sound reception area and the fourth sound reception area is located at an orientation of 150 degrees and 330 degrees.

3. The hearing aid device with warning function integrating image, audio positioning and omnidirectional sound array as claimed in claim 2, characterized in that: When the face image cannot continuously lock onto the voice at the shooting position due to angle deviation, the control unit uses an advanced adaptive combined differential array algorithm to lock onto the human voice with the loudest volume as the voice, and separates the voice and the ambient sound from the audio.

4. The hearing aid device with warning function integrating image, audio positioning and omnidirectional sound array as claimed in claim 3, characterized in that: The advanced adaptive combined differential array algorithm is as follows: S m (t) represents the audio received by the mth microphone chip, Q(t) = [q(t)…q(tK g +1)] T K is the impulse response of speech to the mth microphone chip, and is presented as a zero-mean signal g A set of column vectors of samples, h m =[h m,1 …h m,Kg ] indicates K g Column vector of samples, U m (t) is the ambient sound in the audio, and T is the transposition operator; therefore, the advanced adaptive combined differential array algorithm calculates the mth a The mth microphone chip and the a+1 The audio received by a microphone chip; The directional pattern of the above-mentioned microphone chip is defined by the zero-point steering angle θ0, which is determined by the distance d between the microphone chips. k and delay time τ k Control, as follows: θ0 = cos -1 (-cτ k / d k ), c represents the speed of sound, and the directional pattern is a cardioid pattern (θ0=180, τ k =d k / c), and then by the compensation factor h k The frequency response is as follows:

5. The hearing aid device with warning function integrating image, audio positioning and omnidirectional sound array as claimed in claim 1, characterized in that: The photographing unit comprises two photographing devices, and the two photographing devices are respectively located at the front and rear sides of the neck hanging part.

6. The hearing aid device with warning function integrating image, audio positioning and omnidirectional sound array as claimed in claim 1, characterized in that: The warning unit includes two vibrators, which are respectively located on the left and right sides of the neck hanging part.

7. The hearing aid device with warning function integrating image, audio positioning and omnidirectional sound array as claimed in claim 6, characterized in that: The control unit identifies the direction of the ambient sound via the 360-degree sound receiving unit, and the control unit independently controls one of the vibrators corresponding to the direction of the ambient sound to vibrate.

8. The hearing-assistant device with warning function integrating image, audio positioning and omnidirectional sound array as claimed in claim 1, characterized in that: The hearing aid body comprises two ear hooks and a neck-mounted part, the 360-degree sound receiving unit and the camera unit are arranged on the neck-mounted part, and the warning unit and the playing unit are arranged on the two ear hooks.

9. The hearing-assistant device with warning function integrating image, audio positioning and omnidirectional sound array as claimed in claim 1, characterized in that: The hearing aid body comprises two ear hooks and a head-mounted part. The 360-degree sound receiving unit and the photographing unit are arranged on the head-mounted part. The warning unit and the playing unit are arranged on the two ear hooks.

10. The hearing-assistant device with warning function integrating image, audio positioning and omnidirectional sound array as claimed in claim 1, characterized in that: The hearing aid body includes two ear hooks and a hanging part, the hanging part is provided with a detachable hanging rope, wherein the 360-degree sound receiving unit and the photographic unit are arranged on the hanging part, the warning unit and the playback unit are arranged on the two ear hooks, wherein the hanging part and the two ear hooks are wirelessly connected.

11. An assisted listening method with a warning function integrating image, audio positioning and omnidirectional sound array, used in the assisted listening device with a warning function integrating image, audio positioning and omnidirectional sound array as claimed in any one of claims 1 to 10, characterized in that: The assisted listening method with warning function integrating image, audio positioning and omnidirectional sound array comprises the following steps: Store multiple special environment audios, multiple preset face images and multiple preset voices; The 360-degree sound receiving unit continuously collects an ambient sound and a voice, and the camera unit takes a face image in a variable shooting direction or points to a target in a pointing direction; The control unit compares the ambient sound with the aforementioned special ambient sound audio, and generates a first instruction when the comparison is consistent; The control unit compares the face image with the aforementioned preset face image, and generates a second instruction when the comparison is consistent; The control unit compares the voice with the aforementioned preset voice and generates a third instruction when the comparison matches; The photographing unit activates the pointing direction to generate a fourth instruction; The first instruction is to instruct the playback unit to play the ambient sound at that time and generate a warning message; The second instruction is to instruct the camera unit to lock the face image and to instruct the playback unit to play the voice at the shooting position; The third instruction is to instruct the playback unit to play the voice that matches the aforementioned preset voice; The fourth instruction is to instruct the playback unit to play the ambient sound and / or the voice at the pointing direction; The first instruction, the second instruction, the third instruction and the fourth instruction are executed according to an execution order or a flexible order, wherein the execution order is that the first instruction takes precedence over the second instruction, the second instruction takes precedence over the third instruction, and the third instruction takes precedence over the fourth instruction; the flexible order is that the first instruction has the highest priority, and the fourth instruction takes precedence over the second instruction and the third instruction; When at least two of the first instruction, the second instruction, the third instruction and the fourth instruction exist, the control unit executes the existing one of the first instruction, the second instruction, the third instruction and the fourth instruction according to the aforementioned execution sequence or the maneuvering sequence; When only one of the first instruction, the second instruction, the third instruction and the fourth instruction exists, the control unit only executes the existing one among the first instruction, the second instruction, the third instruction and the fourth instruction.

12. The assisted listening method with warning function integrating image, audio positioning and omnidirectional sound array as claimed in claim 11, characterized in that: The warning information causes the warning unit to vibrate.

13. The assisted listening method with warning function integrating image, audio positioning and omnidirectional sound array as claimed in claim 11, characterized in that: When there are two or more face images that match the aforementioned preset face images, the second instruction is to instruct the camera unit to lock the face image that is closest to a shooting position of the camera unit, and to instruct the playback unit to play the voice that is closest to the shooting position.

14. The assisted listening method with warning function integrating image, audio positioning and omnidirectional sound array as claimed in claim 11, characterized in that: When the photographic unit fails to lock the face image, the control unit locks the voice with the loudest volume at that time by using an advanced adaptive combined differential array algorithm, and controls the playback unit to play the voice with the loudest volume.

15. The assisted listening method with warning function integrating image, audio positioning and omnidirectional sound array as claimed in claim 11, characterized in that: When no face image matches the aforementioned preset face image, the second instruction instructs the playback unit to play the voice that is closest to the pointing direction.

16. The assisted listening method with warning function integrating image, audio positioning and omnidirectional sound array as claimed in claim 11, characterized in that: When there are two or more voices that match the aforementioned preset voice, the second instruction instructs the playback unit to play the voice that is closest to the pointing direction.

17. The assisted listening method with warning function integrating image, audio positioning and omnidirectional sound array as claimed in claim 11, characterized in that: After the photographing unit starts the pointing direction, it further executes a pointing direction priority command. The control unit changes the execution sequence to the maneuvering sequence according to the pointing direction priority command.

Images