Control method of hearing aid glasses

Abstract

The application discloses a control method of hearing aid glasses, and relates to the technical field of intelligent wearing, which comprises the following steps: providing a glasses frame, a visual unit and a hearing unit which are installed on the glasses frame, wherein the visual unit comprises an environment collector and an eye collector, and the hearing unit comprises a sound collector and a sound emitter; collecting environment image information by controlling the environment collector; collecting the gaze direction information of the wearer's eyes by controlling the eye collector; determining the position information of a target sound source in the environment image information according to the environment image information and the gaze direction information; collecting environment sound information by controlling the sound collector; determining the sound information of the target sound source in the environment sound information according to the position information of the target sound source; and controlling the sound emitter to receive the sound information of the target sound source and output the sound information of the target sound source to the human ear. The control method realizes accurate positioning and obtains the sound concerned by the wearer through the linkage of vision and hearing, thereby improving the use experience of the wearer.

Description

Technical Field

[0001] This invention relates to the field of smart wearable technology, and in particular to a control method for hearing aid glasses. Background Technology

[0002] With the increasing aging of society, hearing impairment is becoming a more prominent issue, leading to a growing market demand for hearing aids. Hearing aids, as assistive hearing devices, work by collecting, processing, and amplifying sound signals. Current hearing aids primarily rely on sound signal amplification technology to enhance the overall volume of sound signals, thereby improving the hearing experience for those with hearing loss to some extent.

[0003] Existing hearing aids amplify all sound signals in the environment indiscriminately, including background noise and the effective information the wearer wants to hear. This non-selective amplification method leads to two main problems: first, background noise may mask the effective information the wearer wants to hear, reducing the clarity of the auditory signal; second, prolonged exposure to high-decibel noise environments may cause discomfort to the wearer and affect their physical and mental health.

[0004] While some hearing aids employ active noise cancellation technology, which uses algorithms to cancel out ambient noise and improve sound clarity, its effectiveness is limited when faced with complex sound sources or when background noise is similar to the desired sound timbre. This makes it difficult to effectively distinguish and identify the types of sounds the wearer truly needs, resulting in poor practicality and comfort of the hearing aid. Summary of the Invention

[0005] The main objective of this invention is to propose a control method for hearing aid glasses, which aims to solve the problem that existing hearing aids are unable to effectively distinguish and identify the types of sounds that the wearer truly needs, resulting in poor practicality and comfort of the hearing aids.

[0006] To achieve the above objectives, the present invention proposes a control method for hearing aid glasses, providing hearing aid glasses comprising a frame, a visual unit mounted on the frame, and an auditory unit. The visual unit includes an environmental sensor and an eye sensor, and the auditory unit includes a microphone and a sound generator. The control method for the hearing aid glasses includes the following steps:

[0007] The environmental acquisition device is controlled to collect environmental image information;

[0008] The eye acquisition device is controlled to collect the gaze direction information of the wearer's eyes;

[0009] The location information of the target sound source in the environmental image information is determined based on the collected environmental image information and the gaze direction information;

[0010] Control the microphone to collect ambient sound information;

[0011] The sound information of the target sound source in the ambient sound information is determined based on the location information of the target sound source.

[0012] The speaker is controlled to receive the sound information from the target sound source and output the sound information from the target sound source to the human ear.

[0013] In one embodiment of the present invention, the step of determining the location information of the target sound source in the environmental image information based on the environmental image information and the gaze direction information includes:

[0014] Establish a first coordinate system centered on the environmental data collector or the wearer's head;

[0015] The coordinates of each sound source in the environmental image information in the first coordinate system are determined based on the environmental image information collected by the environmental collector.

[0016] In one embodiment of the present invention, after the step of determining the coordinates of each sound source in the first coordinate system in the environmental image information, the method further includes:

[0017] The coordinates of the eye in the first coordinate system are determined based on the eye position information collected by the human eye collector;

[0018] The gaze direction of the eye is determined based on the rotation direction of the pupil acquired by the human eye acquisition device.

[0019] The coordinates of the target sound source being gazed at by the eyes are determined in the first coordinate system based on the coordinates of the eyes and the direction of gaze.

[0020] In one embodiment of the present invention, the visual unit includes two human eye acquisition devices;

[0021] The step of determining the gaze direction of the eye based on the rotation direction of the pupil acquired by the human eye acquisition device includes:

[0022] The coordinates of the two eyes in the first coordinate system are determined based on the position information of the two eyes collected by the two human eye collectors.

[0023] The gaze direction of the two eyes is determined based on the rotation direction of the two pupils captured by the two human eye acquisition devices;

[0024] The coordinates of the target sound source being gazed at by the two eyes in the first coordinate system are determined based on the intersection of the gaze directions of the two eyes.

[0025] In one embodiment of the present invention, the step of determining the gaze direction of the two eyes based on the rotation direction of the two pupils acquired by the two eye acquisition devices includes:

[0026] The two human eye acquisition devices are controlled to collect the rotation direction of the pupils of the two eyes in real time;

[0027] If the time during which any of the pupils does not move is greater than a first preset time, then the current pupil orientation is taken as the gaze direction of the eye.

[0028] In one embodiment of the present invention, the step of determining the sound information of the target sound source in the ambient sound information based on the location information of the target sound source includes:

[0029] Determine the coordinate information of the microphone in the first coordinate system;

[0030] The first direction of the target sound source relative to the microphone is determined based on the coordinates of the microphone and the coordinates of the target sound source;

[0031] The sound received by the microphone in the first direction is filtered out based on the coordinates of the microphone and the first direction.

[0032] In one embodiment of the present invention, two microphones are provided;

[0033] After the step of filtering out the sound received by the pickup in the first direction based on the coordinates of the pickup and the first direction, the method further includes:

[0034] Determine the coordinates of the other microphone in the first coordinate system;

[0035] The second direction of the target sound source relative to the other microphone is determined based on the coordinates of the other microphone and the coordinates of the target sound source;

[0036] The first sound information corresponding to the coordinates of the target sound source is determined based on the intersection of the first direction and the second direction.

[0037] In one embodiment of the present invention, the hearing aid glasses pre-store audio segment information corresponding to images of different types of sound sources;

[0038] The step of determining the sound information corresponding to the coordinates of the target sound source based on the intersection of the first direction and the second direction further includes:

[0039] The image information of the target sound source is determined based on its coordinates;

[0040] The type of the target sound source is determined based on the image information of the target sound source;

[0041] The second sound information corresponding to the image information is determined based on the type of the target sound source and the pre-stored audio segment information corresponding to the images of the different types of sound sources.

[0042] Compare the first sound information with the second sound information;

[0043] If the frequency band contained in the first sound information is within the frequency band contained in the second sound information, then the microphone is controlled to transmit the first sound information to the speaker.

[0044] If the frequency band of the first sound information is outside the frequency band of the second sound information, the microphone is controlled to collect the sound information of each sound source within a first preset distance from the target sound source, and the collected sound information of each sound source is compared with the first sound information and the second sound information in turn.

[0045] In one embodiment of the present invention, the step of controlling the sound generator to receive the sound information of the target sound source and outputting the sound information of the target sound source to the human ear includes:

[0046] Control the sound generator to reduce the volume of sound sources other than the target sound source;

[0047] Control the sound generator to amplify the sound volume of the target sound source;

[0048] The sound generator is controlled to output the processed sound information of the target sound source to the human ear.

[0049] In one embodiment of the present invention, the control method of the hearing aid glasses includes:

[0050] Based on the control input commands received by the hearing aid glasses, the hearing aid glasses are controlled to switch between preset focus control mode, panoramic control mode and custom control mode;

[0051] In the focus control mode, the location information of the target sound source in the environmental image information is determined according to the environmental image information and the gaze direction information, the sound information of the target sound source in the environmental sound information is determined according to the location information of the target sound source, and the speaker is controlled to output the sound information of the target sound source to the human ear;

[0052] In the overall control mode, the microphone is controlled to collect all sounds in the environment, and the speaker is controlled to transmit the sound to the human ear;

[0053] In the custom control mode, the microphone is controlled to collect specific types of sound, and the sound generator is controlled to transmit the sound to the human ear.

[0054] This invention proposes a control method for hearing aid glasses, wherein the hearing aid glasses include a frame, and a microphone, a sound generator, a visual unit, and an information processing unit mounted on the frame. The visual unit includes an environmental acquisition unit and a human eye acquisition unit, and the auditory unit includes a microphone and a sound generator. The control method for the hearing aid glasses involves first controlling the environmental acquisition unit to acquire environmental image information, controlling the human eye acquisition unit to acquire the wearer's eye gaze direction information, and simultaneously controlling the microphone to acquire ambient sound. Then, the information processing unit determines the location information of the target sound source in the environmental image information based on the environmental image information and the human eye gaze direction information. Next, based on the location information of the target sound source, it determines the sound information of the target sound source in the ambient sound information. Finally, it controls the sound generator to receive the sound information of the target sound source and transmit the sound to the wearer's ear.

[0055] This control method achieves precise positioning through the coordinated use of vision and hearing, thereby acquiring the sound the wearer is interested in. When a hearing-impaired person wears hearing aid glasses, the glasses can filter out sounds emitted from objects in the direction the hearing-impaired person's eyes are focused on, thus matching the image and sound of the target sound source. Therefore, hearing aid glasses, to a certain extent, replace the brain's information processing function, combining vision, hearing, and other sensory systems to specifically acquire the desired sound, thereby achieving directional sound reception based on the wearer's needs. This method not only greatly improves the accuracy of sound recognition but also significantly enhances the wearer's auditory experience, enabling them to more easily capture the desired sound in complex acoustic environments, thereby improving the practicality of hearing aid glasses and the user experience. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0057] Figure 1 This is a schematic diagram of the hardware structure of the hearing aid glasses of the present invention;

[0058] Figure 2 This is a schematic diagram of the structure of the hearing aid glasses of the present invention in one embodiment;

[0059] Figure 3 for Figure 2A structural diagram of hearing aid glasses from another perspective;

[0060] Figure 4 for Figure 2 Right view of hearing aid glasses;

[0061] Figure 5 A flowchart of an embodiment of the control method for hearing aid glasses provided by the present invention;

[0062] Figure 6 for Figure 5 The flowchart shows the step of determining the location information of the target sound source in the environmental image information based on environmental image information and gaze direction information.

[0063] Figure 7 for Figure 6 The flowchart following the step of determining the coordinates of each sound source in the first coordinate system in the environmental image information;

[0064] Figure 8 for Figure 7 The flowchart shows the steps in determining the gaze direction of the eye based on the direction of pupil rotation captured by the human eye acquisition device.

[0065] Figure 9 for Figure 8 A flowchart illustrating the steps involved in determining the gaze direction of both eyes;

[0066] Figure 10 for Figure 5 The flowchart shows the steps in determining the location and sound information of the target sound source in the ambient sound information based on the location information of the target sound source.

[0067] Figure 11 for Figure 10 The flowchart following the step of filtering the sound received by the microphone in the first direction based on the microphone's coordinates and the first direction;

[0068] Figure 12 for Figure 11 The flowchart following the step of determining the sound information of the target sound source based on the intersection of the first and second directions;

[0069] Figure 13 for Figure 5 The flowchart shows the steps in which the control unit receives sound information from the target sound source and outputs the sound information from the target sound source to the human ear.

[0070] Figure 14 The flowchart of another embodiment of the control method for hearing aid glasses provided by the present invention.

[0071] Explanation of icon numbers:

[0072] 1. Hearing aid glasses; 10. Frame; 11. Temples; 12. Eyeglass frame; 20. Visual unit; 21. Environmental sensor; 22. Human eye sensor; 30. Auditory unit; 31. Microphone; 32. Sound generator; 321. In-ear speaker; 322. Bone conduction speaker; 40. Memory; 50. Interface unit; 60. Sensing unit; 70. Power supply unit; 80. Wireless communication unit.

[0073] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0075] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative positional relationship and movement of the components in a specific posture. If the specific posture changes, the directional indications will also change accordingly.

[0076] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the use of "and / or" or "and / or" throughout the text includes three parallel solutions. For example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0077] Figure 1 A schematic diagram of the hardware structure of an optional hearing aid glasses 1 to implement various embodiments of the present invention.

[0078] The hearing aid glasses 1 may include an information processing unit, a vision unit 20, an hearing unit 30, a memory 40, an interface unit 50, a sensing unit 60, a power supply unit 70, a wireless communication unit 80, etc. Figure 1Hearing aid glasses 1 with various components are shown, but it is understood that it is not required to implement all of the components shown. More or fewer components may be implemented alternatively. The elements of hearing aid glasses 1 will be described in detail below.

[0079] The information processing unit typically controls the overall operation of the hearing aid glasses 1. For example, the information processing unit performs related control and processing tasks such as acquiring environmental images, acquiring the direction of the wearer's gaze, acquiring ambient sound, and determining the location and sound information of the target sound source. In addition, the information processing unit can also perform control mode switching processing, so that the wearer can switch between different control modes according to different scenarios or needs.

[0080] The visual unit 20 is used to acquire environmental image information and human eye gaze direction information.

[0081] The auditory unit 30 is used to collect ambient sound information and transmit the filtered target sound source sound to the human ear.

[0082] The memory 40 can store software programs for processing and control operations executed by the information processing unit, as well as specific object sound images and corresponding audio frequency band information (for example, human voices, animal calls, vehicle sounds, etc. all have their specific frequency band characteristics and images).

[0083] The interface unit 50 serves as a connection interface between at least one external device and the hearing aid glasses 1.

[0084] The sensing unit 60 detects the current state of the hearing aid glasses 1 (e.g., whether the hearing aid glasses 1 is on or off), the mode of the hearing aid glasses 1, whether the wearer is touching the hearing aid glasses 1 (i.e., touch input), the working state of each unit of the hearing aid glasses 1, etc., and generates commands or signals for controlling the operation of the hearing aid glasses 1. In addition, the sensing unit 60 can detect whether the power supply unit 70 is providing power or whether the interface unit 50 is connected to an external device.

[0085] The power supply unit 70 receives external or internal power under the control of the information processing unit and provides the appropriate power required to operate the various components and assemblies.

[0086] The wireless communication unit 80 typically includes one or more components that allow radio communication between the hearing aid glasses 1 and a wireless communication device or mobile terminal (e.g., the wearer can connect to and control motor equipment via a mobile phone).

[0087] Based on the hardware structure of the hearing aid glasses 1 described above, various embodiments of the hearing aid glasses 1 of the present invention are proposed. The hearing aid glasses 1 in the following embodiments have the functions of some or all of the above-mentioned components.

[0088] Combination Figures 2 to 4As shown in the figure, an embodiment of the present invention proposes a hearing aid glasses 1, which includes a frame 10 and a visual unit 20 and an auditory unit 30 disposed on the frame 10. The visual unit 20 includes an environmental collector 21 and a human eye collector 22 connected to the information processing unit. The environmental collector 21 is used to collect environmental image information, and the human eye collector 22 is used to collect the gaze direction information of the wearer's eyes. The information processing unit determines the location information of the target sound source in the environmental image information based on the human eye gaze direction information. The auditory unit 30 includes a microphone 31 and a speaker 32 connected to the information processing unit. The microphone 31 is used to collect environmental sound information.

[0089] In this embodiment, the information processing unit determines the sound information of the target sound source in the ambient sound information based on the location information of the target sound source, and outputs it to the human ear through the speaker 32. Other non-target sound sources are filtered or the sound output power is reduced, thereby achieving targeted differentiation and amplification of the sound needed by the wearer, and improving the wearer's user experience.

[0090] Furthermore, the frame 10 includes a frame 12 and two temples 11, which are respectively mounted at both ends of the frame 12 along its length. An environmental sensor 21, an eye sensor 22, and a microphone 31 are all mounted on the frame 12 to facilitate capturing environmental image information in the wearer's forward field of vision, collecting the direction of the wearer's gaze, and capturing sound in front of the wearer. A microphone 32 is mounted on the temple 11 to directly transmit the processed sound to the wearer's ear canal, reducing sound loss during transmission.

[0091] Furthermore, the vision unit 20 includes two eye trackers, which are spaced apart on the side of the frame 12 facing the wearer, and are used to capture the gaze direction information of the wearer's two eyes respectively. The information control unit analyzes and determines the intersection of the gaze directions of the two eyes, which can more accurately determine the target being gazed at by the wearer.

[0092] Furthermore, the hearing unit 30 includes two microphones 31, which are located on the frame 12 and near the rotatable connection between the frame 12 and the two temples 11, so that the hearing aid glasses 1 can simulate the hearing mechanism of human binaural hearing, thereby more accurately locating and directionally processing the sound source.

[0093] Furthermore, the sound generator 32 includes two in-ear speakers 321, each mounted on one of the two temples 11, so that the wearer can insert the in-ear speakers 321 into their ears when using the hearing aid glasses 1. Additionally, each in-ear speaker 321 includes a connected in-ear portion and a connecting portion. The connecting portion is connected to the temple 11, and at least a portion of the connecting portion is made of an elastic material. The in-ear portion is communicatively connected to the information processing unit and is used to transmit sound to the ear. Because the connecting portion can deform, it can adapt to different ear shapes and sizes of wearers.

[0094] Furthermore, the sound generator 32 includes two bone conduction speakers 322, and the ends of the two temples 11 away from the frame 12 are bent to form hanging parts. Both bone conduction speakers 322 are mounted on the hanging parts to better fit the wearer's ear contours, making the hearing aid glasses 1 stable and comfortable to wear. When the wearer moves their head or performs daily activities, the hearing aid glasses 1 can still maintain a stable wearing position, improving the convenience and comfort of use.

[0095] Furthermore, the frame 12 and / or temple 11 have mounting cavities inside, and the information processing unit is located in the mounting cavity to prevent dust and moisture from entering, further protecting the information processing unit. At the same time, the space of the mounting cavity also facilitates the electrical connection between the information processing unit and other components.

[0096] This invention also proposes a method for controlling hearing aid glasses.

[0097] Combination Figure 5 As shown, in one embodiment of the present invention, hearing aid glasses 1 are provided. The hearing aid glasses include a frame 10 and a visual unit 20 and an auditory unit 30 mounted on the frame 10. The visual unit 20 includes an environmental collector 21 and a human eye collector 22, and the auditory unit 30 includes a microphone 31 and a sound generator 32. The control method of the hearing aid glasses includes the following steps:

[0098] S100: Controls the environmental acquisition unit 21 to acquire environmental image information;

[0099] S200: Controls the human eye acquisition device 22 to acquire the wearer's eye gaze direction information;

[0100] S300: Determine the location information of the target sound source in the environmental image information based on the collected environmental image information and gaze direction information;

[0101] S400: Controls microphone 31 to collect ambient sound information;

[0102] S500: Determines the sound information of the target sound source in the ambient sound information based on the location information of the target sound source;

[0103] S600: Controls the speaker 32 to receive sound information from the target sound source and outputs the sound information from the target sound source to the human ear.

[0104] In this embodiment, the hearing aid frame 10 is designed in the form of eyeglasses for easy daily wear. The frame 10 includes a frame and two temples 11, which are connected by hinges, allowing the temples 11 to be folded for easy carrying and storage. The frame 10 has a built-in microcontroller or ASIC integrated into an information processing unit, which implements the aforementioned control functions.

[0105] The environment acquisition device 21 can be a 3D camera or a LiDAR sensor to acquire environmental images and capture the contours and positional information of objects in the environment. For example, when the environment acquisition device 21 is a 3D camera, it can be positioned in front of the frame 10, that is, the 3D camera faces the wearer's face to acquire environmental image information. The 3D camera has depth perception capabilities and can capture three-dimensional image information of the environment. This camera can be a 3D camera based on structured light, time-of-flight (ToF), or stereo vision technology, which can provide high-resolution and high-precision image information. After receiving a control signal, the 3D camera begins to capture environmental images in front of the wearer and transmits the image data back to the information processing unit.

[0106] The human eye acquisition device 22 can be an eye tracker or an infrared corneal reflector, used to capture the direction of eye gaze to help the information processing unit more accurately locate the sound source. For example, when the human eye acquisition device 22 is an eye tracker, the eye tracker is installed on the front or sides of the frame 10 to capture the wearer's eye movements. The eye tracker monitors the wearer's eye movements in real time and determines the direction the wearer is focusing on by the direction of eye rotation.

[0107] The information processing unit combines the information provided by the environmental collector 21 and the human eye collector 22 to calculate the precise location of the target sound source. Specifically, the information processing unit establishes a three-dimensional coordinate system based on the image information of the environment collected by the environmental collector 21, and determines the object in the direction of the human eye's gaze by judging the direction of the human eye's gaze, thus obtaining the location of the target sound source.

[0108] The microphone 31 is mounted at the front of the frame, close to the wearer's face. This layout helps to better capture ambient sound information from in front of the wearer. The microphone 31 can be a high-sensitivity microphone or microphone array, capable of effectively picking up various sound signals from the environment, including speech, music, and other ambient sounds. The speaker 32 is mounted at the end of the temple 11, close to the wearer's ear, and can be an in-ear speaker 321 or a bone conduction speaker 322.

[0109] After receiving the control signal, the microphone 31 begins to collect ambient sound signals and transmits the collected signals to the information processing unit via wired or wireless means. The information processing unit determines the position of the target sound source relative to the microphone 31 based on the location information of the target sound source provided by the environmental collector 21 and the human eye collector 22. The information processing unit then uses a sound source localization algorithm (such as time delay calculation or sound intensity difference calculation) to filter out the sound information of the target sound source from the ambient sound information collected by the microphone 31.

[0110] The speaker 32 is an in-ear speaker 321 or a bone conduction speaker 322. The information processing unit sends the selected target sound signal to the speaker 32. The speaker 32 can process the received target sound source sound signal, such as by enhancing and reducing noise, to improve the signal quality, and then transmit the processed sound to the wearer's ear.

[0111] Through the aforementioned technical means, the control method of the hearing aid glasses in this application achieves precise positioning and acquisition of the sound the wearer is interested in by coordinating vision and hearing. First, images of various objects in the environment are acquired visually. Then, the location of the target sound source is determined by the direction of eye gaze, thus achieving the effect of binding image and sound. When a hearing-impaired person wears hearing aid glasses 1, hearing aid glasses 1 can filter out the sound emitted by objects in the direction of the hearing-impaired person's gaze, thereby achieving the function of matching the image and sound of the target sound source. Therefore, hearing aid glasses, to a certain extent, replace the information processing function of the normal person's brain, combining vision, hearing, and other sensory systems to specifically acquire the desired sound, thus achieving the function of directional sound reception according to the wearer's needs. This method not only greatly improves the accuracy of sound recognition but also significantly improves the wearer's auditory experience, enabling the wearer to more easily capture the desired sound in complex acoustic environments, thereby enhancing the practicality of hearing aid glasses 1 and the wearer's user experience.

[0112] Combination Figure 6 As shown, in one embodiment of the present invention, the step of determining the location information of the target sound source in the environmental image information based on the environmental image information and the gaze direction information includes:

[0113] S310: Establish a first coordinate system centered on the environmental collector 21 or the wearer's head;

[0114] S320: Determine the coordinates of each sound source in the first coordinate system based on the environmental image information collected by the environmental collector 21.

[0115] In this embodiment, the environmental acquisition device 21 uses a 3D camera as an example. The 3D camera is installed at the front end of the frame and located in the center of the frame, facing the wearer. The 3D camera is activated and captures environmental images, which are then sent to the information processing unit. The information processing unit processes the received image data and uses image processing algorithms to identify and locate sound sources in the environment. The image processing algorithm may include steps such as edge detection, feature extraction, and target recognition to accurately identify and locate the position information of each sound source in the environment.

[0116] Specifically, the information processing unit establishes a first coordinate system with the 3D camera as the origin (or the wearer's head as the origin) based on the installation position and orientation of the 3D camera. When the 3D camera scans the environment in front of the wearer, it acquires the three-dimensional coordinates and contour information of each sound source in the environment through depth sensing technology. This coordinate information includes the position of each sound source on the X, Y, and Z axes in the first coordinate system, enabling the information processing unit to accurately determine the coordinates and contour of each sound source in space.

[0117] By employing the aforementioned visual detection and localization technologies, the problem of traditional hearing aids' inability to accurately locate sound sources is solved, significantly improving the accuracy of sound localization. This also facilitates the subsequent matching and binding of visual image information with sound information, achieving visual and auditory linkage. This allows for targeted amplification of sounds of interest to the wearer while reducing background noise interference. This technological solution not only enhances the wearer's auditory experience in complex acoustic environments but also improves user comfort and satisfaction.

[0118] Combination Figure 7 As shown, in one embodiment of the present invention, after the step of determining the coordinates of each sound source in the first coordinate system in the environmental image information, the method further includes:

[0119] S330: Determine the coordinates of the eye in the first coordinate system based on the eye position information collected by the human eye collector 22;

[0120] S340: Determine the gaze direction of the eye based on the direction of pupil rotation captured by the human eye collector 22;

[0121] S350: Determine the coordinates of the target sound source being gazed at in the first coordinate system based on the coordinates of the eye and the direction of gaze.

[0122] In this embodiment, the human eye acquisition device 22 is an eye tracker, which is installed at the front end of the frame and located inside the frame, facing the wearer's eyes. The eye tracker can monitor the wearer's eye movements in real time and determine the direction of the wearer's gaze.

[0123] The eye tracker determines the position and direction of eye movement by emitting near-infrared light and capturing the reflected light. The information processing unit receives the eye data collected by the eye tracker and processes it to determine the specific position of the wearer's eyes in a first coordinate system and the direction of the wearer's gaze.

[0124] Under the control of the information processing unit, and through the aforementioned technical means, after the 3D camera acquires the image and location information of each sound source in the environment, the target sound source in the first coordinate system facing the direction of eye gaze can be obtained based on the coordinates of the eye in the first coordinate system and the direction of eye gaze.

[0125] This method, combining a 3D camera and an eye tracker, solves the problem of traditional hearing aids failing to accurately determine the wearer's focus, significantly improving sound localization accuracy. By selectively amplifying sounds of interest to the wearer while reducing background noise interference, this technology not only enhances the wearer's auditory experience in complex acoustic environments but also improves user comfort and satisfaction.

[0126] Combination Figure 8 As shown, in one embodiment of the present invention, the visual unit 20 includes two eye collectors 22;

[0127] The step of determining the gaze direction based on the pupil rotation direction captured by the human eye collector 22 includes:

[0128] S360: Determine the coordinates of the two eyes in the first coordinate system based on the position information of the two eyes collected by the two eye collectors 22;

[0129] S370: Determine the gaze direction of the two eyes based on the rotation direction of the pupils of the two eyes captured by the two eye collectors 22;

[0130] S380: Determine the coordinates of the target sound source in the first coordinate system based on the intersection of the gaze directions of the two eyes.

[0131] In this embodiment, two eye sensors 22 are respectively installed on the inside of the frame and face the wearer's two eyes. These two eye sensors 22 can independently monitor the eye movements of the wearer's two eyes, thereby more accurately determining the direction of the wearer's gaze.

[0132] Specifically, the two eye sensors 22 acquire infrared data from each eye, and the information processing unit first processes this data to obtain the coordinates of the two eyes in a first coordinate system and their gaze direction. Subsequently, the information processing unit comprehensively analyzes the gaze directions of both eyes and, using triangulation, determines the coordinates of the target sound source that the wearer is simultaneously focusing on in the first coordinate system. This binocular positioning method provides higher accuracy and stability, especially in complex acoustic environments, enabling more accurate localization of the target sound source of interest to the wearer.

[0133] Combination Figure 9 As shown, in one embodiment of the present invention, determining the gaze direction of the two eyes based on the rotation direction of the pupils of the two eyes captured by two eye acquisition devices includes:

[0134] S371: Controls the two-eye acquisition device 22 to collect the rotation direction of the pupils of both eyes in real time;

[0135] S372: If the pupil of any eye does not move for a period of time longer than a first preset time, then the current pupil direction is taken as the gaze direction of the eye.

[0136] In this embodiment, an eye tracker collects the wearer's eye position data multiple times per second (e.g., 100 times) and transmits this data to an information processing unit. The information processing unit analyzes this data in real time to determine the wearer's eye movement trajectory. If the wearer's eyes remain in one direction for more than a first preset time (e.g., 1 second), the information processing unit identifies that direction as the direction the wearer is currently focusing on. Then, based on the wearer's eye coordinates and the first direction, combined with the coordinate information provided by the 3D camera, the information processing unit calculates the coordinates of the target sound source that the wearer is looking at in a first coordinate system.

[0137] By setting a preset time, the information processing unit can more accurately determine whether the wearer is truly focused on a particular direction. A brief eye fixation may be an unintentional glance or a slight eye movement, while a longer fixation time is more likely to indicate that the wearer is intently focused on a target. This reduces misjudgments and improves the accuracy of target sound source localization.

[0138] Setting an appropriate first preset time can balance response speed and accuracy. If the first preset time is too short, for example, a few microseconds, the hearing aid glasses 1 will frequently switch target sound sources; if the first preset time is too long, the response time of the hearing aid glasses 1 will be longer, affecting the wearer's experience.

[0139] Combination Figure 10 As shown, in one embodiment of the present invention, the step of determining the sound information of the target sound source in the ambient sound information based on the location information of the target sound source includes:

[0140] S510: Determine the coordinate information of the microphone 31 in the first coordinate system;

[0141] S520: Determine the first direction of the target sound source relative to the microphone 31 based on the coordinates of the microphone 31 and the coordinates of the target sound source;

[0142] S530: Based on the coordinates of the microphone 31 and the first direction, filter out the sound received by the microphone 31 in the first direction.

[0143] In this embodiment, firstly, the information processing unit determines the position of the microphone 31 in the first coordinate system using a built-in sensor or pre-set coordinate information. Since the position of the microphone 31 is relatively fixed on the frame 10, this coordinate information is fixed. Then, the information processing unit uses the coordinate information of the target sound source provided by the vision unit 20, combined with the coordinate information of the microphone 31, to calculate the direction of the target sound source relative to the microphone 31; this direction is also known as the first direction. Through geometric calculations (e.g., vector operations), the information processing unit can accurately determine the direction of the target sound source relative to the microphone 31. Afterward, the information processing unit uses beamforming technology to filter out the sound received in the first direction from the ambient sound collected by the microphone 31. By processing the sound signal collected by the microphone 31, the information processing unit extracts the sound signal in the first direction, thereby filtering out the sound emitted by the target sound source.

[0144] In this way, the information processing unit can selectively filter out sounds that interest the wearer, reducing background noise interference and thus improving the wearer's auditory experience. This technology not only improves the accuracy of sound localization but also enhances the wearer's experience, especially in complex acoustic environments, better meeting the wearer's needs and improving comfort and satisfaction.

[0145] Combination Figure 11 As shown, in one embodiment of the present invention, two microphones 31 are provided;

[0146] Following the step of filtering the sound received by microphone 31 in the first direction based on the coordinates of microphone 31 and the first direction, the method further includes:

[0147] S540: Determine the coordinates of the other pickup 31 in the first coordinate system;

[0148] S550: Determine the second direction of the target sound source relative to the other microphone 31 based on the coordinates of the other microphone 31 and the coordinates of the target sound source;

[0149] S560: Determine the first sound information corresponding to the coordinates of the target sound source based on the intersection of the first and second directions.

[0150] In this embodiment, both microphones 31 are mounted on the frame 12 so that the hearing aid can simulate the human binaural hearing mechanism, thereby more accurately locating and directing the sound source.

[0151] First, the information processing unit determines the positions of the two microphones 31 in the first coordinate system, and calculates the direction of the target sound source relative to the two microphones 31, namely the first direction and the second direction, based on the coordinate information of the target sound source provided by the vision unit 20.

[0152] The information processing unit processes the sound signals collected by the two microphones 31 and, by combining the intersection of the first and second directions, determines the coordinates of the target sound source from the collected first sound information. In this way, the information processing unit can more accurately locate and filter the sound emitted by the target sound source, further improving the accuracy and precision of sound localization.

[0153] Combination Figure 12 As shown, in one embodiment of the present invention, the hearing aid glasses 1 pre-stores audio segment information corresponding to images of different types of sound sources;

[0154] The step of determining the sound information corresponding to the coordinates of the target sound source based on the intersection of the first and second directions also includes:

[0155] S571: Determine the image information of the target sound source based on its coordinates;

[0156] S572: Determine the type of the target sound source based on the image information of the target sound source;

[0157] S573: Determine the second sound information corresponding to the image information based on the type of the target sound source and the pre-stored audio segment information corresponding to the images of different types of sound sources;

[0158] S574: Compare the first audio information with the second audio information;

[0159] S575: If the sound frequency band contained in the first sound information is within the sound frequency band contained in the second sound information, then control the microphone 31 to transmit the first sound information to the speaker 32.

[0160] S576: If the sound frequency band contained in the first sound information is outside the sound frequency band contained in the second sound information, then control the microphone 31 to collect the sound information of each sound source within a first preset distance from the target sound source, and compare the collected sound information of each sound source with the first sound information and the second sound information in sequence.

[0161] In this embodiment, in most cases, the position of the object is the same as the position of the sound it emits; that is, the image and sound of the target sound source have the same coordinates in the first coordinate system. However, in some scenarios, such as when the display screen and the sound broadcast in a subway are located in different positions in space, the position of the image seen by the hearing-impaired person is not the position of the sound they want to hear, which will prevent the hearing-impaired person from obtaining the sound they want to hear.

[0162] Therefore, after determining the first sound information of the target sound source, the information processing unit further verifies whether the sound of the target sound source is the sound that the hearing-impaired person needs to obtain by comparing the first sound information with the audio frequency segment information corresponding to different types of sound sources pre-stored in the hearing aid glasses.

[0163] Specifically, the information processing unit has a built-in database containing audio frequency band information corresponding to images of different types of sound sources. This audio frequency band information is derived from a large amount of experimental data and statistical analysis, which helps the information processing unit identify different types of sound sources. For example, human speech, animal calls, and vehicle sounds all have their specific frequency band characteristics.

[0164] The information processing unit determines the coordinates of the target sound source being gazed at by the wearer and the corresponding image based on the environmental image information collected by the environmental collector 21 and the eye gaze direction collected by the human eye collector. The information processing unit analyzes and processes the image and identifies its type, thus determining the type of object the wearer is currently gazing at (e.g., a person, a dog, a vehicle, etc.). Based on the known type of the target sound source, the unit filters the sound frequency band emitted by the object being gazed at (i.e., the second sound information) from pre-stored sound frequency bands of different types of objects. Therefore, the second sound information is processed by the visual unit, determined from pre-stored sound frequency band information of different types of objects. The first sound information is processed by the auditory unit, determined from the ambient sound collected by the microphone based on the coordinates of the target sound source.

[0165] Therefore, it is necessary to compare the first and second sound information, using a mechanism that links vision and hearing to resolve the issue of different coordinates between the target image and the target sound. Specifically, if the frequency band contained in the first sound is within the frequency band contained in the second sound, the information processing unit confirms that this is the sound emitted by the target sound source (i.e., the target image and target sound coordinates are the same at this time), and sends it to the speaker 32. This ensures that the wearer hears the target sound they are interested in, rather than other background noise or misidentified sounds.

[0166] If the frequency band contained in the first sound information is outside the frequency band contained in the second sound information, the information processing unit considers that there is a problem of different coordinates between the target image and the target sound. In this case, the information processing unit filters the sound of the target sound source again. Based on the original location of the target sound source, it performs sound comparison within a space of a first preset distance from the coordinates of the target sound source, and sequentially confirms whether there are other sound sources within this range that are located in the frequency band corresponding to the second sound information. If a matching sound source is found, the information processing unit sends a new sound signal to the transmitter 32; otherwise, it expands the first preset distance again based on the previous filtering space for filtering and comparison until the most matching sound signal is found.

[0167] The first preset distance can be a fixed value or a variable value. For example, the first preset distance can be set to 10 centimeters, and the filtering radius can be increased by 10 centimeters each time the filtering range is expanded. Alternatively, the distance from the target sound source to the origin of the three-dimensional coordinate system (the center of the environmental collector 21) can be set to L, and the first preset distance can be set to 0.1*L. That is, the farther the target sound source is from the wearer, the greater the increase in the first preset distance when expanding the filtering range. Since there are more sound sources farther away from the wearer, the environmental image information collected by the environmental collector 21 will be more complex, and the sound filtering accuracy will decrease. Therefore, by setting the first preset distance, the target sound source that the wearer is interested in can be identified more accurately and quickly.

[0168] Combination Figure 13 As shown, in one embodiment of the present invention, the step of controlling the sound generator 32 to receive sound information from a target sound source and outputting the sound information from the target sound source to the human ear includes:

[0169] S610: Controls the sound generator 32 to reduce the volume of sound sources other than the target sound source;

[0170] S620: Controls the sound generator 32 to enhance the sound volume of the target sound source;

[0171] S630: Controls the sound generator 32 to transmit the processed sound information of the target sound source to the human ear.

[0172] In this embodiment, the sound volume is processed by controlling the speaker 32 to reduce or eliminate the sound volume of non-target sound sources, or to increase the sound volume of target sound sources, thereby making the target sound more prominent and ensuring that the wearer can hear the target sound more clearly.

[0173] Specifically, this can be achieved through various audio processing techniques, such as dynamic range compression, frequency response adjustment, or gain control. Dynamic range compression ensures that the volume of the target audio source remains within a comfortable range, preventing discomfort caused by sudden volume changes. Frequency response adjustment can adjust the gain or attenuation of specific frequency ranges based on the wearer's hearing characteristics to optimize the listening experience. Gain control amplifies the target audio source's signal, making it stand out more against background noise.

[0174] These techniques effectively reduce the volume of sound sources other than the target sound source while simultaneously increasing the volume of the target sound source. This dual-processing method not only improves sound clarity but also enhances the wearer's auditory experience. This technique is particularly important in complex and noisy acoustic environments because it significantly improves the wearer's focus on and understanding of the target sound.

[0175] In other embodiments, it may also include only reducing the sound volume of sound sources other than the target sound source or increasing the sound volume of the target sound source.

[0176] Combination Figure 14 As shown, in one embodiment of the present invention, the control method for hearing aid glasses includes:

[0177] S700: Based on the control input commands received by the hearing aid glasses, it controls the hearing aid glasses to switch between preset focus control mode, panoramic control mode and custom control mode;

[0178] S710: In focus control mode, the location information of the target sound source in the environmental image information is determined according to the environmental image information and the gaze direction information, the sound information of the target sound source in the environmental sound information is determined according to the location information of the target sound source, and the speaker 32 is controlled to output the sound information of the target sound source to the human ear.

[0179] S720: In panoramic control mode, controls the microphone 31 to collect all sounds in the environment and controls the speaker 32 to transmit the sound to the human ear;

[0180] S730: In custom control mode, the microphone 31 is controlled to collect specific types of sound, and the speaker 32 is controlled to transmit the sound to the human ear.

[0181] In this embodiment, in focus control mode, the hearing aid glasses 1, through the control methods provided in the above embodiments, selectively filter the sound signals of the target sound source that the wearer is interested in, and ultimately transmit the sound to the wearer's ears. This mode is suitable when the wearer wants to concentrate on listening to sounds from a specific direction, such as listening to the speaker in a meeting or talking to someone in a noisy environment.

[0182] In panoramic control mode, microphone 31 captures all sounds in the environment and transmits them to the wearer's ear via speaker 32. This mode does not filter or amplify sound; instead, it allows the wearer to hear all sounds in the entire environment. Panoramic control mode is suitable for situations where the wearer wants a comprehensive understanding of their surroundings, such as while walking outdoors or participating in social activities, where the wearer can hear natural sounds and conversations.

[0183] In custom control mode, the wearer can select specific sound types and control the microphone 31 to capture these specific sound types. The wearer can set the sound types they are interested in, such as human voices, music, alarm sounds, etc. The information processing unit has built-in information on different sound frequency bands, capable of identifying and enhancing the sound types specified by the wearer while attenuating other irrelevant sounds. This mode is suitable for wearers who wish to focus on specific types of sounds, such as listening only to instrumental music at a concert or only to the voices of family members at home.

[0184] The hearing aid glasses 1 are equipped with operation buttons to enable human-computer interaction. They can also be connected to a mobile terminal, allowing the wearer to interact with the device. Based on the wearer's input, the hearing aid glasses 1 switch between focus mode, panoramic control mode, and custom control mode. The information processing unit responds to the wearer's commands in real time, adjusting the processing method to meet the wearer's needs. This flexible mode-switching function enables the hearing aid to adapt to various auditory environments and wearer preferences, providing a personalized auditory experience.

[0185] The above description is merely an exemplary embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention specification and drawings under the technical concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A method for controlling hearing aid glasses, characterized in that: A hearing aid glasses are provided, the hearing aid glasses including a frame, a visual unit and an auditory unit mounted on the frame, the visual unit including an environmental collector and a human eye collector, and the auditory unit including a microphone and a sound generator; The method for controlling the hearing aid glasses includes the following steps: The environmental acquisition device is controlled to collect environmental image information; The eye acquisition device is controlled to collect the gaze direction information of the wearer's eyes; The location information of the target sound source in the environmental image information is determined based on the collected environmental image information and the gaze direction information; Control the microphone to collect ambient sound information; The sound information of the target sound source in the ambient sound information is determined based on the location information of the target sound source. The speaker is controlled to receive the sound information from the target sound source and output the sound information from the target sound source to the human ear.

2. The control method for hearing aid glasses as described in claim 1, characterized in that, The step of determining the location information of the target sound source in the environmental image information based on the environmental image information and the gaze direction information includes: Establish a first coordinate system centered on the environmental data collector or the wearer's head; The coordinates of each sound source in the environmental image information in the first coordinate system are determined based on the environmental image information collected by the environmental collector.

3. The control method for hearing aid glasses as described in claim 2, characterized in that, After determining the coordinates of each sound source in the first coordinate system in the environmental image information, the method further includes: The coordinates of the eye in the first coordinate system are determined based on the eye position information collected by the human eye collector; The gaze direction of the eye is determined based on the rotation direction of the pupil acquired by the human eye acquisition device. The coordinates of the target sound source being gazed at by the eyes are determined in the first coordinate system based on the coordinates of the eyes and the direction of gaze.

4. The control method for hearing aid glasses as described in claim 3, characterized in that, The visual unit includes two human eye acquisition devices; The step of determining the gaze direction of the eye based on the rotation direction of the pupil acquired by the human eye acquisition device includes: The coordinates of the two eyes in the first coordinate system are determined based on the position information of the two eyes collected by the two human eye collectors. The gaze direction of the two eyes is determined based on the rotation direction of the two pupils captured by the two human eye acquisition devices; The coordinates of the target sound source being gazed at by the two eyes in the first coordinate system are determined based on the intersection of the gaze directions of the two eyes.

5. The control method for hearing aid glasses as described in claim 4, characterized in that, The step of determining the gaze direction of the two eyes based on the rotation direction of the two pupils captured by the two eye acquisition devices includes: The two human eye acquisition devices are controlled to collect the rotation direction of the pupils of the two eyes in real time; If the time during which any of the pupils does not move is greater than a first preset time, then the current pupil orientation is taken as the gaze direction of the eye.

6. The control method for hearing aid glasses as described in any one of claims 2 to 5, characterized in that, The step of determining the sound information of the target sound source in the ambient sound information based on the location information of the target sound source includes: Determine the coordinate information of the microphone in the first coordinate system; The first direction of the target sound source relative to the microphone is determined based on the coordinates of the microphone and the coordinates of the target sound source; The sound received by the microphone in the first direction is filtered out based on the coordinates of the microphone and the first direction.

7. The control method for hearing aid glasses as described in claim 6, characterized in that, The microphone is provided in two parts; After the step of filtering out the sound received by the pickup in the first direction based on the coordinates of the pickup and the first direction, the method further includes: Determine the coordinates of the other microphone in the first coordinate system; The second direction of the target sound source relative to the other microphone is determined based on the coordinates of the other microphone and the coordinates of the target sound source; The first sound information corresponding to the coordinates of the target sound source is determined based on the intersection of the first direction and the second direction.

8. The control method for hearing aid glasses as described in claim 7, characterized in that, The hearing aid glasses have pre-stored audio frequency segment information corresponding to images of different types of sound sources; The step of determining the sound information corresponding to the coordinates of the target sound source based on the intersection of the first direction and the second direction further includes: The image information of the target sound source is determined based on its coordinates; The type of the target sound source is determined based on the image information of the target sound source; The second sound information corresponding to the image information is determined based on the type of the target sound source and the pre-stored audio segment information corresponding to the images of the different types of sound sources. Compare the first sound information with the second sound information; If the frequency band contained in the first sound information is within the frequency band contained in the second sound information, then the microphone is controlled to transmit the first sound information to the speaker. If the frequency band of the first sound information is outside the frequency band of the second sound information, the microphone is controlled to collect the sound information of each sound source within a first preset distance from the target sound source, and the collected sound information of each sound source is compared with the first sound information and the second sound information in turn.

9. The control method for hearing aid glasses as described in any one of claims 1 to 5, characterized in that, The step of controlling the sound generator to receive the sound information from the target sound source and outputting the sound information from the target sound source to the human ear includes: Control the sound generator to reduce the volume of sound sources other than the target sound source; Control the sound generator to amplify the sound volume of the target sound source; The sound generator is controlled to output the processed sound information of the target sound source to the human ear.

10. The control method for hearing aid glasses as described in any one of claims 1 to 5, characterized in that, The control method for the hearing aid glasses includes: Based on the control input commands received by the hearing aid glasses, the hearing aid glasses are controlled to switch between preset focus control mode, panoramic control mode and custom control mode; In the focus control mode, the location information of the target sound source in the environmental image information is determined according to the environmental image information and the gaze direction information, the sound information of the target sound source in the environmental sound information is determined according to the location information of the target sound source, and the speaker is controlled to output the sound information of the target sound source to the human ear; In the overall control mode, the microphone is controlled to collect all sounds in the environment, and the speaker is controlled to transmit the sound to the human ear; In the custom control mode, the microphone is controlled to collect specific types of sound, and the sound generator is controlled to transmit the sound to the human ear.

Images