Intelligent glasses for hearing aid, hearing aid method and hearing aid system

By integrating microphone and speakers in smart glasses and real-time hearing aid functions using wireless communication modules, the existing smart glasses are solved, and the functionality and user experience of the product are improved.

CN119071706APending Publication Date: 2024-12-03SOLOS TECH SHENZHEN LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310639291.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing smart glasses have single functions, expensive prices, and lack real-time hearing aids, resulting in low product viscosity.

Method used

A smart glasses are designed, including two temples, a sound pickup device and multiple speakers, and are equipped with a wireless communication module to obtain voice data from preset directions through at least one microphone, and amplify and send to the target speaker through the wireless communication module. Play to realize real-time hearing aid function.

Benefits of technology

The functions of smart glasses are expanded, product stickiness is improved, equipment weight and power consumption is reduced, manufacturing costs are reduced, and sound quality is improved by reducing echoes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071706A_ABST
    Figure CN119071706A_ABST
Patent Text Reader

Abstract

The invention discloses intelligent glasses for hearing aid and a hearing aid method and system.The intelligent glasses comprise two glasses legs, a pickup device, a plurality of loudspeakers and a wireless communication module, and the pickup device comprises at least one microphone used for obtaining first voice data from a first preset direction; the wireless communication module is used for amplifying the first voice data to obtain amplified voice data and sending the amplified voice data to the target loudspeaker for playing, and the target loudspeaker and the at least one microphone are arranged on different glasses legs respectively. The hearing aid function based on the intelligent glasses is realized, and the viscosity of the product can be increased.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of smart glasses, and in particular, to smart glasses for hearing assistance, a hearing assistance method, and an auxiliary system. Background Art

[0002] With the development of computer technology, smart glasses are becoming more and more popular. However, existing smart glasses are expensive, and in addition to their own functions as smart glasses, they usually only have the functions of listening to music and making or receiving calls. The functions are relatively simple and the product viscosity is low. Summary of the invention

[0003] The embodiments of the present application provide smart glasses for hearing assistance, a hearing assistance method, and an auxiliary system, which are used to realize a real-time hearing assistance function based on smart glasses, thereby increasing product viscosity.

[0004] On the one hand, an embodiment of the present application provides a pair of smart glasses for hearing aid, comprising: two temples, a sound pickup device, a plurality of speakers, and a wireless communication module;

[0005] The multiple speakers and the sound pickup device are arranged on at least one of the two temples, the wireless communication module is arranged in the cavity of any one of the two temples and is electrically connected to the sound pickup device and the multiple speakers, and the sound pickup device includes at least one microphone;

[0006] The at least one microphone is used to obtain first voice data from a first preset direction;

[0007] The wireless communication module is used to amplify the first voice data to obtain amplified voice data, and send the amplified voice data to a target speaker among the multiple speakers for playback, wherein the target speaker and the at least one microphone are respectively arranged on different temples.

[0008] On one hand, an embodiment of the present application further provides a hearing aid method based on smart glasses, wherein the smart glasses include: two temples, a wireless communication module, and a sound pickup device and a plurality of speakers electrically connected to the wireless communication module, wherein the sound pickup device includes at least one microphone, and the method includes:

[0009] Acquiring first voice data from a first preset direction through the at least one microphone;

[0010] amplifying the first voice data through the wireless communication module to obtain amplified voice data;

[0011] The amplified voice data is played through a target speaker among the multiple speakers, wherein the target speaker and the at least one microphone are respectively arranged on different temples.

[0012] The embodiment of the present application also provides an auxiliary system, including: smart glasses and a smart mobile terminal;

[0013] The smart glasses include: two temples, a sound pickup device, a plurality of speakers, and a wireless communication module, wherein the plurality of speakers and the sound pickup device are arranged on at least one of the two temples, the wireless communication module is arranged in a cavity of any one of the two temples and is electrically connected to the sound pickup device and the plurality of speakers, and the sound pickup device includes at least one microphone;

[0014] The wireless communication module is used to control the switching of the working modes of the smart glasses in response to the control operation of the user, and the working modes include: a hearing aid mode and a translation mode;

[0015] The at least one microphone is used to obtain first voice data from a first preset direction in the hearing aid mode or the translation mode;

[0016] The wireless communication module is used for, in the hearing aid mode, amplifying the first voice data to obtain amplified voice data, and sending the amplified voice data to a target speaker among the multiple speakers for playback, wherein the target speaker and the at least one microphone are respectively arranged on different temples;

[0017] The wireless communication module is further used to send the first voice data to the smart mobile terminal in the translation mode;

[0018] The intelligent mobile terminal is used to receive the first voice data and send second voice data to the wireless communication module, wherein the second voice data is obtained by translating the first voice data;

[0019] The wireless communication module is further used to receive second voice data sent by the smart mobile terminal, and send the second voice data to the target speaker for playing.

[0020] In each embodiment of the present application, at least one microphone in the smart glasses is used to obtain first voice data from a first preset direction, and the first voice data is amplified by a wireless communication module in the smart glasses and the amplified first voice data is sent to a target speaker among the multiple speakers for playback, wherein the target speaker and the at least one microphone are respectively arranged on different temples, thereby realizing a real-time hearing aid function based on the smart glasses, thereby expanding the function of the smart glasses and improving product stickiness. At the same time, since the microphone on one side obtains the voice data to be amplified from a specific direction and the speaker on the other side plays the amplified voice data, the echo can be reduced and the quality of the played sound can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Figure 1 A schematic diagram of the structure of smart glasses provided in one embodiment of the present application;

[0023] Figure 2 A schematic diagram of the internal structure of smart glasses provided by another embodiment of the present application;

[0024] Figure 3 for Figure 2 A schematic diagram of the external structure of the smart glasses in the illustrated embodiment;

[0025] Figure 4 A schematic diagram of an auxiliary system provided in one embodiment of the present application;

[0026] Figure 5 for Figure 4 A schematic diagram of the hardware structure of the intelligent mobile terminal in the auxiliary system shown;

[0027] Figure 6 A schematic diagram of the implementation process of a hearing aid method based on smart glasses provided in one embodiment of the present application;

[0028] Figure 7 A schematic diagram of the sound beam direction of the microphone in different working modes in the hearing aid method based on smart glasses provided in an embodiment of the present application;

[0029] Figure 8 A schematic diagram of the operating state and working mode of smart glasses in the hearing aid method based on smart glasses provided in an embodiment of the present application;

[0030] Fig. 9 A schematic diagram of a process of processing a voice signal in a call mode in a hearing aid method based on smart glasses provided in an embodiment of the present application;

[0031] Fig.10 A schematic diagram of a process of processing a speech signal in a hearing aid mode in a hearing aid method based on smart glasses provided in an embodiment of the present application;

[0032] Fig.11 for Fig.10 Schematic diagram of the feedback cancellation process in . DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] See also Figure 1 , Figure 1 This is a schematic diagram of the structure of the smart glasses provided in one embodiment of the present application. For ease of explanation, only the parts related to the embodiment of the present application are shown in the figure. Figure 1 As shown, the smart glasses 10 include: a wireless communication module 101, a sound pickup device 102, a plurality of speakers 103 and two temples 104. The wireless communication module 101 is electrically connected to the sound pickup device 102 and the plurality of speakers 103.

[0035] The multiple speakers 103 and the sound pickup device 102 are arranged on at least one of the two temples 104 , and the wireless communication module 101 is arranged in the cavity of any one of the two temples 104 and is electrically connected to the sound pickup device 102 and the multiple speakers 103 .

[0036] The sound pickup device 102 includes at least one microphone. Optionally, the sound pickup device 102 includes at least one directional microphone. Further, the sound pickup device 102 may also include at least one omnidirectional microphone. For example, the sound pickup device 102 may be a microphone array consisting of at least one directional microphone and / or at least one omnidirectional microphone.

[0037] Further, the wireless communication module 101 may include, but is not limited to: a controller 1011, a voice data processor 1012, and a wireless signal transceiver 1013. Among them, the controller 1011 is mainly used for UI (User Interface), task operation and real-time operating system operation. The voice data processor 1012 is mainly used to execute the operation of the more time-consuming signal processing algorithm and the Bluetooth wireless communication algorithm. The wireless signal transceiver can be, for example, a wireless signal transceiver.

[0038] The wireless signal transceiver can, but is not limited to, use at least one of the Bluetooth protocol, WiFi (Wireless Fidelity) protocol, NFC (Near Field Communication) protocol, ZigBee, DLNA (DIGITAL LIVING NETWORK ALLIANCE) protocol, UWB (Ultra Wideband, carrier-free communication), RFID (Radio Frequency Identification) protocol and cellular mobile communication (Cellular Mobile Communication) protocol for data transmission.

[0039] Specifically, at least one microphone is used to obtain first voice data from a first preset direction.

[0040] The wireless communication module 101 is used to amplify the first voice data to obtain amplified voice data, and send the amplified voice data to a target speaker among the multiple speakers 103 for playback, wherein the target speaker and at least one microphone are respectively arranged on different temples.

[0041] In this embodiment, by using at least one microphone in the smart glasses to obtain first voice data from a first preset direction, using the wireless communication module in the smart glasses to amplify the first voice data and sending the amplified first voice data to the target speaker among the multiple speakers for playback, wherein the target speaker and the at least one microphone are respectively arranged on different temples, a real-time hearing aid function based on smart glasses is realized, thereby expanding the function of smart glasses and improving product stickiness. In addition, since there is no need to install additional hearing aids on the smart glasses, the weight of the smart glasses can be reduced, the power consumption can be reduced, and the manufacturing cost of the smart glasses can be reduced. At the same time, since the microphone on one side obtains the voice data to be amplified from a specific direction and the speaker on the other side plays the amplified voice data, the echo can be reduced and the quality of the played sound can be improved.

[0042] See also Figure 2 and Figure 3 , Figure 2 This is a schematic diagram of the internal structure of smart glasses provided by another embodiment of the present application. Figure 3 for Figure 2 The external structure diagram of the smart glasses in the embodiment shown is as follows. Figure 2 and Figure 3 As shown, the smart glasses 20 include: a frame 201, two temples 202, at least one sound pickup device 203 (for ease of understanding, only one is shown in the figure), a first speaker 205, a second speaker 206 and a wireless communication module 204.

[0043] The temple 202 is connected to the frame 201, the first speaker 205, the second speaker 206 and at least one sound pickup device 203 are arranged on at least one of the two temples 202, and the wireless communication module 204 is arranged in the cavity of any one of the two temples 202 and is electrically connected to the first speaker 205, the second speaker 206 and at least one sound pickup device 203.

[0044] The sound pickup device 203 is used to obtain first voice data from a first preset direction.

[0045] The wireless communication module 204 is used to amplify the first voice data to obtain amplified voice data, and send the amplified voice data to a target speaker among the first speaker 205 and the second speaker 206 for playback, wherein the target speaker and the sound pickup device 203 are respectively arranged on different temples.

[0046] The wireless communication module 204 is also used to control the switching of the working modes of the smart glasses, which include a call mode, a hearing aid mode and a translation mode.

[0047] The sound pickup device 203 includes: at least one directional microphone and / or at least one omnidirectional microphone. When the sound pickup device 203 is an omnidirectional microphone, the wireless communication module 204 is also used to perform a first beamforming process on the voice data acquired by the at least one sound pickup device 203 in the call mode, so that the sound beam of the at least one sound pickup device 203 is directed downward, that is, corresponding to the user's mouth. At this time, the sound source of the voice data may be the wearer of the smart glasses; the voice data acquired by the at least one sound pickup device 203 is subjected to a second beamforming process in the hearing aid mode and the translation mode, so that the sound beam of the at least one sound pickup device 203 is directed forward. At this time, the sound source of the voice data may be the conversation partner of the wearer of the smart glasses. After switching the working mode, the wireless communication module 204 can automatically control the omnidirectional microphone to point to the corresponding direction according to the switched working mode.

[0048] Optionally, the two temples 202 include a first temple 202A and a second temple 202B, the front end of the first temple 202A and the front end of the second temple 202B are respectively connected to two sides of the frame 201, and at least one sound pickup device 203 is installed at the front end of the first temple 202A.

[0049] The first speaker 205 and the second speaker 206 are used to output voice data or music data. The voice data includes: the voice data obtained by the sound pickup device 203 after being processed by the wireless communication module 204, and the call voice data sent by the smart mobile terminal received by the wireless communication module 204. The music data includes: the music data sent by the smart mobile terminal received by the wireless communication module 204.

[0050] Preferably, the first speaker 205 is mounted on the first temple 202A, and an output port of the first speaker 205 is located at the rear end of the first temple 202A.

[0051] The second speaker 206 is mounted on the second temple 202B, and an output port of the second speaker 206 is located at the rear end of the second temple 202B.

[0052] In hearing aid mode and translation mode, only the open speaker located on one temple is used to output voice data, thereby achieving the effect of reducing echo.

[0053] For example, in the hearing aid mode, if the sound pickup device of the smart glasses is located on the left temple, the amplified voice signal is only output to the speaker configured on the right temple for playback. If the sound pickup device is located on the right temple, the amplified voice will be output to the speaker configured on the left temple for playback.

[0054] Optionally, in another embodiment of the present application, when a microphone and a speaker are provided on each temple, each microphone and each speaker have their own corresponding unique identification information, and the corresponding relationship between the unique identification information of each microphone and each speaker and their positions is stored in the wireless communication module 204. The wireless communication module 204 is further used to determine the installation position of the microphone for acquiring the first voice data according to the corresponding relationship, and determine the target microphone according to the installation position and the corresponding relationship.

[0055] It can be understood that for the hearing aid mode, the smart glasses can independently amplify the voice signal in front of the wearer without connecting to any mobile terminal device.

[0056] Since the sound pickup device is installed at the front end of the temple and the output port of each speaker is installed at the rear end of the temple, there can be a sufficient distance between the sound pickup device and the output port of each speaker, thereby effectively reducing the situation of loop generation between the sound pickup device and the speaker, and reducing the probability of echo and howling during use.

[0057] In addition, after the user wears the smart glasses, the tail end of the temple is closest to the user's ear. Installing the speaker outlet at the tail end of the temple can make the speaker outlet closest to the user's ear, thereby improving the efficiency of sound output.

[0058] The first speaker 205 and the second speaker 206 are preferably mono speakers. The mono speakers respectively installed on the two temples can be combined to achieve stereo sound effects.

[0059] Optionally, the wireless communication module 204 includes: a controller 2041 , a voice data processor 2042 and a wireless signal transceiver 2043 .

[0060] The controller 2041 is used to control the switching of the working modes of the smart glasses. The controller 2041 is preferably a MCU (Microcontroller Unit).

[0061] The voice data processor 2042 is used to process the voice data. The voice data processor 2042 is preferably a DSP (Digital Signal Processing) or a voice data processing integrated circuit. The voice data processing integrated circuit is a common circuit, and the present application does not specifically limit its structure.

[0062] In another embodiment of the present application, the sound pickup device 203 is preferably a directional microphone, which is used to directionally pick up the voice signal and then send it to the MCU / DSP for voice processing.

[0063] Furthermore, the sound pickup device 102 also includes two microphones for picking up voice signals from a second preset direction. The user can pick up voice signals from the second preset direction through the two microphones on the smart glasses to issue voice commands and answer calls. The use of two microphones to pick up sound can accurately determine the direction of the sound pickup, and the processing of the sound pickup data of the two microphones can achieve noise reduction. The voice signal is picked up from the two microphones of the smart glasses, and the noise-reduced voice is played on the speakers of the smart glasses and sounds are emitted.

[0064] The wireless signal transceiver 2043 is used for data interaction with the smart mobile terminal.

[0065] Optionally, the voice data processor 2042 includes a voice equalizer. The wireless signal transceiver 2043 is also used to receive the volume adjustment control instruction sent by the smart mobile terminal and send it to the voice data processor 2042. The voice data processor 2042 is also used to adjust the sound data to be output to the frequency band and volume sound data pointed to by the volume adjustment control instruction by using the voice equalizer, and send the adjusted sound data to the speaker pointed to by the volume adjustment control instruction. The frequency band pointed to by the volume adjustment control instruction can be low frequency, medium frequency or high frequency. The volume adjustment can be to increase the volume or to decrease the volume.

[0066] Furthermore, the voice data processor 2042 is also used to perform voice equalization processing (using a voice equalizer) and output volume processing on the data on the downlink channel in the call mode, and to perform echo cancellation processing, beamforming processing and noise suppression processing on the data on the uplink channel using a preset acoustic echo cancellation algorithm (Acoustic Echo Cancellation), beamforming algorithm (Beamforming) and noise suppression algorithm (Noise Cancellation).

[0067] Furthermore, the voice data processor 2042 is also used to perform feedback cancellation processing, voice activity detection processing, noise suppression processing, voice equalization processing and user speech detection processing on the voice data in the hearing aid mode or the translation mode using the preset feedback cancellation algorithm (Feedback Cancellation), voice activity detection processing, noise suppression algorithm, voice equalizer algorithm (Equalizer) and user speech detection algorithm, and send the voice data after the voice equalization processing to the corresponding speaker for output. At the same time, the voice data after the voice equalization processing is also used as reference data for the above-mentioned feedback cancellation processing during the processing process.

[0068] Optionally, the smart glasses further include: at least one sensor 207 electrically connected to the wireless communication module 204 .

[0069] At least one sensor 207 is mounted on the inner side and / or the outer side of the first temple 102A and / or the second temple 102B.

[0070] Specifically, the at least one sensor 207 may include, but is not limited to, at least one of a touch sensor, a proximity sensor, an accelerometer, a gyroscope, a magnetic induction sensor, and an inertial measurement unit.

[0071] Optionally, the inertial measurement unit is a 9-axis sensor. The 9-axis sensor is used to collect the user's motion data (3-axis accelerometer data, 3-axis gyroscope data, 3-axis magnetometer data), and send it to the smart mobile terminal through the wireless communication module 204 for monitoring the status of the user's head and body.

[0072] Preferably, the at least one sensor 207 includes: a 9-axis sensor, at least one touch sensor and at least one proximity sensor.

[0073] At least one touch sensor is installed on the outer side of the first temple 102A and / or the second temple 102B.

[0074] Optionally, at least one touch sensor is used to detect a first control operation of the user, and send data of the detected first control operation to the controller 2041, where the first control operation is used to adjust the volume. The controller 2041 is also used to control and adjust the volume of the sound output by the smart glasses in response to the first control operation according to the data of the first control operation.

[0075] The first control operation includes a control operation for increasing the volume and a control operation for decreasing the volume. The control operation for increasing the volume corresponds to an action of the user's finger sweeping on the touch sensor toward the ear, and the control operation for decreasing the volume corresponds to an action of the user's finger sweeping on the touch sensor toward the frame (i.e., away from the ear).

[0076] Optionally, at least one touch sensor is further used as an input device to detect a second control operation of the user, and send data of the detected second control operation to the controller 2041. The controller 2041 is also used to control switching of the working mode of the smart glasses according to the data of the second control operation and in response to the second control operation.

[0077] Optionally, at least one proximity sensor is installed on the inner side of the first temple 202A and / or the second temple 202B, and is used to detect whether the user wears or takes off the smart glasses and obtain the time when the user does not wear the smart glasses, and send the detection result to the controller 2041. The controller 2041 is used to control the play or pause of music and control the power on or off of the smart glasses 20 according to the detection result. For example, when the proximity sensor detects that the user wears the smart glasses, the music data is played, when the proximity sensor detects that the user takes off the smart glasses, the music data is stopped from being played, and when the proximity sensor detects that the user does not wear the smart glasses for more than a preset time, the power is turned off.

[0078] The number of each of the above sensors is preferably one to reduce the overall weight of the smart glasses. However, in specific applications, according to actual needs, the number of various sensors is not limited to one. For example, to improve the accuracy of the detection result, a proximity sensor can be set on each of the two temples.

[0079] Optionally, the controller 2041 is also used to send the data acquired by each sensor to the smart mobile terminal via the above-mentioned wireless signal transceiver.

[0080] Optionally, the sound pickup device 203 may include a first directional microphone and a second directional microphone. The first directional microphone is used to obtain voice data in front of the user, and the second directional microphone is used to obtain voice data below the user. The first directional microphone and the second directional microphone may be located at different positions of the temples, respectively. Furthermore, the controller 2041 may also control the directional microphones at corresponding positions to pick up sound according to the working mode of the smart glasses 20, thereby reducing noise in the acquired voice data and improving the speed of signal processing.

[0081] Optionally, the smart glasses also include a battery 208, which is installed on the first temple 202A and electrically connected to the wireless communication module 204, so as to provide power for all hardware components on the smart glasses, such as the wireless communication module 204, sensors, speakers, and microphones.

[0082] Optionally, the smart glasses further include: an input device electrically connected to the wireless communication module 204. The input device is disposed on the frame 201 and at least one of the two temples 202.

[0083] Optionally, the input device includes: at least one of a touch sensor and at least one control button (not shown in the figure). The at least one control button is installed on the outside of the first temple 202A and / or the second temple 202B, and is electrically connected to the wireless communication module 204. The at least one control button is used to trigger the wireless communication module 204 to control the switching of the working mode or operating state of the smart glasses. The operating state includes an idle state and a working state, and the working state includes the call mode, the hearing aid mode, and the translation mode.

[0084] Specifically, the wireless communication module 204 is also used to switch the working mode of the smart glasses to the hearing aid mode in response to a first switching instruction triggered by the user based on the input device, and in the hearing aid mode, amplify the first voice data from the first preset direction.

[0085] The wireless communication module 204 is also used to switch the working mode of the smart glasses to a translation mode in response to a second switching instruction triggered by the user based on the input device, and in the translation mode, send the first voice data to the smart mobile terminal, receive the second voice data sent by the smart mobile terminal, and send the second voice data to the target speaker for playback, wherein the second voice data is translated from the first voice data.

[0086] The wireless communication module 204 is also used to switch the working mode of the smart glasses to a call mode in response to a third switching instruction triggered by the user based on the input device. In this call mode, the third voice data from the second preset direction obtained by the microphone is sent to the target terminal, and the fourth voice data sent by the target terminal is received and output to all speakers for playback.

[0087] When the user of the smart glasses uses the smart glasses to directly communicate with the user of other terminals, the target terminal is the other terminal. When the user of the smart glasses uses the smart glasses to communicate with the user of other terminals via the smart mobile terminal, the target terminal is the smart mobile terminal.

[0088] The first preset direction points to the front of the user, and the second preset direction points to the bottom of the user.

[0089] The electronic components of the above-mentioned smart glasses can be connected through a bus.

[0090] It should be noted that the relationship between the components of the above-mentioned smart glasses can be a replacement relationship or a superposition relationship. That is, all the components in the above-mentioned embodiment can be installed on one smart glass, or a part of the above-mentioned components can be selectively installed according to needs. When it is a replacement relationship, the smart glasses are also provided with a connection interface for peripherals, and the connection interface can be at least one of a PS / 2 interface, a serial interface, a parallel interface, an IEEE1394 interface, a USB (Universal Serial Bus) interface, etc. The function of the replaced component can be realized by a peripheral connected to the connection interface, such as: an external speaker, an external sensor, etc.

[0091] The controller 2041 is also used to determine whether there is a playback device that establishes a data connection with the smart glasses, such as an external speaker or an external hearing aid; if so, the second voice data or the amplified first voice data is sent to the playback device for playback through the playback device; if not, the second voice data or the amplified first voice data is output through a local speaker. In this way, when there is an independent playback device, the power consumption of the smart glasses can be further reduced by using the independent playback device to play the voice data.

[0092] In this embodiment, by using at least one microphone in the smart glasses to obtain the first voice data from the first preset direction, using the wireless communication module in the smart glasses to amplify the first voice data and send the amplified first voice data to the target speaker among the multiple speakers for playback, wherein the target speaker and the at least one microphone are respectively arranged on different temples, the real-time hearing aid function based on the smart glasses is realized, thereby expanding the function of the smart glasses and improving the product stickiness. In addition, since there is no need to install additional hearing aids on the smart glasses, the weight of the smart glasses can be reduced, the power consumption can be reduced, and the manufacturing cost of the smart glasses can be reduced. At the same time, since the microphone on one side obtains the voice data to be amplified from a specific direction and the amplified voice data is played by the speaker on the other side, the echo can be reduced and the quality of the played sound can be improved. In addition, through the interaction between the smart glasses and the smart mobile terminal, the real-time translation function based on the smart glasses is also realized, further expanding the function of the smart glasses, and since the voice data to be translated is obtained from a specific direction, the quality of the voice data to be translated can be improved, thereby improving the translation efficiency.

[0093] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an auxiliary system provided in one embodiment of the present application. Figure 4 As shown, the auxiliary system includes: smart glasses 301, a smart mobile terminal 302 and a cloud computer device 303.

[0094] The smart glasses include 301: two temples, a sound pickup device, a plurality of speakers and a wireless communication module, wherein the sound pickup device includes at least one microphone.

[0095] The wireless communication module is used to control the switching of the working modes of the smart glasses in response to the user's control operation, and the working modes include: hearing aid mode, translation mode and call mode.

[0096] The at least one microphone is used to obtain first voice data from a first preset direction in the hearing aid mode or the translation mode;

[0097] The wireless communication module is used to amplify the first voice data to obtain amplified voice data in the hearing aid mode, and send the amplified voice data to a target speaker among the multiple speakers for playback, wherein the target speaker and the at least one microphone are respectively arranged on different temples.

[0098] The wireless communication module is also used to send the first voice data to the smart mobile terminal in the translation mode, receive the second voice data sent by the smart mobile terminal, and send the second voice data to the target speaker for playback, wherein the second voice data is obtained by translating the first voice data.

[0099] The at least one microphone is further used to obtain third voice data from a second preset direction in the call mode.

[0100] The wireless communication module is further used to send the third voice data to the target terminal and receive the fourth voice data sent by the target terminal.

[0101] The multiple speakers are also used to play the fourth voice data.

[0102] For details on the structure and functions of the smart glasses 301, please refer to the above Figures 1 to 3 Related description in the illustrated embodiment.

[0103] The smart mobile terminal 302 may include, but is not limited to: a cellular phone, a smart phone, other wireless communication devices, a personal digital assistant, an audio player, other media players, a music recorder, a video recorder, a camera, other media recorders, a smart radio, a laptop computer, a personal digital assistant (PDA), a portable multimedia player (PMP), a moving picture expert group (MPEG-1 or MPEG-2) audio layer 3 (MP3) player, a digital camera, and a smart wearable device (such as a smart watch, a smart bracelet, etc.). The smart mobile terminal 302 is installed with an Android or IOS operating system.

[0104] The smart mobile terminal 302 is used to perform data interaction with the smart glasses 301 and the cloud computer device 303. Specifically, for example, the first voice data sent by the smart glasses 301 is received, and the first voice data is translated into the second voice data of the target language locally at the smart mobile terminal 302, or the first voice data or the first text data corresponding to the first voice data is sent to the cloud computer device 303 for translation; the second text data of the translated target language returned by the cloud computer device 303 is received, and the second text data is converted into the second voice data and sent to the smart glasses 301. For another example, when performing target tasks such as playing music data and making calls, the played music data and the received call voice data are sent to the smart glasses 301. The communication protocol used by the smart mobile terminal 302 when performing data interaction with the smart glasses 301 is consistent with the communication protocol used by the smart glasses 301.

[0105] Optionally, the smart mobile terminal 302 determines whether to send the first voice data or the first text data corresponding to the first voice data to the cloud computer device 303 according to a preset conversion mark.

[0106] Specifically, the smart mobile terminal 302 is also used to obtain a preset conversion mark. If the preset conversion mark is a first mark, the first voice data is converted into the first text data, and the first text data is sent to the cloud computer device 303. If the preset conversion mark is a second mark, the first voice data is directly sent to the cloud computer device 303.

[0107] Furthermore, the smart mobile terminal 302 is also used to obtain the network speed and determine whether the network speed is greater than a preset speed; if it is not greater than the preset speed, the conversion mark is set to the first mark; if it is greater than the preset speed, the conversion mark is set to the second mark.

[0108] Furthermore, the intelligent mobile terminal 302 is further configured to, in response to a setting operation of the user, set the conversion mark to the mark pointed to by the setting operation.

[0109] like Figure 5 As shown, the smart mobile terminal 302 may include a control circuit, which may include a storage and processing circuit 300. The storage and processing circuit 300 may include a memory, such as a hard disk drive memory, a non-volatile memory (such as a flash memory or other electronically programmable memory with limited deletion for forming a solid-state drive, etc.), a volatile memory (such as a static or dynamic random access memory, etc.), etc., which is not limited in the embodiment of the present application. The processing circuit in the storage and processing circuit 300 can be used to control the operation of the smart mobile terminal 302. The processing circuit can be implemented based on one or more microprocessors, microcontrollers, digital signal processors, baseband processors, power management units, audio codec chips, application-specific integrated circuits, display driver integrated circuits, etc.

[0110] The storage and processing circuit 300 can be used to run the software in the smart mobile terminal 302, such as: artificial intelligence housekeeper application, Internet browsing application, Voice over Internet Protocol (VOIP) phone call application, email application, media playback application, operating system function, etc. These software can be used to perform some control operations, such as data processing and analysis of motion data sent by the smart glasses 301 based on a preset analysis algorithm, image acquisition based on a camera, ambient light measurement based on an ambient light sensor, proximity sensor measurement based on a proximity sensor, information display function based on a status indicator such as a status indicator light of a light-emitting diode, touch event detection based on a touch sensor, functions associated with displaying information on multiple (e.g., layered) displays, operations associated with performing wireless communication functions, operations associated with collecting and generating audio signals, control operations associated with collecting and processing button press event data, and other functions in the smart mobile terminal 302, etc., which are not limited by the embodiments of the present application.

[0111] Furthermore, the memory stores executable program code, and the processor connected to the memory calls the executable program code stored in the memory to execute the relevant steps in the following method embodiments of the present application to realize the various functions of the smart mobile terminal 302 in each embodiment of the present application.

[0112] The smart mobile terminal 302 may further include an input / output circuit 420. The input / output circuit 420 may be used to enable the smart mobile terminal 302 to input and output data, that is, to allow the smart mobile terminal 302 to receive data from an external device and also to allow the smart mobile terminal 302 to output data from the smart mobile terminal 302 to an external device. The input / output circuit 420 may further include a sensor 320. The sensor 320 may include an ambient light sensor, a proximity sensor based on light and capacitance, a touch sensor (e.g., based on a light touch sensor and / or a capacitive touch sensor, wherein the touch sensor may be part of a touch display screen or may be used independently as a touch sensor structure), an accelerometer, and other sensors.

[0113] The input / output circuit 420 may also include one or more displays, such as the display 140. The display 140 may include a liquid crystal display, an organic light emitting diode display, an electronic ink display, a plasma display, a display using other display technologies, or a combination of several of them. The display 140 may include a touch sensor array (i.e., the display 140 may be a touch display screen). The touch sensor may be a capacitive touch sensor formed by an array of transparent touch sensor electrodes (e.g., indium tin oxide (ITO) electrodes), or may be a touch sensor formed using other touch technologies, such as sonic touch, pressure-sensitive touch, resistive touch, optical touch, etc., which is not limited in the embodiments of the present application.

[0114] The smart mobile terminal 302 may also include an audio component 360. The audio component 360 may be used to provide audio input and output functions for the smart mobile terminal 302. The audio component 360 in the smart mobile terminal 302 may include a speaker, a sound pickup device, a buzzer, a tone generator, and other components for generating and detecting sound.

[0115] The communication circuit 380 can be used to provide the smart mobile terminal 302 with the ability to communicate with external devices. The communication circuit 380 may include analog and digital input / output interface circuits, and wireless communication circuits based on radio frequency signals and / or optical signals. The wireless communication circuit in the communication circuit 380 may include a radio frequency transceiver circuit, a power amplifier circuit, a low noise amplifier, a switch, a filter, and an antenna. For example, the wireless communication circuit in the communication circuit 380 may include a circuit for supporting near field communication (NFC) by transmitting and receiving near field connection electromagnetic signals. For example, the communication circuit 380 may include a near field communication antenna and a near field communication transceiver. The communication circuit 380 may also include a cellular phone transceiver and antenna, a wireless local area network transceiver circuit and antenna such as Bluetooth, WiFi, ZigBee, DLNA, UWB, RFID, etc.

[0116] The smart mobile terminal 302 may further include a battery, power management circuits and other input / output units 400. The input / output units 400 may include buttons, joysticks, click wheels, scroll wheels, touch pads, keypads, keyboards, cameras, light emitting diodes and other status indicators.

[0117] The user can input commands through the input / output circuit 420 to control the operation of the smart mobile terminal 302 , and can use the output data of the input / output circuit 420 to receive status information and other outputs from the smart mobile terminal 302 .

[0118] Furthermore, the smart mobile terminal 302 is also used to pair with the smart glasses 301 based on the Bluetooth protocol, and after the pairing is successful, the played music data is sent to the smart glasses 301 so that the music data can be played through the smart glasses 301.

[0119] The smart glasses 301 are also used to send the first voice data to be translated to the smart mobile terminal 302 for translation after the pairing is successful.

[0120] Optionally, the smart mobile terminal 302 is further used to obtain GPS data through a GPS module configured on the smart mobile terminal, and send the obtained GPS data to the smart glasses 301 for positioning the smart glasses 301.

[0121] Optionally, the smart mobile terminal 302 is also used to receive and store the motion data sent by the smart glasses 301 in real time, calculate the motion index according to the motion data and the GPS data, generate real-time voice data for notifying or reminding the user of the motion status according to the calculation result, and send the real-time voice data to the smart glasses 301 for output. The real-time voice data includes the notification or reminder voice of the motion index calculation result.

[0122] Specifically, a client program, such as an artificial intelligence housekeeper App, may be installed on the smart mobile terminal 302, through which data interaction operations with the smart glasses 301 are performed, parameters of the smart glasses 301 are configured based on user operations on the App, and motion data sent by the smart glasses 301 are processed and analyzed, such as running index calculation, posture monitoring and reminders, etc.

[0123] The above-mentioned music data and real-time voice data belong to the data of the downlink channel of the smart glasses 301. The smart glasses 301 perform voice equalization processing and output volume control processing on the received music data and real-time voice data through the voice data processor 1042, and then send them to the first speaker 201 and the second speaker 202 for output.

[0124] Specifically, the motion data includes data obtained by the smart glasses 301 through a 9-axis sensor (i.e., Ax, Ay, Az; Gx, Gy, Gz; Mx, My, Mz), and the motion index is a running index, and the running index includes: pace, distance, number of steps, left and right head balance, number of steps, stride length and step frequency. Among them, the 9-axis sensor refers to an accelerometer, a gyroscope and a magnetic induction sensor. The data measured by these three types of sensors can be decomposed into forces of the three directional axes of X, Y, and Z in the spatial coordinate system, so they are also called 3-axis accelerometers, 3-axis gyroscopes and 3-axis magnetic induction sensors.

[0125] The smart mobile terminal 302 is also used to perform index calculation, posture monitoring and exercise reminder operations according to the motion data obtained by the 9-axis sensor and the local GPS data, and display the index calculation results in real time through the display of the smart mobile terminal 302. Furthermore, the smart mobile terminal 302 can also generate voice prompt information according to the index calculation results, and send the voice prompt information to the smart glasses 301 via Bluetooth, so as to notify or remind the user in real time through the smart glasses 301.

[0126] Optionally, the smart mobile terminal 302 is also used to respond to voice commands sent by the smart glasses 301 to perform operations of making, answering or hanging up a call, and during a call, send the received call voice data to the smart glasses 301 so that the call voice data can be played through the speakers on the smart glasses 301.

[0127] The voice command is obtained by the voice data processor of the smart glasses 301 by performing voice command recognition processing on the voice data obtained by the sound pickup device of the smart glasses 301 using a preset voice recognition algorithm.

[0128] It can be understood that at least two directional and / or omnidirectional microphones are provided on the body of the smart glasses 301 in the present application, and the user can obtain voice data through the microphone on the smart glasses 301 for issuing voice commands and answering calls. The sound emitted by the user is received by the microphone on the smart glasses 301 and played through the speaker of the smart glasses 301 after noise reduction processing. Therefore, during a call, the user can put the smart mobile terminal 302 in a pocket or on a table, and the free hands can be used for other purposes, thereby improving the convenience of answering calls.

[0129] The cloud computing device 303 may be, for example, a cloud server or a server cluster, which is used to interact with the smart mobile terminal 302, store data sent by the smart mobile terminal 302 and / or the smart glasses 301 (e.g., all sensor data acquired by the smart glasses), and process the data based on a preset processing logic. Specifically, the cloud computing device 303 receives the first voice data sent by the smart mobile terminal 302 or the first text data corresponding to the first voice data, and then translates the first voice data or the first text data into the second text data of the target language, and returns it to the smart mobile terminal 302. For example: assuming that the source language is Chinese and the target language is English, if the smart mobile terminal 302 sends Chinese voice data 1, the voice data 1 is first converted into Chinese text data 1, and then the text data 1 is translated into English text data 2 and sent to the smart mobile terminal 302; if the smart mobile terminal 302 sends the text data 1 corresponding to the Chinese voice data 1, the text data 1 is translated into English text data 2 and sent to the smart mobile terminal 302.

[0130] The source language recognition operation may be performed by the smart mobile terminal 302, or may also be performed by the cloud computing device 303. If the source language recognition operation is performed by the smart mobile terminal 302, then when the smart mobile terminal 302 sends the first voice data, it sends the identification information of the source language and the target language to the cloud computing device 303, so that the cloud computing device 303 determines the source language and the target language according to the indication information.

[0131] Optionally, the source language and the target language may also be determined by the smart mobile terminal 302 based on the user's selection operation on the smart glasses 301 or the smart mobile terminal 302 .

[0132] Specifically, the user may also select the source language and / or the target language by using the control buttons or touch sensors on the smart glasses 301 or the smart mobile terminal 302. If the user performs the selection operation through the smart glasses 301, the smart glasses 301 sends the identification information of the source language and / or the target language selected by the user to the smart mobile terminal 302 while sending the first voice data to the smart mobile terminal 302.

[0133] In an actual application, when a user makes a selection operation through the smart glasses 301, the smart glasses 301 can also output corresponding voice prompts in real time according to the user's operation to help the user quickly select the source language and / or the target language from multiple alternative languages.

[0134] Preferably, the Bluetooth protocol is used as the communication protocol between the smart mobile terminal 302 and the smart glasses 301, and the cellular mobile communication protocol (such as 2G, 3G, 4G, 5G protocol, etc.) is used as the communication protocol between the cloud computer device 303 and the smart mobile terminal 302.

[0135] It should be noted that the specific implementation process of the functions of the smart glasses 301, the smart mobile terminal 302 and the cloud computer device 303 in this embodiment can also refer to the relevant descriptions in other embodiments.

[0136] In this embodiment, the real-time hearing aid and real-time translation functions based on smart glasses are realized, thereby expanding the functions of smart glasses and improving product stickiness. In addition, since there is no need to install additional hearing aids on smart glasses, the weight of smart glasses can be reduced, power consumption can be reduced, and the manufacturing cost of smart glasses can be reduced. At the same time, since the microphone on one side obtains the voice data to be amplified from a specific direction and the loudspeaker on the other side plays the amplified voice data, the echo can be reduced, the quality of the played voice data and the voice data to be translated can be improved, and the translation efficiency can be improved.

[0137] See also Figure 6 , Figure 6 The following is a schematic diagram of the implementation process of the hearing aid method based on smart glasses provided in one embodiment of the present application. The method can be applied to Figures 1 to 3 The smart glasses shown. Figure 6 As shown, the method comprises the following steps:

[0138] S501, obtaining first voice data from a first preset direction through at least one microphone in the smart glasses;

[0139] S502, amplifying the first voice data through the wireless communication module in the smart glasses to obtain amplified voice data;

[0140] S503: Play the amplified voice data through a target speaker in the smart glasses, wherein the target speaker and the at least one microphone are respectively arranged on different temples of the smart glasses.

[0141] In an embodiment, at least one microphone in the smart glasses is used to obtain first voice data from a first preset direction, and the first voice data is amplified by a wireless communication module in the smart glasses and the amplified first voice data is sent to a target speaker among the multiple speakers for playback, wherein the target speaker and the at least one microphone are respectively arranged on different temples, thereby realizing a real-time hearing aid function based on the smart glasses, thereby expanding the function of the smart glasses and improving product stickiness. At the same time, since the microphone on one side obtains the voice data to be amplified from a specific direction and the speaker on the other side plays the amplified voice data, the echo can be reduced and the quality of the played sound can be improved.

[0142] Optionally, in another embodiment of the present application, the smart glasses have an idle state and a working state that can be switched freely, wherein the smart glasses in the working state also include three working modes: a hearing aid mode, a call mode, and a translation mode. In the call mode, the user can use the telephone call function of the smart glasses to have a telephone conversation with the person on the other end of the network through a mobile communication terminal and a wireless communication network. In the hearing aid mode, the user can use the hearing aid function of the smart glasses to have a face-to-face conversation with other people. In the translation mode, the user can use the translation function of the smart glasses for real-time translation, which can help the user better understand the voice content from speakers of different languages.

[0143] Furthermore, in another embodiment of the present application, the method further includes the following steps:

[0144] The wireless communication module responds to the first switching instruction to switch the working mode of the smart glasses to the hearing aid mode.

[0145] Step S502, amplifying the first voice data by the wireless communication module in the smart glasses to obtain amplified voice data specifically includes:

[0146] In the hearing aid mode, the first voice data is amplified by the wireless communication module to obtain the amplified voice data.

[0147] Optionally, in another embodiment of the present application, a hearing aid function can be further automatically provided for a specific user, and the output of the second voice data includes: determining whether the currently logged-in user is the target user; if it is the target user, amplifying the second voice data through the wireless communication module, and outputting the amplified second voice data through the target speaker; if it is not the target user, outputting the second voice data through the target speaker.

[0148] Specifically, before using the functions provided by the smart glasses, the user can register a corresponding user account through the smart mobile terminal, and after logging into the operating system of the smart glasses through the smart mobile terminal using the user account, set whether the user account is used by a specific user with hearing aid needs. If the user selects that the user account is used by a specific user with hearing aid needs, the smart mobile terminal adds a corresponding mark to the user account according to the user's setting operation.

[0149] Each time after the smart glasses complete Bluetooth pairing with the smart mobile terminal, they send a query request to the smart mobile terminal to confirm whether the currently logged-in user is the target user, and save the query result locally. Each time before the smart glasses output the second voice data, they determine whether an additional hearing aid function is required based on the query result saved locally, that is, whether the translated voice data needs to be amplified. In response to the query request, the smart mobile terminal confirms whether the currently logged-in user is the target user based on the above-mentioned mark, and returns the confirmation result as the query result to the smart glasses.

[0150] Furthermore, in another embodiment of the present application, the method further includes the following steps:

[0151] S601, switching the working mode of the smart glasses to a translation mode in response to a second switching instruction through the wireless communication module;

[0152] S602: in the translation mode, obtaining first voice data from a first preset direction through the at least one microphone;

[0153] S603, sending the first voice data to the smart mobile terminal through the wireless communication module, and receiving second voice data sent by the smart mobile terminal, wherein the second voice data is obtained by translating the first voice data;

[0154] S604: Play the second voice data through the target speaker.

[0155] Specifically, for real-time language translation, the voice signal is picked up by the smart glasses from the microphone, and the smart glasses process the voice signal locally and transmit it to the smart mobile terminal. After receiving the first voice data, the smart mobile terminal can use a local translation engine to translate the first voice data into the second voice data in the target language.

[0156] Alternatively, the built-in application of the smart mobile terminal uses a cloud-based AI (Artificial Intelligence) system to translate the voice signal from the source language to the target language in real time, and sends the translated voice signal back to the smart glasses and plays it for the user.

[0157] Specifically, the smart mobile terminal can choose to send the first voice data or the first text data corresponding to the first voice data to the cloud computer device for translation according to the preset conversion mark, receive the second text data in the target language returned by the cloud computer device, convert the second text data into the second voice data and send it to the smart glasses.

[0158] Optionally, the smart mobile terminal may also display the second text data through a local display while sending the second voice data to the smart glasses, thereby helping the user to more intuitively understand the content of the other party's conversation in the form of graphics and / or text.

[0159] Furthermore, in another embodiment of the present application, the method further includes the following steps:

[0160] S701, switching the working mode of the smart glasses to a call mode in response to a third switching instruction through the wireless communication module;

[0161] S702, in the call mode, obtaining third voice data from a second preset direction through the microphone, and sending the third voice data to the target terminal through the wireless communication module;

[0162] S703. Receive the fourth voice data sent by the target terminal through the wireless communication module, and play the fourth voice data through multiple speakers in the smart glasses.

[0163] The first preset direction points to the front of the user, and the second preset direction points to the bottom of the user, that is, the direction of the user's mouth.

[0164] In other words, if Figure 7 As shown, in the translation mode and the hearing aid mode, the audio beam points forward, and the microphone captures the speech signal from the front. In the conversation mode, the speech signal is captured from the direction of the user's mouth, that is, the microphone captures the speech signal downward.

[0165] In addition, in translation mode and hearing aid mode, only one speaker on the temple can be used to output voice data, that is, one side picks up the sound and the other side broadcasts it, which can reduce echo. In call mode, the speakers on all temples can be used to output voice data. In this way, the playback requirements of different working modes can be taken into account, thus achieving a better playback effect.

[0166] Optionally, the user can trigger the above switching instruction by using an input device configured on the smart glasses, such as a physical button or a virtual button (touch sensor). For example, the smart glasses can be provided with a plurality of state control buttons corresponding to different operating states and working modes, and the smart glasses respond to the switching instruction triggered by the control operation of the user clicking the button through the wireless communication module, and control the smart glasses to enter the operating state or working mode corresponding to the clicked button.

[0167] Optionally, according to the mode selection operation of the user on the smart glasses or smart mobile terminal, any one of the hearing aid mode and the translation mode can be selected as the commonly used listening mode. Figure 8 As shown, if the smart glasses are in an idle state, the physical buttons or virtual buttons (touch sensors) can be used to manually switch between the call mode and the listening mode. For example, if the smart glasses are in the call mode, pressing the physical button will change the smart glasses from the call mode to the listening mode, and pressing the physical button again will change back to the call mode, thereby reducing the number of buttons and manufacturing costs, while improving the intelligence of the operation and making the operation more in line with the user's usage habits.

[0168] Alternatively, when the user's target control operation is detected by the touch sensor, the wireless communication module responds to the switching instruction triggered by the target control operation to control the switching of the working mode of the smart glasses to a call mode or a listening mode, wherein the specific form of the target control operation can be customized by the user. For example, the action of long pressing the touch sensor or a preset sliding gesture can be set as the target control operation.

[0169] Furthermore, the DSP in the wireless communication module can execute various corresponding algorithms on the voice data.

[0170] like Fig. 9 As shown, in the call mode, the smart glasses perform voice equalization processing and output volume processing on the data of the downlink channel through the DSP, and perform echo cancellation processing, first beamforming processing and noise suppression processing on the data of the uplink channel in sequence. The data of the downlink channel is input through the wireless signal transceiver in the wireless communication module. The wireless signal transceiver preferably uses the Bluetooth protocol as the communication protocol.

[0171] Furthermore, in the call mode, the smart glasses receive the wireless signal input (such as Fig. 9The call voice data from the smart mobile terminal (wireless Bluetooth input of the smart glasses in the embodiment of the present invention) is processed by voice equalization and output volume control in sequence, and the call voice data after the output volume control processing is sent to the speaker of the smart glasses for output. At the same time, the DSP performs echo cancellation processing on the voice data obtained by the sound pickup device, using the call voice data after the output volume processing as a reference signal, and then performs beamforming processing and noise suppression processing, and outputs the data after the noise suppression processing to the smart mobile terminal through the wireless signal transceiver.

[0172] Since the distant person's speech will be played through the speaker, the distant voice signal will be received by the microphone, thus generating a loop. The echo cancellation processing of the voice data of the uplink channel uses the echo cancellation algorithm to compare the output signal of the speaker with the input signal of the microphone, thereby eliminating the echo and interrupting the loop chain between the speaker and the microphone.

[0173] The noise suppression processing for the voice data of the uplink channel uses a noise suppression algorithm to reduce or eliminate the volume of the noise and amplify the volume of the other party's speech at the same time. By using the noise suppression algorithm, even if the user is in a place with a lot of ambient noise, the noisy ambient noise cannot be heard from a distance, and the clear voice of the user of the smart glasses can be heard by the smart glasses.

[0174] The voice equalization processing performed on the voice data of the downlink channel is to use a voice equalizer to perform voice equalization processing on the distant voice signal to strengthen the frequency signal of the user's weak hearing, thereby achieving the purpose of compensating the weak hearing frequency signal.

[0175] The output volume control processing of the voice data of the downlink channel is to adjust the output volume of the speaker using the output volume control algorithm.

[0176] Optionally, the wireless signal transceiver adopts at least one of the Bluetooth protocol, Wireless Fidelity protocol, Near Field Communication protocol, ZigBee, Digital Living Network Alliance protocol, carrier-free communication protocol, radio frequency identification protocol and cellular mobile communication protocol as a communication protocol for data interaction with the smart mobile terminal.

[0177] Furthermore, in other embodiments of the present application, before acquiring the first voice data or the third voice data through the sound pickup device, the method further includes:

[0178] The wireless communication module is used to perform Bluetooth pairing with the smart mobile terminal based on the Bluetooth protocol to establish a data transmission channel between the smart glasses and the smart mobile terminal. Subsequent data interaction between the smart glasses and the smart mobile terminal can be carried out through the data transmission channel.

[0179] like Fig.10As shown, in the hearing aid mode, in order to achieve hearing loss assistance, the smart glasses use the DSP to perform feedback cancellation processing, voice activity detection processing, noise suppression processing, voice equalization processing, and user speech detection processing on the voice data, and send the voice data after voice equalization processing to the speakers of the smart glasses for output. At the same time, the voice data after voice equalization processing is used as reference data for the feedback cancellation processing. Furthermore, before sending the voice data after voice equalization processing to the speakers of the smart glasses for output, the voice data after voice equalization processing can also be processed for output gain and output volume control. The specific processing method is the same as Fig. 9 The output volume control processing method is the same as shown in Fig. 9 The relevant instructions will not be repeated here.

[0180] In the hearing aid mode, since the sound waves output by the speaker will be immediately received by the microphone array and form a loop, a harsh howling sound will be generated. Feedback cancellation processing is to use the feedback cancellation algorithm to eliminate the echo by comparing the output signal of the speaker and the input signal of the microphone array, and interrupt the loop chain between the speaker and the microphone array. Specifically, Fig.11 As shown in the figure, the microphone acquires the speech signal in front, but because the general microphone is omnidirectional, the speech signal output by the speaker is also acquired by the microphone at the same time. The speech signal of the speaker is also fed back to the input of the adaptive filter through the internal connection. The adaptive filter adjusts the LMS (least mean square) adaptive filter coefficient according to the previous speaker output data, thereby deleting the signal output by the speaker (i.e. f(n)) from the microphone input.

[0181] In addition, in the hearing aid mode, the other party's voice needs to be amplified so that users with hearing impairments can hear what the other party is saying. However, in the process of amplifying the voice, the voice and noise will be amplified together, causing discomfort to the user. Noise suppression processing uses a noise suppression algorithm to reduce or eliminate the noise volume and amplify the other party's voice volume at the same time. The algorithm specifically divides the voice signal of each frame into speaking and silent. If the frame is detected as a silent frame, the algorithm uses the voice signal of this frame to update the average noise energy value. If the frame is defined as a speaking frame, the voice signal of this frame will be subtracted from the average noise energy value to achieve noise suppression.

[0182] In addition, in the hearing aid mode, users with hearing impairment may not be able to hear or cannot hear clearly the sound of a certain frequency. Voice equalization processing is to use a voice equalizer to strengthen the sound signal of the specific frequency, so as to achieve the purpose of compensating the sound signal of the specific frequency.

[0183] In addition, since the microphone array of the smart glasses is very close to the user's mouth, when the user speaks, the microphone array will receive a large signal and play it on the speaker of the smart glasses, so that when the user speaks, he will hear his own voice through the speaker. User voice detection processing uses the user voice detection algorithm to continuously detect the signal received by the microphone array and perform average energy estimation analysis and detection. If the average energy estimation value is lower than the preset average energy threshold, the signal is detected as the user's voice.

[0184] The voice activity detection process is similar to the user voice detection process, except that the average energy threshold is lower than that of the user voice detection algorithm, thereby detecting the voice of the other party speaking.

[0185] Optionally, in another embodiment of the present application, an independent playback device (such as an external speaker, an external hearing aid) can also be used to play the voice data, and the above-mentioned step of outputting the amplified voice data or the second voice data may specifically include: determining whether there is a playback device that establishes a data connection with the smart glasses through the wireless communication module; if so, sending the amplified voice data or the second voice data to the playback device through the wireless communication module, so that the amplified voice data or the second voice data is played through the playback device; if not, outputting the amplified voice data or the second voice data through the target speaker. Like this, when there is an independent playback device, using an independent playback device to play voice data can reduce the power consumption of the smart glasses. Among them, the wireless communication module of the smart glasses can determine whether there is a playback device that establishes a data connection with the smart glasses based on the local Bluetooth pairing log and the connection status of the peripheral interface.

[0186] In the several embodiments provided in the present application, it should be understood that the disclosed smart glasses, auxiliary systems and hearing aid methods can be implemented in other ways. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not performed. Another point is that the connections or direct connections or communication connections shown or discussed can be indirect connections or communication connections through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0187] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0188] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0189] The above is a description of the smart glasses, hearing aid method and auxiliary system provided by the present application. For technicians in this field, according to the ideas of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A pair of smart glasses for hearing aids, It is characterized in that include: Two temples, a sound pickup device, multiple speakers, and a wireless communication module; The multiple speakers and the sound pickup device are arranged on at least one of the two temples, the wireless communication module is arranged in the cavity of any one of the two temples and is electrically connected to the sound pickup device and the multiple speakers, and the sound pickup device includes at least one microphone; The at least one microphone is used to obtain first voice data from a first preset direction; The wireless communication module is used to amplify the first voice data to obtain amplified voice data, and send the amplified voice data to a target speaker among the multiple speakers for playback, wherein the target speaker and the at least one microphone are respectively arranged on different temples.

2. The smart glasses according to claim 1, It is characterized in that Also includes: A spectacles frame, and an input device electrically connected to the wireless communication module, wherein the input device is arranged on the spectacles frame and at least one of the two spectacles legs; The wireless communication module is further used to switch the working mode of the smart glasses to the hearing aid mode in response to a first switching instruction triggered by the user based on the input device; The wireless communication module is further configured to amplify the first voice data in the hearing aid mode.

3. The smart glasses according to claim 2, It is characterized in that The wireless communication module is further used to switch the working mode of the smart glasses to a translation mode in response to a second switching instruction triggered by the user based on the input device; The wireless communication module is also used to send the first voice data to the smart mobile terminal in the translation mode, receive the second voice data sent by the smart mobile terminal, and send the second voice data to the target speaker for playback, where the second voice data is translated from the first voice data.

4. The smart glasses according to claim 2, It is characterized in that The wireless communication module is further configured to switch the working mode of the smart glasses to a call mode in response to a third switching instruction triggered by the user based on the input device; The at least one microphone is further used to obtain third voice data from a second preset direction in the call mode; The wireless communication module is further used to send the third voice data to the target terminal and receive the fourth voice data sent by the target terminal; The multiple speakers are also used to play the fourth voice data.

5. The smart glasses according to claim 4, It is characterized in that The first preset direction points to the front of the user, and the second preset direction points to the bottom of the user.

6. A hearing aid method based on smart glasses, It is characterized in that The smart glasses include: two temples, a wireless communication module, and a sound pickup device and a plurality of speakers electrically connected to the wireless communication module, wherein the sound pickup device includes at least one microphone, and the method includes: Acquiring first voice data from a first preset direction through the at least one microphone; amplifying the first voice data through the wireless communication module to obtain amplified voice data; The amplified voice data is played through a target speaker among the multiple speakers, wherein the target speaker and the at least one microphone are respectively arranged on different temples.

7. The method according to claim 6, It is characterized in that The method further comprises: In response to a first switching instruction, the wireless communication module switches the working mode of the smart glasses to a hearing aid mode; The step of amplifying the first voice data by the wireless communication module to obtain amplified voice data includes: In the hearing aid mode, the first voice data is amplified by the wireless communication module to obtain amplified voice data.

8. The method according to claim 6, It is characterized in that The method further comprises: In response to a second switching instruction, the wireless communication module switches the working mode of the smart glasses to a translation mode; In the translation mode, acquiring the first voice data through the at least one microphone; Sending the first voice data to the smart mobile terminal through the wireless communication module, and receiving the second voice data sent by the smart mobile terminal, wherein the second voice data is obtained by translating the first voice data; The second voice data is played through the target speaker.

9. The method according to claim 8, It is characterized in that The smart mobile terminal receives the first voice data sent by the smart glasses, and according to a preset conversion mark, selects to send the first voice data or the first text data corresponding to the first voice data to a cloud computer device for translation, receives the second text data in the target language returned by the cloud computer device, converts the second text data into the second voice data and sends it to the smart glasses.

10. The method according to claim 6, It is characterized in that The method further comprises: In response to a third switching instruction, the wireless communication module switches the working mode of the smart glasses to a call mode; In the call mode, acquiring third voice data from a second preset direction through the at least one microphone; Sending the third voice data to a target terminal through the wireless communication module; The fourth voice data sent by the target terminal is received through the wireless communication module, and the fourth voice data is played through the multiple speakers.

11. The method according to claim 10, It is characterized in that The first preset direction points to the front of the user, and the second preset direction points to the bottom of the user.

12. An auxiliary system, It is characterized in that include: Smart glasses and smart mobile terminals; The smart glasses include: two temples, a sound pickup device, a plurality of speakers, and a wireless communication module, wherein the plurality of speakers and the sound pickup device are arranged on at least one of the two temples, the wireless communication module is arranged in a cavity of any one of the two temples and is electrically connected to the sound pickup device and the plurality of speakers, and the sound pickup device includes at least one microphone; The wireless communication module is used to control the switching of the working modes of the smart glasses in response to the control operation of the user, and the working modes include: a hearing aid mode and a translation mode; The at least one microphone is used to obtain first voice data from a first preset direction in the hearing aid mode or the translation mode; The wireless communication module is used for, in the hearing aid mode, amplifying the first voice data to obtain amplified voice data, and sending the amplified voice data to a target speaker among the multiple speakers for playback, wherein the target speaker and the at least one microphone are respectively arranged on different temples; The wireless communication module is further used to send the first voice data to the smart mobile terminal in the translation mode; The intelligent mobile terminal is used to receive the first voice data and send second voice data to the wireless communication module, wherein the second voice data is obtained by translating the first voice data; The wireless communication module is further used to receive second voice data sent by the smart mobile terminal, and send the second voice data to the target speaker for playing.

13. The system of claim 12, It is characterized in that The system also includes a cloud computing device; The smart mobile terminal is further used to send the first voice data or the first text data corresponding to the first voice data as data to be translated to the cloud computer device; The cloud computer device is used to translate the data to be translated into second text data in a target language, and send the second text data to the smart mobile terminal; The smart mobile terminal is further used to convert the second text data into the second voice data and send the second voice data to the smart glasses.

14. The system of claim 13, It is characterized in that The smart mobile terminal is also used to obtain a preset conversion mark, and if the preset conversion mark is a first mark, convert the first voice data into the first text data and send the first text data to the cloud computer device, and if the preset conversion mark is a second mark, send the first voice data to the cloud computer device.

15. The system of claim 14, It is characterized in that The intelligent mobile terminal is also used to obtain the network speed and determine whether the network speed is greater than a preset speed. If it is not greater than the preset speed, the conversion mark is set to the first mark; if it is greater than the preset speed, the conversion mark is set to the second mark.

16. The system of claim 14, It is characterized in that The intelligent mobile terminal is further configured to, in response to a setting operation of a user, set the conversion mark to the mark pointed to by the setting operation.

17. The system of claim 12, It is characterized in that The working mode also includes a conversation mode; The at least one microphone is further used to obtain third voice data from a second preset direction in the call mode; The wireless communication module is further used to send the third voice data to the target terminal and receive the fourth voice data sent by the target terminal; The multiple speakers are also used to play the fourth voice data.