Smart glasses for hearing assistance, hearing assistance method, and auxiliary system

Abstract

Smart glasses for hearing assistance, a hearing assistance method, and a system. The smart glasses comprise: two glasses temples, a sound pickup apparatus, a plurality of loudspeakers and a wireless communication module, wherein the sound pickup apparatus comprises at least one microphone for acquiring first speech data from a first preset direction; and the wireless communication module is used for amplifying the first speech data to obtain amplified speech data, and sending the amplified speech data to a target loudspeaker for playing, and the target loudspeaker and the at least one microphone are respectively arranged on different glasses temples. By means of the present application, a hearing assistance function based on smart glasses is realized, and the product stickiness can be increased.

Description

Smart glasses for hearing assistance, hearing assistance method and auxiliary system

[0001] This application claims priority to the Chinese patent application filed with the Patent Office of the State Intellectual Property Office of China on May 31, 2023, with application number CN 202310639291.8 and entitled “Smart glasses, hearing aid method and auxiliary system for hearing aid”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the technical field of smart glasses, and in particular to smart glasses for hearing assistance, a hearing assistance method, and an auxiliary system. Background Art

[0003] With the development of computer technology, smart glasses are becoming more and more popular. However, existing smart glasses are expensive, and in addition to their own functions as smart glasses, they usually only have the functions of listening to music and making or receiving calls. The functions are relatively simple and the product stickiness is low. Technical issues

[0004] The embodiments of the present application provide smart glasses for hearing assistance, a hearing assistance method, and an auxiliary system, which are used to realize a real-time hearing assistance function based on smart glasses, thereby increasing product stickiness. Technical Solutions

[0005] On the one hand, an embodiment of the present application provides smart glasses for hearing assistance, comprising: two temples, a sound pickup device, a plurality of speakers, and a wireless communication module;

[0006] The multiple speakers and the sound pickup device are arranged on at least one of the two temples, the wireless communication module is arranged in a cavity of any one of the two temples and is electrically connected to the sound pickup device and the multiple speakers, and the sound pickup device includes at least one microphone;

[0007] The at least one microphone is configured to acquire first voice data from a first preset direction;

[0008] The wireless communication module is used to amplify the first voice data to obtain amplified voice data, and send the amplified voice data to a target speaker among the multiple speakers for playback, wherein the target speaker and the at least one microphone are respectively arranged on different temples.

[0009] In one aspect, an embodiment of the present application further provides a hearing aid method based on smart glasses, wherein the smart glasses include: two temples, a wireless communication module, and a sound pickup device and multiple speakers electrically connected to the wireless communication module, wherein the sound pickup device includes at least one microphone. The method includes:

[0010] acquiring first voice data from a first preset direction through the at least one microphone;

[0011] amplifying the first voice data by the wireless communication module to obtain amplified voice data;

[0012] The amplified voice data is played through a target speaker among the multiple speakers, wherein the target speaker and the at least one microphone are respectively arranged on different temples.

[0013] In one aspect, an embodiment of the present application further provides an auxiliary system, comprising: smart glasses and a smart mobile terminal;

[0014] The smart glasses include: two temples, a sound pickup device, a plurality of speakers, and a wireless communication module, wherein the plurality of speakers and the sound pickup device are arranged on at least one of the two temples, the wireless communication module is arranged in a cavity of any one of the two temples and is electrically connected to the sound pickup device and the plurality of speakers, and the sound pickup device includes at least one microphone;

[0015] The wireless communication module is configured to control switching of the operating modes of the smart glasses in response to a user's control operation, wherein the operating modes include: a hearing aid mode and a translation mode;

[0016] The at least one microphone is configured to acquire first voice data from a first preset direction in the hearing aid mode or the translation mode;

[0017] The wireless communication module is configured to, in the hearing aid mode, amplify the first voice data to obtain amplified voice data, and send the amplified voice data to a target speaker among the plurality of speakers for playback, wherein the target speaker and the at least one microphone are respectively disposed on different temples;

[0018] The wireless communication module is further configured to send the first voice data to the smart mobile terminal in the translation mode;

[0019] The intelligent mobile terminal is configured to receive the first voice data and send second voice data to the wireless communication module, wherein the second voice data is obtained by translating the first voice data;

[0020] The wireless communication module is further configured to receive second voice data sent by the smart mobile terminal, and send the second voice data to the target speaker for playback. Beneficial effects

[0021] In each embodiment of the present application, at least one microphone in the smart glasses is used to obtain first voice data from a first preset direction, and the wireless communication module in the smart glasses is used to amplify the first voice data and send the amplified first voice data to a target speaker among the multiple speakers for playback, wherein the target speaker and the at least one microphone are respectively arranged on different temples, thereby realizing a real-time hearing aid function based on the smart glasses, thereby expanding the functions of the smart glasses and improving product stickiness. At the same time, since the microphone on one side obtains the voice data to be amplified from a specific direction and the speaker on the other side plays the amplified voice data, the echo can be reduced and the quality of the played sound can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] FIG1 is a schematic diagram of the structure of smart glasses provided in one embodiment of the present application;

[0024] FIG2 is a schematic diagram of the internal structure of smart glasses provided by another embodiment of the present application;

[0025] FIG3 is a schematic diagram of the external structure of the smart glasses in the embodiment shown in FIG2 ;

[0026] FIG4 is a schematic diagram of an auxiliary system provided in one embodiment of the present application;

[0027] FIG5 is a schematic diagram of the hardware structure of the intelligent mobile terminal in the auxiliary system shown in FIG4 ;

[0028] FIG6 is a schematic diagram of an implementation flow of a hearing aid method based on smart glasses provided in one embodiment of the present application;

[0029] FIG7 is a schematic diagram of the sound beam direction of the microphone in different working modes in the hearing aid method based on smart glasses provided in an embodiment of the present application;

[0030] FIG8 is a schematic diagram of the operating state and working mode of smart glasses in the hearing aid method based on smart glasses provided in an embodiment of the present application;

[0031] FIG9 is a schematic diagram of a process for processing a voice signal in a call mode in a hearing aid method based on smart glasses provided in an embodiment of the present application;

[0032] FIG10 is a schematic diagram of a process of processing a speech signal in a hearing aid mode in a hearing aid method based on smart glasses provided in an embodiment of the present application;

[0033] FIG. 11 is a schematic diagram of the feedback cancellation process in FIG. 10 . Modes for Carrying Out the Invention

[0034] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0035] Referring to Figure 1 , Figure 1 is a schematic diagram of the structure of smart glasses provided in one embodiment of the present application. For ease of illustration, only the portions relevant to the present embodiment are shown in the figure. As shown in Figure 1 , smart glasses 10 include: a wireless communication module 101, a sound pickup device 102, multiple speakers 103, and two temples 104. The wireless communication module 101 is electrically connected to the sound pickup device 102 and the multiple speakers 103.

[0036] The multiple speakers 103 and the sound pickup device 102 are arranged on at least one of the two temples 104 , and the wireless communication module 101 is arranged in the cavity of any one of the two temples 104 and is electrically connected to the sound pickup device 102 and the multiple speakers 103 .

[0037] The sound pickup device 102 includes at least one microphone. Optionally, the sound pickup device 102 includes at least one directional microphone. Further, the sound pickup device 102 may also include at least one omnidirectional microphone. For example, the sound pickup device 102 may be a microphone array consisting of at least one directional microphone and / or at least one omnidirectional microphone.

[0038] Furthermore, the wireless communication module 101 may include, but is not limited to, a controller 1011, a voice data processor 1012, and a wireless signal transceiver 1013. The controller 1011 is primarily used for user interface (UI), task operations, and real-time operating system operations. The voice data processor 1012 is primarily used to execute time-consuming signal processing algorithms and Bluetooth wireless communication algorithms. The wireless signal transceiver may be, for example, a wireless signal transceiver.

[0039] The wireless signal transceiver can use, but is not limited to, at least one of the following protocols for data transmission: Bluetooth protocol, WiFi (Wireless Fidelity) protocol, NFC (Near Field Communication) protocol, ZigBee protocol, DLNA (DIGITAL LIVING NETWORK ALLIANCE) protocol, UWB (Ultra Wideband, carrier-free communication), RFID (Radio Frequency Identification) protocol, and cellular mobile communication protocol.

[0040] Specifically, at least one microphone is used to obtain first voice data from a first preset direction.

[0041] The wireless communication module 101 is used to amplify the first voice data to obtain amplified voice data, and send the amplified voice data to a target speaker among the multiple speakers 103 for playback, wherein the target speaker and at least one microphone are respectively arranged on different temples.

[0042] In this embodiment, at least one microphone in the smart glasses captures first voice data from a first preset direction, amplifies the first voice data using the wireless communication module in the smart glasses, and transmits the amplified first voice data to a target speaker among the multiple speakers for playback. The target speaker and the at least one microphone are located on different temples. This implements real-time hearing assistance functionality within the smart glasses, thereby expanding the functionality of the smart glasses and improving product appeal. Furthermore, since no additional hearing aids are required, the weight of the smart glasses is reduced, power consumption is lowered, and manufacturing costs are lowered. Furthermore, since the microphone on one side captures the voice data to be amplified from a specific direction and the speaker on the other side plays the amplified voice data, echoes are reduced and the quality of the playback sound is improved.

[0043] Referring to Figures 2 and 3 , Figure 2 is a schematic diagram of the internal structure of smart glasses provided in another embodiment of the present application, and Figure 3 is a schematic diagram of the external structure of the smart glasses in the embodiment shown in Figure 2 . As shown in Figures 2 and 3 , smart glasses 20 include: a frame 201, two temples 202, at least one sound pickup device 203 (only one is shown in the figure for ease of understanding), a first speaker 205, a second speaker 206, and a wireless communication module 204.

[0044] The temple 202 is connected to the frame 201, the first speaker 205, the second speaker 206 and the at least one sound pickup device 203 are arranged on at least one of the two temples 202, and the wireless communication module 204 is arranged in the cavity of any one of the two temples 202 and is electrically connected to the first speaker 205, the second speaker 206 and the at least one sound pickup device 203.

[0045] The sound pickup device 203 is configured to acquire first voice data from a first preset direction.

[0046] The wireless communication module 204 is used to amplify the first voice data to obtain amplified voice data, and send the amplified voice data to a target speaker among the first speaker 205 and the second speaker 206 for playback, wherein the target speaker and the sound pickup device 203 are respectively arranged on different temples.

[0047] The wireless communication module 204 is also used to control the switching of the working modes of the smart glasses, which include a call mode, a hearing aid mode, and a translation mode.

[0048] The sound pickup device 203 includes: at least one directional microphone and / or at least one omnidirectional microphone. When the sound pickup device 203 is an omnidirectional microphone, the wireless communication module 204 is further configured to perform a first beamforming process on the voice data captured by the at least one sound pickup device 203 in call mode, so that the sound beam of the at least one sound pickup device 203 is directed downward, i.e., in correspondence with the user's mouth. In this case, the source of the voice data may be the wearer of the smart glasses. In the hearing aid mode and the translation mode, the voice data captured by the at least one sound pickup device 203 is subjected to a second beamforming process, so that the sound beam of the at least one sound pickup device 203 is directed forward. In this case, the source of the voice data may be the person the wearer of the smart glasses is conversing with. After switching operating modes, the wireless communication module 204 can automatically control the omnidirectional microphone to point in the corresponding direction according to the switched operating mode.

[0049] Optionally, the two temples 202 include a first temple 202A and a second temple 202B, the front ends of the first temple 202A and the second temple 202B are respectively connected to two sides of the frame 201, and at least one sound pickup device 203 is installed at the front end of the first temple 202A.

[0050] The first speaker 205 and the second speaker 206 are used to output voice data or music data. The voice data includes: voice data obtained by the sound pickup device 203 after being processed by the wireless communication module 204, and call voice data sent by the smart mobile terminal and received by the wireless communication module 204. The music data includes: music data sent by the smart mobile terminal and received by the wireless communication module 204.

[0051] Preferably, the first speaker 205 is mounted on the first temple 202A, and the output port of the first speaker 205 is located at the rear end of the first temple 202A.

[0052] The second speaker 206 is mounted on the second temple 202B, and an output port of the second speaker 206 is located at the rear end of the second temple 202B.

[0053] In hearing aid mode and translation mode, only the open speaker located on one temple is used to output voice data, thereby achieving the effect of reducing echo.

[0054] For example, in this hearing aid mode, if the sound pickup device of the smart glasses is located on the left temple, the amplified voice signal will only be output to the speaker configured on the right temple for playback. If the sound pickup device is located on the right temple, the amplified voice will be output to the speaker configured on the left temple for playback.

[0055] Optionally, in another embodiment of the present application, when a microphone and a speaker are provided on each temple, each microphone and each speaker has corresponding unique identification information, and the wireless communication module 204 stores the corresponding relationship between the unique identification information of each microphone and each speaker and their respective locations. The wireless communication module 204 is further configured to determine the installation location of the microphone for acquiring the first voice data based on the corresponding relationship, and to determine the target microphone based on the installation location and the corresponding relationship.

[0056] It can be understood that for the hearing aid mode, the smart glasses can independently amplify the voice signal in front of the wearer without connecting to any mobile terminal device.

[0057] Since the sound pickup device is installed at the front end of the temple and the output port of each speaker is installed at the rear end of the temple, there can be a sufficient distance between the sound pickup device and the output port of each speaker, thereby effectively reducing the situation of loop generation between the sound pickup device and the speaker, and reducing the probability of echo and howling during use.

[0058] In addition, after the user wears the smart glasses, the tail end of the temple is closest to the user's ear. Installing the speaker outlet at the tail end of the temple can make the speaker outlet closest to the user's ear, thereby improving the efficiency of sound output.

[0059] The first speaker 205 and the second speaker 206 are preferably mono speakers. The mono speakers respectively installed on the two temples can be combined to achieve a stereo sound effect.

[0060] Optionally, the wireless communication module 204 includes: a controller 2041 , a voice data processor 2042 and a wireless signal transceiver 2043 .

[0061] The controller 2041 is used to control the switching of the working modes of the smart glasses. The controller 2041 is preferably an MCU (Microcontroller Unit).

[0062] Voice data processor 2042 is used to process voice data. Voice data processor 2042 is preferably a DSP (Digital Signal Processing) or a voice data processing integrated circuit. Voice data processing integrated circuits are commonly used, and this application does not specifically limit their structure.

[0063] In another embodiment of the present application, the sound pickup device 203 is preferably a directional microphone, which is used to directionally pick up voice signals and then send them to the MCU / DSP for voice processing.

[0064] Furthermore, the sound pickup device 102 also includes two microphones for picking up voice signals from a second preset direction. The two microphones on the smart glasses can pick up voice signals from the second preset direction for use in issuing voice commands and answering calls. Using two microphones to pick up sound can accurately determine the direction of the sound pickup, and processing the sound data from the two microphones can achieve noise reduction. The voice signal is picked up by the two microphones of the smart glasses, and after noise reduction, it is played back through the speakers of the smart glasses and produces sound.

[0065] The wireless signal transceiver 2043 is used for data interaction with the smart mobile terminal.

[0066] Optionally, the voice data processor 2042 includes a voice equalizer. The wireless signal transceiver 2043 is further configured to receive a volume adjustment control instruction sent by the smart mobile terminal and transmit it to the voice data processor 2042. The voice data processor 2042 is further configured to utilize the voice equalizer to adjust the sound data to be output to the frequency band and volume indicated by the volume adjustment control instruction, and transmit the adjusted sound data to the speaker indicated by the volume adjustment control instruction. The frequency band indicated by the volume adjustment control instruction can be low frequency, medium frequency, or high frequency. Volume adjustment can be to increase or decrease the volume.

[0067] Furthermore, the voice data processor 2042 is also used to perform voice equalization processing (using a voice equalizer) and output volume processing on the data on the downlink channel in call mode, and to perform echo cancellation processing, beamforming processing, and noise suppression processing on the data on the uplink channel using a preset acoustic echo cancellation algorithm (Acoustic Echo Cancellation), beamforming algorithm (Beamforming), and noise suppression algorithm (Noise Cancellation).

[0068] Furthermore, the voice data processor 2042 is further configured to, in hearing aid mode or translation mode, perform feedback cancellation, voice activity detection, noise suppression, voice equalization, and user speech detection processing on the voice data using a preset feedback cancellation algorithm, voice activity detection, noise suppression, voice equalization, and user speech detection algorithm, and then transmit the processed voice data to the corresponding speaker for output. Furthermore, the processed voice data also serves as reference data for the feedback cancellation processing during the processing process.

[0069] Optionally, the smart glasses further include: at least one sensor 207 electrically connected to the wireless communication module 204 .

[0070] At least one sensor 207 is mounted on the inner side and / or outer side of the first temple 102A and / or the second temple 102B.

[0071] Specifically, the at least one sensor 207 may include, but is not limited to, at least one of a touch sensor, a proximity sensor, an accelerometer, a gyroscope, a magnetic induction sensor, and an inertial measurement unit.

[0072] Optionally, the inertial measurement unit is a 9-axis sensor. The 9-axis sensor is used to collect the user's motion data (3-axis accelerometer data, 3-axis gyroscope data, and 3-axis magnetometer data) and send it to the smart mobile terminal via the wireless communication module 204 for monitoring the status of the user's head and body.

[0073] Preferably, the at least one sensor 207 includes: a 9-axis sensor, at least one touch sensor and at least one proximity sensor.

[0074] At least one touch sensor is installed on the outside of the first temple 102A and / or the second temple 102B.

[0075] Optionally, at least one touch sensor is configured to detect a first control operation performed by a user and transmit data of the detected first control operation to controller 2041, where the first control operation is used to adjust the volume. Controller 2041 is further configured to control and adjust the volume of sound output by the smart glasses in response to the first control operation based on the data of the first control operation.

[0076] The first control operation includes a control operation for increasing the volume and a control operation for decreasing the volume. The control operation for increasing the volume corresponds to an action of the user's finger swiping on the touch sensor toward the ear, and the control operation for decreasing the volume corresponds to an action of the user's finger swiping on the touch sensor toward the frame (i.e., away from the ear).

[0077] Optionally, at least one touch sensor is further configured to serve as an input device to detect a second control operation of the user and send data of the detected second control operation to the controller 2041. The controller 2041 is further configured to control switching of the operating mode of the smart glasses based on the data of the second control operation and in response to the second control operation.

[0078] Optionally, at least one proximity sensor is mounted on the inner side of the first temple 202A and / or the second temple 202B to detect whether the user is wearing or removing the smart glasses, obtain the length of time the user has not been wearing the smart glasses, and transmit the detection results to the controller 2041. The controller 2041 is configured to control the playback or pause of music and the power on or off of the smart glasses 20 based on the detection results. For example, the controller 2041 may play music data when the proximity sensor detects that the user is wearing the smart glasses, stop playing the music data when the proximity sensor detects that the user has removed the smart glasses, and shut down the smart glasses 20 when the proximity sensor detects that the user has not been wearing the smart glasses for a preset period of time.

[0079] The number of each of the above sensors is preferably one to reduce the overall weight of the smart glasses. However, in specific applications, the number of each sensor is not limited to one according to actual needs. For example, to improve the accuracy of the detection results, a proximity sensor can be set on each of the two temples.

[0080] Optionally, the controller 2041 is further configured to send the data acquired by each sensor to the smart mobile terminal via the wireless signal transceiver.

[0081] Optionally, the sound pickup device 203 may include a first directional microphone and a second directional microphone. The first directional microphone is used to capture voice data from the user's front, and the second directional microphone is used to capture voice data from below the user. The first directional microphone and the second directional microphone may be located at different positions on the temples. Furthermore, the controller 2041 may also control the directional microphones at corresponding positions to capture sound based on the operating mode of the smart glasses 20, thereby reducing noise in the captured voice data and improving signal processing speed.

[0082] Optionally, the smart glasses also include a battery 208, which is installed on the first temple 202A and electrically connected to the wireless communication module 204 to provide power for all hardware components on the smart glasses, such as the wireless communication module 204, sensors, speakers, and microphones.

[0083] Optionally, the smart glasses further include an input device electrically connected to the wireless communication module 204. The input device is provided on the frame 201 and at least one of the two temples 202.

[0084] Optionally, the input device includes at least one of a touch sensor and at least one control button (not shown). The at least one control button is mounted on the outside of the first temple 202A and / or the second temple 202B and is electrically connected to the wireless communication module 204. The at least one control button is used to trigger the wireless communication module 204 to control switching of the operating mode or operating state of the smart glasses. The operating state includes an idle state and an operating state, and the operating state includes a call mode, a hearing aid mode, and a translation mode.

[0085] Specifically, the wireless communication module 204 is also used to switch the working mode of the smart glasses to the hearing aid mode in response to a first switching instruction triggered by the user based on the input device, and in the hearing aid mode, amplify the first voice data from the first preset direction.

[0086] The wireless communication module 204 is also used to switch the working mode of the smart glasses to a translation mode in response to a second switching instruction triggered by the user based on the input device, and in the translation mode, send the first voice data to the smart mobile terminal, receive the second voice data sent by the smart mobile terminal, and send the second voice data to the target speaker for playback, wherein the second voice data is translated from the first voice data.

[0087] The wireless communication module 204 is also used to switch the working mode of the smart glasses to a call mode in response to a third switching instruction triggered by the user based on the input device. In this call mode, the third voice data from the second preset direction obtained by the microphone is sent to the target terminal, and the fourth voice data sent by the target terminal is received and output to all speakers for playback.

[0088] When the user of the smart glasses uses the smart glasses to directly communicate with the user of another terminal, the target terminal is the other terminal. When the user of the smart glasses uses the smart glasses to communicate with the user of another terminal via a smart mobile terminal, the target terminal is the smart mobile terminal.

[0089] The first preset direction points to the front of the user, and the second preset direction points to the bottom of the user.

[0090] The electronic components of the smart glasses can be connected via a bus.

[0091] It should be noted that the relationships between the components of the aforementioned smart glasses can be either replacement or superposition. That is, a pair of smart glasses can be equipped with all of the components described in this embodiment, or, depending on the needs, selectively install a portion of the components. In the case of replacement, the smart glasses also include a peripheral connection interface, which can be, for example, at least one of a PS / 2 interface, a serial interface, a parallel interface, an IEEE1394 interface, and a USB (Universal Serial Bus) interface. The functionality of the replaced component can be implemented by a peripheral connected to the interface, such as an external speaker or sensor.

[0092] The controller 2041 is further configured to determine whether a playback device, such as an external speaker or hearing aid, has established a data connection with the smart glasses. If so, the controller 2041 transmits the second voice data or the amplified first voice data to the playback device for playback. If not, the controller 2041 outputs the second voice data or the amplified first voice data through a local speaker. In this manner, when an independent playback device is available, using the independent playback device to play the voice data can further reduce the power consumption of the smart glasses.

[0093] In this embodiment, by utilizing at least one microphone in the smart glasses to capture first voice data from a first preset direction, utilizing a wireless communication module in the smart glasses to amplify the first voice data, and transmitting the amplified first voice data to a target speaker among the multiple speakers for playback, wherein the target speaker and the at least one microphone are located on different temples, a real-time hearing aid function is implemented within the smart glasses, thereby expanding the functionality of the smart glasses and improving product stickiness. Furthermore, since no additional hearing aids are required, the weight of the smart glasses is reduced, power consumption is lowered, and manufacturing costs are lowered. Furthermore, since the microphone on one side captures the voice data to be amplified from a specific direction and the speaker on the other side plays the amplified voice data, echoes are reduced and the quality of the playback sound is improved. Furthermore, through interaction between the smart glasses and a smart mobile terminal, a real-time translation function is implemented within the smart glasses, further expanding the functionality of the smart glasses. Furthermore, since the voice data to be translated is captured from a specific direction, the quality of the translated voice data is improved, thereby enhancing translation efficiency.

[0094] 4 , which is a schematic diagram of the structure of an auxiliary system provided in an embodiment of the present application. As shown in FIG4 , the auxiliary system includes: smart glasses 301 , a smart mobile terminal 302 , and a cloud computing device 303 .

[0095] The smart glasses include 301: two temples, a sound pickup device, a plurality of speakers, and a wireless communication module. The sound pickup device includes at least one microphone.

[0096] The wireless communication module is used to control the switching of the working modes of the smart glasses in response to the user's control operation, and the working modes include: hearing aid mode, translation mode and call mode.

[0097] The at least one microphone is configured to acquire first voice data from a first preset direction in the hearing aid mode or the translation mode;

[0098] The wireless communication module is used to amplify the first voice data to obtain amplified voice data in the hearing aid mode, and send the amplified voice data to a target speaker among the multiple speakers for playback, wherein the target speaker and the at least one microphone are respectively arranged on different temples.

[0099] The wireless communication module is also used to send the first voice data to the smart mobile terminal in the translation mode, receive the second voice data sent by the smart mobile terminal, and send the second voice data to the target speaker for playback, wherein the second voice data is translated from the first voice data.

[0100] The at least one microphone is further configured to acquire third voice data from a second preset direction in the call mode.

[0101] The wireless communication module is further configured to send the third voice data to a target terminal and receive fourth voice data sent by the target terminal.

[0102] The multiple speakers are also used to play the fourth voice data.

[0103] For details on the structure and functions of the smart glasses 301 , please refer to the relevant descriptions in the embodiments shown in FIG. 1 to FIG. 3 .

[0104] Smart mobile terminal 302 may include, but is not limited to, cellular phones, smartphones, other wireless communication devices, personal digital assistants (PDAs), audio players, other media players, music recorders, video recorders, cameras, other media recorders, smart radios, laptop computers, personal digital assistants (PDAs), portable multimedia players (PMPs), Moving Picture Experts Group (MPEG-1 or MPEG-2) Audio Layer 3 (MP3) players, digital cameras, and smart wearable devices (such as smart watches and smart bracelets). The smart mobile terminal 302 may be installed with an Android or iOS operating system.

[0105] The smart mobile terminal 302 is used to exchange data with the smart glasses 301 and the cloud computing device 303. Specifically, for example, it receives first voice data sent by the smart glasses 301, translates the first voice data into second voice data in the target language locally on the smart mobile terminal 302, or sends the first voice data or first text data corresponding to the first voice data to the cloud computing device 303 for translation; receives the translated second text data in the target language returned by the cloud computing device 303, converts the second text data into second voice data, and sends the second voice data to the smart glasses 301. For another example, when performing target tasks such as playing music data or making a call, the played music data and received call voice data are sent to the smart glasses 301. The communication protocol used by the smart mobile terminal 302 when interacting with the smart glasses 301 is consistent with the communication protocol used by the smart glasses 301.

[0106] Optionally, the smart mobile terminal 302 determines whether to send the first voice data or the first text data corresponding to the first voice data to the cloud computer device 303 according to a preset conversion mark.

[0107] Specifically, the smart mobile terminal 302 is also used to obtain a preset conversion mark. If the preset conversion mark is a first mark, the first voice data is converted into the first text data and the first text data is sent to the cloud computer device 303. If the preset conversion mark is a second mark, the first voice data is directly sent to the cloud computer device 303.

[0108] Furthermore, the smart mobile terminal 302 is also used to obtain the network speed and determine whether the network speed is greater than the preset speed; if it is not greater than the preset speed, the conversion mark is set to the first mark; if it is greater than the preset speed, the conversion mark is set to the second mark.

[0109] Furthermore, the smart mobile terminal 302 is further configured to, in response to a setting operation by the user, set the conversion mark to the mark pointed to by the setting operation.

[0110] As shown in FIG5 , a smart mobile terminal 302 may include control circuitry, which may include storage and processing circuitry 300. The storage and processing circuitry 300 may include memory, such as hard drive memory, non-volatile memory (e.g., flash memory or other electronically programmable, erasable memory such as that used in solid-state drives), volatile memory (e.g., static or dynamic random access memory), etc., though this is not limited in the present embodiment. The processing circuitry within the storage and processing circuitry 300 may be used to control the operation of the smart mobile terminal 302. The processing circuitry may be implemented based on one or more microprocessors, microcontrollers, digital signal processors, baseband processors, power management units, audio codec chips, application-specific integrated circuits, display driver integrated circuits, and the like.

[0111] The storage and processing circuit 300 may be used to run software in the smart mobile terminal 302, such as an artificial intelligence manager application, an internet browsing application, a Voice over Internet Protocol (VOIP) phone call application, an email application, a media playback application, operating system functions, etc. Such software may be used to perform certain control operations, such as data processing and analysis of motion data transmitted by the smart glasses 301 based on a preset analysis algorithm, image acquisition using a camera, ambient light measurement using an ambient light sensor, proximity sensor measurement using a proximity sensor, information display functions implemented using status indicators such as light-emitting diodes (LEDs), touch event detection using a touch sensor, functions associated with displaying information on multiple (e.g., layered) displays, operations associated with performing wireless communication functions, operations associated with collecting and generating audio signals, control operations associated with collecting and processing button press event data, and other functions in the smart mobile terminal 302, without limitation in the present embodiment.

[0112] Furthermore, the memory stores executable program code, and the processor connected to the memory calls the executable program code stored in the memory to execute the relevant steps in the following method embodiments of the present application to realize the various functions of the smart mobile terminal 302 in each embodiment of the present application.

[0113] The smart mobile terminal 302 may also include input / output circuitry 420. The input / output circuitry 420 may be used to enable the smart mobile terminal 302 to input and output data, i.e., to allow the smart mobile terminal 302 to receive data from external devices and to output data to external devices. The input / output circuitry 420 may further include sensors 320. Sensors 320 may include ambient light sensors, light- and capacitance-based proximity sensors, touch sensors (e.g., light-based touch sensors and / or capacitive touch sensors, where the touch sensor may be part of a touch display or used independently as a touch sensor structure), accelerometers, and other sensors.

[0114] The input / output circuitry 420 may also include one or more displays, such as display 140. Display 140 may include a liquid crystal display, an organic light-emitting diode display, an electronic ink display, a plasma display, or displays using other display technologies, or a combination thereof. Display 140 may include a touch sensor array (i.e., display 140 may be a touchscreen display). The touch sensors may be capacitive touch sensors formed by an array of transparent touch sensor electrodes (e.g., indium tin oxide (ITO) electrodes), or may be touch sensors formed using other touch technologies, such as acoustic touch, pressure-sensitive touch, resistive touch, optical touch, etc., without limitation in the present embodiment.

[0115] The smart mobile terminal 302 may further include an audio component 360. The audio component 360 may be used to provide audio input and output functions for the smart mobile terminal 302. The audio component 360 in the smart mobile terminal 302 may include a speaker, a sound pickup device, a buzzer, a tone generator, and other components for generating and detecting sound.

[0116] The communication circuit 380 can be used to provide the smart mobile terminal 302 with the ability to communicate with external devices. The communication circuit 380 may include analog and digital input / output interface circuits, and wireless communication circuits based on radio frequency signals and / or optical signals. The wireless communication circuitry in the communication circuit 380 may include radio frequency transceiver circuitry, power amplifier circuitry, low-noise amplifiers, switches, filters, and antennas. For example, the wireless communication circuitry in the communication circuit 380 may include circuitry for supporting near-field communication (NFC) by transmitting and receiving near-field connection electromagnetic signals. For example, the communication circuit 380 may include an NFC antenna and an NFC transceiver. The communication circuit 380 may also include a cellular phone transceiver and antenna, as well as wireless local area network (WLAN) transceiver circuitry and antennas such as Bluetooth, WiFi, ZigBee, DLNA, UWB, and RFID.

[0117] The smart mobile terminal 302 may further include a battery, power management circuits and other input / output units 400. The input / output units 400 may include buttons, joysticks, click wheels, scroll wheels, touch pads, keypads, keyboards, cameras, light emitting diodes and other status indicators.

[0118] The user can input commands through the input / output circuit 420 to control the operation of the smart mobile terminal 302 , and can use the output data of the input / output circuit 420 to receive status information and other outputs from the smart mobile terminal 302 .

[0119] Furthermore, the smart mobile terminal 302 is also used to pair with the smart glasses 301 based on the Bluetooth protocol, and after successful pairing, the played music data is sent to the smart glasses 301 so that the music data can be played through the smart glasses 301.

[0120] The smart glasses 301 are further configured to send the first voice data to be translated to the smart mobile terminal 302 for translation after successful pairing.

[0121] Optionally, the smart mobile terminal 302 is further configured to obtain GPS data through a GPS module configured on the smart mobile terminal, and send the obtained GPS data to the smart glasses 301 for positioning the smart glasses 301.

[0122] Optionally, the smart mobile terminal 302 is further configured to receive and store in real time the motion data sent by the smart glasses 301, calculate the motion index based on the motion data and the GPS data, generate real-time voice data for notifying or reminding the user of the motion status based on the calculation results, and send the real-time voice data to the smart glasses 301 for output. The real-time voice data includes a notification or reminder voice of the motion index calculation results.

[0123] Specifically, a client program, such as an artificial intelligence housekeeper App, can be installed on the smart mobile terminal 302, which can perform data interaction operations with the smart glasses 301, configure the parameters of the smart glasses 301 based on the user's operations on the App, and process and analyze the motion data sent by the smart glasses 301, such as: running index calculation, posture monitoring and reminders, etc.

[0124] The above-mentioned music data and real-time voice data belong to the data of the downlink channel of the smart glasses 301. The smart glasses 301 perform voice equalization processing and output volume control processing on the received music data and real-time voice data through the voice data processor 1042, and then send them to the first speaker 201 and the second speaker 202 for output.

[0125] Specifically, the motion data includes data obtained by the smart glasses 301 through a 9-axis sensor (i.e., Ax, Ay, Az; Gx, Gy, Gz; Mx, My, Mz). The motion indicators are running indicators, which include: pace, distance, number of steps, left and right head balance, number of steps, stride length, and cadence. Among them, the 9-axis sensor refers to the accelerometer, gyroscope, and magnetic induction sensor. The data measured by these three types of sensors can be decomposed into forces in the three directions of X, Y, and Z in the spatial coordinate system, and are therefore also called 3-axis accelerometers, 3-axis gyroscopes, and 3-axis magnetic induction sensors.

[0126] The smart mobile terminal 302 is further configured to perform index calculations, posture monitoring, and exercise reminders based on the motion data acquired by the 9-axis sensor and local GPS data, and to display the index calculation results in real time on the display of the smart mobile terminal 302. Furthermore, the smart mobile terminal 302 can also generate voice prompts based on the index calculation results and send the voice prompts to the smart glasses 301 via Bluetooth, so that the smart glasses 301 can provide real-time voice notifications or reminders to the user.

[0127] Optionally, the smart mobile terminal 302 is also used to respond to voice commands sent by the smart glasses 301 to perform operations such as making, answering or hanging up a call, and during a call, send the received call voice data to the smart glasses 301 so that the call voice data can be played through the speakers on the smart glasses 301.

[0128] The voice command is obtained by the voice data processor of the smart glasses 301 by performing voice command recognition processing on the voice data obtained by the sound pickup device of the smart glasses 301 using a preset voice recognition algorithm.

[0129] As will be appreciated, the smart glasses 301 of the present application are equipped with at least two directional and / or omnidirectional microphones. Users can use the microphones on the smart glasses 301 to capture voice data for issuing voice commands and answering calls. The user's voice is picked up by the microphones on the smart glasses 301 and, after noise reduction processing, is played back through the speakers of the smart glasses 301. Therefore, during a call, the user can place the smart mobile terminal 302 in their pocket or on a table, freeing their hands for other purposes, thereby improving the convenience of answering calls.

[0130] The cloud computing device 303 may be, for example, a cloud server or server cluster, configured to interact with the smart mobile terminal 302, store data sent by the smart mobile terminal 302 and / or the smart glasses 301 (e.g., all sensor data acquired by the smart glasses), and process the data based on pre-set processing logic. Specifically, the cloud computing device 303 receives first voice data or first text data corresponding to the first voice data sent by the smart mobile terminal 302, then translates the first voice data or first text data into second text data in a target language and returns the data to the smart mobile terminal 302. For example, assuming the source language is Chinese and the target language is English, if the smart mobile terminal 302 sends voice data 1 in Chinese, the voice data 1 is first converted into text data 1 in Chinese, then the text data 1 is translated into text data 2 in English and sent to the smart mobile terminal 302. If the smart mobile terminal 302 sends text data 1 corresponding to the voice data 1 in Chinese, the text data 1 is translated into text data 2 in English and sent to the smart mobile terminal 302.

[0131] The source language recognition operation can be performed by the smart mobile terminal 302 or by the cloud computing device 303. If the source language recognition operation is performed by the smart mobile terminal 302, then when the smart mobile terminal 302 transmits the first voice data, it also transmits the identification information of the source language and the target language to the cloud computing device 303, so that the cloud computing device 303 can determine the source language and the target language based on the identification information.

[0132] Optionally, the source language and the target language may also be determined by the smart mobile terminal 302 based on a selection operation by the user on the smart glasses 301 or the smart mobile terminal 302 .

[0133] Specifically, the user may also select the source language and / or target language by using control buttons or touch sensors on the smart glasses 301 or the smart mobile terminal 302. If the user performs the selection operation through the smart glasses 301, the smart glasses 301 will send identification information of the source language and / or target language selected by the user to the smart mobile terminal 302 at the same time as sending the first voice data to the smart mobile terminal 302.

[0134] In an actual application, when a user makes a selection operation through the smart glasses 301, the smart glasses 301 can also output corresponding voice prompts in real time according to the user's operation to help the user quickly select the source language and / or target language from multiple alternative languages.

[0135] Preferably, the Bluetooth protocol is used as the communication protocol between the smart mobile terminal 302 and the smart glasses 301, and the cellular mobile communication protocol (such as 2G, 3G, 4G, 5G protocol, etc.) is used as the communication protocol between the cloud computer device 303 and the smart mobile terminal 302.

[0136] It should be noted that the specific implementation process of the functions of the smart glasses 301, the smart mobile terminal 302 and the cloud computer device 303 in this embodiment can also refer to the relevant descriptions in other embodiments.

[0137] In this embodiment, real-time hearing assistance and translation capabilities are implemented in smart glasses, thereby expanding the functionality of the smart glasses and increasing product stickiness. Furthermore, since no additional hearing aids are required, the weight of the smart glasses is reduced, power consumption is lowered, and manufacturing costs are reduced. Furthermore, because a microphone on one side captures amplified voice data from a specific direction and a speaker on the other side plays the amplified voice data, echoes are reduced, improving the quality of the played voice data and the translated voice data, thereby increasing translation efficiency.

[0138] Referring to FIG6 , FIG6 is a schematic diagram illustrating a process flow for implementing a hearing aid method based on smart glasses according to an embodiment of the present application. The method can be applied to the smart glasses shown in FIG1 to FIG3 . As shown in FIG6 , the method includes the following steps:

[0139] S501: Acquire first voice data from a first preset direction through at least one microphone in the smart glasses;

[0140] S502: amplify the first voice data through a wireless communication module in the smart glasses to obtain amplified voice data;

[0141] S503: Play the amplified voice data through a target speaker in the smart glasses, wherein the target speaker and the at least one microphone are respectively arranged on different temples of the smart glasses.

[0142] In an embodiment, at least one microphone in the smart glasses is used to obtain first voice data from a first preset direction, and the first voice data is amplified by a wireless communication module in the smart glasses and the amplified first voice data is sent to a target speaker among the multiple speakers for playback, wherein the target speaker and the at least one microphone are respectively arranged on different temples, thereby realizing a real-time hearing aid function based on the smart glasses, thereby expanding the functions of the smart glasses and improving product stickiness. At the same time, since the microphone on one side obtains the voice data to be amplified from a specific direction and the speaker on the other side plays the amplified voice data, the echo can also be reduced and the quality of the played sound can be improved.

[0143] Optionally, in another embodiment of the present application, the smart glasses have an idle state and a working state that can be switched freely, wherein the smart glasses in the working state also include three working modes: hearing aid mode, call mode and translation mode. In call mode, the user can use the telephone call function of the smart glasses to have a telephone conversation with the person on the other end of the network through a mobile communication terminal and a wireless communication network. In hearing aid mode, the user can use the hearing aid function of the smart glasses to have a face-to-face conversation with other people. In translation mode, the user can use the translation function of the smart glasses for real-time translation, which can help the user better understand the voice content from users of different languages.

[0144] Furthermore, in another embodiment of the present application, the method further includes the following steps:

[0145] The wireless communication module responds to the first switching instruction to switch the working mode of the smart glasses to the hearing aid mode.

[0146] Step S502, amplifying the first voice data by the wireless communication module in the smart glasses to obtain amplified voice data, specifically includes:

[0147] In the hearing aid mode, the first voice data is amplified by the wireless communication module to obtain the amplified voice data.

[0148] Optionally, in another embodiment of the present application, a hearing aid function can be further automatically provided for a specific user, and the output of the second voice data includes: determining whether the currently logged-in user is the target user; if it is the target user, amplifying the second voice data through the wireless communication module, and outputting the amplified second voice data through the target speaker; if it is not the target user, outputting the second voice data through the target speaker.

[0149] Specifically, before using the various functions provided by the smart glasses, the user can register a corresponding user account through the smart mobile terminal. After logging into the operating system of the smart glasses using the user account through the smart mobile terminal, the user can set whether the user account is used by a specific user with hearing aid needs. If the user selects that the user account is used by a specific user with hearing aid needs, the smart mobile terminal will add a corresponding mark to the user account based on the user's setting operation.

[0150] After completing Bluetooth pairing with a smart mobile terminal, the smart glasses send a query request to the smart mobile terminal to confirm whether the currently logged-in user is the target user and save the query result locally. Before outputting the second voice data, the smart glasses use the locally saved query result to determine whether additional hearing aid functionality is required, namely, whether amplification of the translated voice data is required. In response to the query request, the smart mobile terminal confirms whether the currently logged-in user is the target user based on the aforementioned flag and returns the confirmation result to the smart glasses as the query result.

[0151] Furthermore, in another embodiment of the present application, the method further includes the following steps:

[0152] S601: Responding to a second switching instruction via the wireless communication module, switching the working mode of the smart glasses to a translation mode;

[0153] S602: In the translation mode, obtaining first voice data from a first preset direction through the at least one microphone;

[0154] S603, sending the first voice data to the smart mobile terminal through the wireless communication module, and receiving second voice data sent by the smart mobile terminal, wherein the second voice data is obtained by translating the first voice data;

[0155] S604: Play the second voice data through the target speaker.

[0156] Specifically, for real-time language translation, the smart glasses pick up a voice signal from a microphone, process it locally, and transmit it to the smart mobile terminal. After receiving the first voice data, the smart mobile terminal can use a local translation engine to translate the first voice data into the second voice data in the target language.

[0157] Alternatively, an application built into the smart mobile terminal uses a cloud-based AI (Artificial Intelligence) system to translate the voice signal from the source language to the target language in real time, and sends the translated voice signal back to the smart glasses and plays it for the user.

[0158] Specifically, the smart mobile terminal can choose to send the first voice data or the first text data corresponding to the first voice data to the cloud computer device for translation according to the preset conversion mark, receive the second text data in the target language returned by the cloud computer device, convert the second text data into the second voice data and send it to the smart glasses.

[0159] Optionally, the smart mobile terminal may also display the second text data through a local display while sending the second voice data to the smart glasses, thereby helping the user to understand the content of the other party's conversation more intuitively in the form of graphics and / or text.

[0160] Furthermore, in another embodiment of the present application, the method further includes the following steps:

[0161] S701: In response to a third switching instruction, the wireless communication module switches the working mode of the smart glasses to a call mode.

[0162] S702, in the call mode, acquiring third voice data from a second preset direction through the microphone, and sending the third voice data to the target terminal through the wireless communication module;

[0163] S703: Receive the fourth voice data sent by the target terminal through the wireless communication module, and play the fourth voice data through multiple speakers in the smart glasses.

[0164] The first preset direction points to the front of the user, and the second preset direction points to the bottom of the user, that is, the direction of the user's mouth.

[0165] In other words, as shown in Figure 7, in translation mode and hearing aid mode, the audio beam points forward, and the microphone captures speech signals from the front. In conversation mode, speech signals are captured from the direction of the user's mouth, that is, the microphone captures speech signals downward.

[0166] Furthermore, in translation and hearing-aid modes, only one temple speaker is used to output voice data, meaning one side picks up audio while the other plays it back, thus reducing echo. In call mode, however, all temple speakers are used to output voice data. This allows for optimal playback quality in different modes.

[0167] Alternatively, the user can trigger the switching command by utilizing an input device configured on the smart glasses, such as a physical button or a virtual button (touch sensor). For example, the smart glasses may be provided with multiple state control buttons corresponding to different operating states and working modes. The smart glasses, via the wireless communication module, respond to the switching command triggered by the user clicking a button, controlling the smart glasses to enter the operating state or working mode corresponding to the clicked button.

[0168] Optionally, based on the user's mode selection operation on the smart glasses or smart mobile terminal, either hearing aid mode or translation mode can be set as the commonly used listening mode. As shown in Figure 8, if the smart glasses are in an idle state, the aforementioned physical buttons or virtual buttons (touch sensors) can be used to manually switch between call mode and listening mode. For example, if the smart glasses are in call mode, pressing the physical button will change the smart glasses from call mode to listening mode, and pressing the physical button again will change it back to call mode. This can reduce the number of buttons and manufacturing costs, while also improving the intelligence of the operation and making it more in line with user usage habits.

[0169] Alternatively, when the user's target control operation is detected through the touch sensor, the wireless communication module responds to the switching instruction triggered by the target control operation to control the switching of the working mode of the smart glasses to the call mode or the listening mode, wherein the specific form of the target control operation can be customized by the user. For example, the action of long pressing the touch sensor or a preset sliding gesture can be set as the target control operation.

[0170] Furthermore, the DSP in the wireless communication module can execute various corresponding algorithms on the voice data.

[0171] As shown in Figure 9, in call mode, the smart glasses use the DSP to perform voice equalization and output volume processing on the downlink channel data, and perform echo cancellation, first beamforming, and noise suppression on the uplink channel data. Downlink channel data is input via the wireless signal transceiver in the wireless communication module. This wireless signal transceiver preferably uses Bluetooth as the communication protocol.

[0172] Furthermore, in the call mode, the smart glasses use the DSP to perform voice equalization processing and output volume control processing on the call voice data from the smart mobile terminal input through the wireless signal transceiver (such as the wireless Bluetooth input of the smart glasses in Figure 9), and send the call voice data after the output volume control processing to the speaker of the smart glasses for output. At the same time, the DSP uses the call voice data after the output volume processing as a reference signal to perform echo cancellation processing on the voice data obtained by the sound pickup device, and then performs beamforming processing and noise suppression processing, and outputs the data after the noise suppression processing to the smart mobile terminal through the wireless signal transceiver.

[0173] Because the voice signal from the remote party is played back through the speaker, it is received by the microphone, creating a loop. Echo cancellation processing on the uplink voice data uses an echo cancellation algorithm to compare the speaker output signal with the microphone input signal, eliminating the echo and interrupting the loop between the speaker and microphone.

[0174] Noise suppression is applied to uplink voice data using a noise suppression algorithm to reduce or eliminate noise while simultaneously amplifying the voice of the other party. This allows the user to hear the clear voice of the smart glasses even in a noisy environment without being able to hear the background noise.

[0175] The voice equalization processing performed on the voice data of the downlink channel is to use a voice equalizer to perform voice equalization processing on the distant voice signal to strengthen the frequency signal of the user's weak hearing, thereby achieving the purpose of compensating for the weak hearing frequency signal.

[0176] The output volume control processing of the voice data of the downlink channel is to adjust the output volume of the speaker using the output volume control algorithm.

[0177] Optionally, the wireless signal transceiver adopts at least one of the Bluetooth protocol, Wi-Fi protocol, near field communication protocol, ZigBee protocol, Digital Living Network Alliance protocol, carrier-free communication protocol, radio frequency identification protocol and cellular mobile communication protocol as a communication protocol for data interaction with the smart mobile terminal.

[0178] Furthermore, in other embodiments of the present application, before obtaining the first voice data or the third voice data through the sound pickup device, the method further includes:

[0179] The wireless communication module is paired with the smart mobile terminal based on the Bluetooth protocol to establish a data transmission channel between the smart glasses and the smart mobile terminal. Subsequent data exchange between the smart glasses and the smart mobile terminal can be carried out through this data transmission channel.

[0180] As shown in Figure 10, in hearing aid mode, in order to provide hearing loss assistance, the smart glasses use the DSP to perform feedback cancellation processing, voice activity detection processing, noise suppression processing, voice equalization processing, and user speech detection processing on the voice data, and send the voice data after voice equalization processing to the speakers of the smart glasses for output. At the same time, the voice data after voice equalization processing is used as reference data for the feedback cancellation processing. Furthermore, before the voice data after voice equalization processing is sent to the speakers of the smart glasses for output, the voice data after voice equalization processing can also be processed for output gain and output volume control. The specific processing method is the same as the output volume control processing method shown in Figure 9. Please refer to the relevant description of Figure 9 above for details, and will not be repeated here.

[0181] In hearing aid mode, the sound waves output by the speaker are instantly received by the microphone array and form a loop, resulting in a piercing howl. Feedback cancellation utilizes a feedback cancellation algorithm to eliminate the echo by comparing the speaker's output signal with the microphone array's input signal, thereby interrupting the loop between the speaker and microphone array. Specifically, as shown in Figure 11, the microphone captures the speech signal from the front, but because typical microphones are omnidirectional, the speaker's speech signal is also captured by the microphone. The speaker's speech signal is also fed back to the input of the adaptive filter via internal wiring. The adaptive filter adjusts the LMS (least mean square) adaptive filter coefficients based on the previous speaker output data, thereby removing the speaker's output signal (i.e., f(n)) from the microphone input.

[0182] Furthermore, in hearing-aided mode, the voice of the other party needs to be amplified to enable hearing-impaired users to clearly hear what is being said. However, this amplification process also amplifies both speech and noise, causing discomfort to the user. Noise suppression uses a noise suppression algorithm to reduce or eliminate the noise volume while simultaneously amplifying the volume of the other party's speech. Specifically, the algorithm divides each frame's speech signal into speech and silence. If a frame is detected as a silence frame, the algorithm uses the speech signal from that frame to update the average noise energy value. If a frame is defined as a speech frame, the average noise energy value is subtracted from the speech signal from that frame, achieving noise suppression.

[0183] Furthermore, in hearing-aid mode, hearing-impaired users may simply be unable to hear or be unable to clearly hear sounds of a certain frequency. Voice equalization uses a voice equalizer to enhance the sound signal of that specific frequency, thereby compensating for the sound signal at that specific frequency.

[0184] Furthermore, because the microphone array of the smart glasses is very close to the user's mouth, when the user speaks, the microphone array receives a strong signal and plays it through the smart glasses' speakers. This allows the user to hear their own voice through the speakers when they speak. User voice detection uses a user voice detection algorithm to continuously monitor the signal received by the microphone array and perform average energy analysis and detection. If the average energy estimate falls below a preset average energy threshold, the signal is detected as the user's voice.

[0185] The voice activity detection process is similar to the user voice detection process, except that the average energy threshold is lower than that of the user voice detection algorithm, thereby detecting the voice of the other party speaking.

[0186] Optionally, in another embodiment of the present application, an independent playback device (such as an external speaker, an external hearing aid) can also be used to play the voice data. The above-mentioned step of outputting the amplified voice data or the second voice data may specifically include: determining whether there is a playback device that has established a data connection with the smart glasses through the wireless communication module; if so, sending the amplified voice data or the second voice data to the playback device through the wireless communication module so that the amplified voice data or the second voice data is played through the playback device; if not, outputting the amplified voice data or the second voice data through the target speaker. In this way, when there is an independent playback device, using the independent playback device to play the voice data can reduce the power consumption of the smart glasses. Among them, the wireless communication module of the smart glasses can determine whether there is a playback device that has established a data connection with the smart glasses based on the local Bluetooth pairing log and the connection status of the peripheral interface.

[0187] In the several embodiments provided herein, it should be understood that the disclosed smart glasses, assistive systems, and hearing assistance methods can be implemented in other ways. For example, multiple modules or components can be combined or integrated into another system, or some features can be omitted or not implemented. Furthermore, the connections or direct or communicative connections shown or discussed can be indirect or communicative connections through some interface, device, or module, and can be electrical, mechanical, or other forms.

[0188] It should be noted that for the aforementioned method embodiments, for ease of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0189] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0190] The above is a description of the smart glasses, hearing aid method, and auxiliary system provided in this application. For those skilled in the art, based on the ideas of the embodiments of this application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A pair of smart glasses for hearing aid, characterized in that: include: Two temples, a sound pickup device, multiple speakers, and a wireless communication module; The multiple speakers and the sound pickup device are arranged on at least one of the two temples, the wireless communication module is arranged in the cavity of any one of the two temples and is electrically connected to the sound pickup device and the multiple speakers, and the sound pickup device includes at least one microphone; The at least one microphone is used to obtain first voice data from a first preset direction; The wireless communication module is used to amplify the first voice data to obtain amplified voice data, and send the amplified voice data to a target speaker among the multiple speakers for playback, wherein the target speaker and the at least one microphone are respectively arranged on different temples.

2. The smart glasses according to claim 1, wherein: Also includes: A spectacles frame, and an input device electrically connected to the wireless communication module, wherein the input device is arranged on the spectacles frame and at least one of the two spectacles legs; The wireless communication module is further used to switch the working mode of the smart glasses to the hearing aid mode in response to a first switching instruction triggered by the user based on the input device; The wireless communication module is further configured to amplify the first voice data in the hearing aid mode.

3. The smart glasses according to claim 2, wherein: The wireless communication module is further used to switch the working mode of the smart glasses to a translation mode in response to a second switching instruction triggered by the user based on the input device; The wireless communication module is also used to send the first voice data to the smart mobile terminal in the translation mode, receive the second voice data sent by the smart mobile terminal, and send the second voice data to the target speaker for playback, where the second voice data is translated from the first voice data.

4. The smart glasses according to claim 2, wherein: The wireless communication module is further configured to switch the working mode of the smart glasses to a call mode in response to a third switching instruction triggered by the user based on the input device; The at least one microphone is further used to obtain third voice data from a second preset direction in the call mode; The wireless communication module is further used to send the third voice data to a target terminal and receive fourth voice data sent by the target terminal; The multiple speakers are also used to play the fourth voice data.

5. The smart glasses according to claim 4, wherein: The first preset direction points to the front of the user, and the second preset direction points to the bottom of the user.

6. A hearing aid method based on smart glasses, characterized in that: The smart glasses include: two temples, a wireless communication module, and a sound pickup device and a plurality of speakers electrically connected to the wireless communication module, wherein the sound pickup device includes at least one microphone, and the method includes: Acquiring first voice data from a first preset direction through the at least one microphone; amplifying the first voice data through the wireless communication module to obtain amplified voice data; The amplified voice data is played through a target speaker among the multiple speakers, wherein the target speaker and the at least one microphone are respectively arranged on different temples.

7. The method according to claim 6, characterized in that The method further comprises: In response to a first switching instruction, the wireless communication module switches the working mode of the smart glasses to a hearing aid mode; The step of amplifying the first voice data by the wireless communication module to obtain amplified voice data includes: In the hearing aid mode, the first voice data is amplified by the wireless communication module to obtain amplified voice data.

8. The method according to claim 6, characterized in that The method further comprises: In response to a second switching instruction, the wireless communication module switches the working mode of the smart glasses to a translation mode; In the translation mode, acquiring the first voice data through the at least one microphone; Sending the first voice data to the smart mobile terminal through the wireless communication module, and receiving the second voice data sent by the smart mobile terminal, wherein the second voice data is obtained by translating the first voice data; The second voice data is played through the target speaker.

9. The method according to claim 8, characterized in that The smart mobile terminal receives the first voice data sent by the smart glasses, and according to a preset conversion mark, selects to send the first voice data or the first text data corresponding to the first voice data to a cloud computer device for translation, receives the second text data in the target language returned by the cloud computer device, converts the second text data into the second voice data and sends it to the smart glasses.

10. The method according to claim 6, characterized in that The method further comprises: In response to a third switching instruction, the wireless communication module switches the working mode of the smart glasses to a call mode; In the call mode, acquiring third voice data from a second preset direction through the at least one microphone; Sending the third voice data to a target terminal through the wireless communication module; The fourth voice data sent by the target terminal is received through the wireless communication module, and the fourth voice data is played through the multiple speakers.

11. The method according to claim 10, wherein: The first preset direction points to the front of the user, and the second preset direction points to the bottom of the user.

12. An auxiliary system, characterized in that: include: Smart glasses and smart mobile terminals; The smart glasses include: two temples, a sound pickup device, a plurality of speakers, and a wireless communication module, wherein the plurality of speakers and the sound pickup device are arranged on at least one of the two temples, the wireless communication module is arranged in a cavity of any one of the two temples and is electrically connected to the sound pickup device and the plurality of speakers, and the sound pickup device includes at least one microphone; The wireless communication module is used to control the switching of the working modes of the smart glasses in response to the control operation of the user, and the working modes include: a hearing aid mode and a translation mode; The at least one microphone is used to obtain first voice data from a first preset direction in the hearing aid mode or the translation mode; The wireless communication module is used for, in the hearing aid mode, amplifying the first voice data to obtain amplified voice data, and sending the amplified voice data to a target speaker among the multiple speakers for playback, wherein the target speaker and the at least one microphone are respectively arranged on different temples; The wireless communication module is further used to send the first voice data to the smart mobile terminal in the translation mode; The intelligent mobile terminal is used to receive the first voice data and send second voice data to the wireless communication module, wherein the second voice data is obtained by translating the first voice data; The wireless communication module is further used to receive second voice data sent by the smart mobile terminal, and send the second voice data to the target speaker for playing.

13. The system according to claim 12, wherein: The system also includes a cloud computing device; The smart mobile terminal is further used to send the first voice data or the first text data corresponding to the first voice data as data to be translated to the cloud computer device; The cloud computer device is used to translate the data to be translated into second text data in a target language, and send the second text data to the smart mobile terminal; The smart mobile terminal is further used to convert the second text data into the second voice data and send the second voice data to the smart glasses.

14. The system according to claim 13, wherein: The smart mobile terminal is also used to obtain a preset conversion mark, and if the preset conversion mark is a first mark, convert the first voice data into the first text data and send the first text data to the cloud computer device, and if the preset conversion mark is a second mark, send the first voice data to the cloud computer device.

15. The system according to claim 14, wherein: The intelligent mobile terminal is also used to obtain the network speed and determine whether the network speed is greater than a preset speed. If it is not greater than the preset speed, the conversion mark is set to the first mark; if it is greater than the preset speed, the conversion mark is set to the second mark.

16. The system of claim 14, wherein: The intelligent mobile terminal is further configured to, in response to a setting operation of a user, set the conversion mark to the mark pointed to by the setting operation.

17. The system of claim 12, wherein: The working mode also includes a conversation mode; The at least one microphone is further used to obtain third voice data from a second preset direction in the call mode; The wireless communication module is further used to send the third voice data to a target terminal and receive fourth voice data sent by the target terminal; The multiple speakers are also used to play the fourth voice data.

Images