Wearable sound pickup device and sound pickup method

Abstract

The application provides a wearable sound pickup device and a sound pickup method. The wearable sound pickup device comprises a wearable sound pickup device body, the wearable sound pickup device body comprising an extension component; a sound pickup array mounted in the wearable sound pickup device body and used for long-distance sound pickup, the sound pickup array comprising at least two sound pickups, and the arrangement direction of each sound pickup being consistent with the extension direction of the extension component; and a processor connected with the sound pickup array. The wearable sound pickup device can collect the sound of a wearer, the sound of the surrounding environment and / or the sound emitted by a specified target through the sound pickup array, and the collected sound can be processed and output by the processor connected with the sound pickup array, so that the wearable sound pickup device can obtain a sound pickup capability beyond the hearing distance of human ears through the sound pickup array, the wearer can conveniently hear the sound in the distance or collect the audio signal in the distance, and the user experience is improved.

Description

[Technical Field]

[0001] This application relates to the field of sound pickup, and more specifically, to a wearable sound pickup device and a sound pickup method. [Background Technology]

[0002] In normal scenarios, the volume of normal conversation between people is about 40 decibels, and the hearing distance of the human ear is about 5 meters. However, existing smart glasses and hearing aids typically have 2 to 3 microphones, which are mainly designed to pick up sounds from the wearer or nearby, and it is difficult to provide the ability to pick up sounds beyond the hearing distance of the human ear.

[0003] Therefore, how to provide a wearable sound pickup device capable of long-distance sound pickup is a technical problem that needs to be solved by those skilled in the art. [Summary of the Invention]

[0004] To address the problem that existing wearable sound pickup devices cannot provide sound pickup capabilities beyond the human ear's hearing range, this application provides a wearable sound pickup device and a sound pickup method.

[0005] In a first aspect, this technical solution provides a wearable audio pickup device, comprising: a wearable audio pickup device body, the wearable audio pickup device body including an extension component; a microphone array installed inside the wearable audio pickup device body for long-distance audio pickup, the microphone array including at least two microphones, the arrangement direction of each microphone in the microphone array being consistent with the extension direction of the extension component; and a processor connected to the microphone array. In some embodiments, the wearable audio pickup device provided in this application can collect the voice of the wearer, ambient sound, and / or sound emitted by a designated target through the microphone array installed inside the wearable audio pickup device body, and process and output the collected sound through the processor connected to the microphone array, so as to realize the ability of the wearable audio pickup device to obtain sound pickup beyond the hearing distance of the human ear through the microphone array, making it convenient for the wearer to hear distant sounds or collect distant audio signals, thus improving the user experience.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, the wearable audio pickup device is specifically a smart pair of glasses, and the extension component is the temple of the smart glasses; for each microphone in the microphone array, the temple is provided with a pickup channel matching the microphone, and the pickup channel has at least one sound inlet hole. In some embodiments, the pickup channel in this application is the channel through which the microphone picks up sound; external sounds enter the pickup channel through the sound inlet hole and are picked up by the microphone. The function of the pickup channel is to change the pickup direction of the microphone and enhance the pickup effect.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the microphone includes two back-to-back microphone ports, and the microphone channel has two inlet holes located on the outer surface of the temple of the smart glasses. Each inlet hole corresponds to one microphone port, and the microphone channel connects the corresponding inlet hole and microphone port. The straight line containing the two inlet holes is aligned with the straight line containing the temple of the smart glasses. In some embodiments, when a user wears the smart glasses, sounds from above and below the user will cancel each other out because the distances to the two inlet holes are the same. Consequently, the microphone cannot pick up the sound, allowing it to block sounds from above and below the user. Sounds from directions other than those in front and behind the user will also be attenuated because their components above and below the user are canceled out. When a user wears smart glasses, the distance difference between the sounds coming from in front of and behind the user is greatest due to the two sound inlets. At this distance, the interference between the sounds entering the pickup channel from the two inlets is minimized, allowing the microphone to pick up the least amount of interfered sound. This results in the best sound pickup performance when capturing sounds from both directions. By designing two sound inlets for the microphone's pickup channel, the microphone gains directional sound pickup capabilities.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the wearable audio pickup device further includes a microphone connected to the processor and installed inside the nose bridge connector of the smart glasses. In some embodiments, the microphone installed inside the nose bridge connector of the smart glasses is specifically designed to pick up the wearer's voice to perform audio enhancement processing on the wearer's voice during calls or recordings.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the microphone installed inside the nose bridge connector of the smart glasses has two sound inlets located on the outer surface of the nose bridge connector of the smart glasses, with the direction of the straight line of the two sound inlets pointing vertically downwards. In some embodiments, when the user wears the smart glasses, sounds from the front and back of the user will cancel each other out because the distances to the two sound inlets are the same, thus preventing the microphone from picking up the sound. This allows the microphone to block sounds from the front and back of the user. Sounds from directions other than the front and back of the user will also be weakened because their components in front of and behind the user are canceled out. When the user wears the smart glasses, sounds from above and below the user will have the greatest distance difference to the two sound inlets. At this time, the interference between the sounds entering the microphone from the two sound inlets one above and one below is minimal, allowing the microphone to pick up the least amount of interfered sound. This enables the microphone to achieve the best sound pickup effect when picking up sounds from above and below the user. By setting two sound inlets in the microphone's pickup channel, the microphone can pick up the user's voice in a directional manner.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the microphone array is mounted inside the temple of the smart glasses. In some embodiments, by mounting the microphone array inside the temple of the smart glasses, directional sound pickup capability is provided while avoiding the discomfort and impact on the wearer's appearance caused by mounting it on the outside of the smart glasses.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the wearable audio pickup device is specifically a wireless earphone, and the extension component is the ear stem of the wireless earphone; the microphone array is installed inside the ear stem of the wireless earphone. In some embodiments, by installing the microphone array inside the ear stem of the wireless earphone, it is possible to provide audio pickup capabilities that exceed the human hearing distance while avoiding the discomfort and impact on the wearer's appearance caused by installing it on the outside of the wireless earphone.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, for each microphone in the microphone array, the earpiece is provided with a pickup channel matching the microphone, and the pickup channel has at least one sound inlet hole. In some embodiments, the pickup channel is the channel through which the microphone picks up sound; external sounds enter the pickup channel through the sound inlet hole and are picked up by the microphone. The function of the pickup channel is to change the pickup direction of the microphone and enhance the pickup effect.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the pickup channel has two sound inlets located on the outer surface of the ear stem of the wireless earphone, and the straight line of the two sound inlets is aligned with the direction of the straight line of the ear stem of the wireless earphone. In some embodiments, when a user wears the wireless earphone, sounds from the front and rear of the user will cancel each other out because the distances to the two sound inlets are the same. Therefore, the microphone cannot pick up the sound, allowing it to block sounds from the front and rear of the user. Sounds from directions other than the user's vertical direction will also be attenuated because their components in front of and behind the user are canceled out. Sounds from above and below the user will have the greatest distance difference to the two sound inlets, resulting in minimal interference between the sounds entering the pickup channel one after the other. This allows the microphone to pick up the least amount of interfered sound, enabling the wireless earphone to pick up the wearer's voice more clearly and accurately.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, each of the microphones in the microphone array is specifically a directional microphone. In some embodiments, a directional microphone refers to a microphone with high sensitivity to a specific angle, which can pick up sound from a specific direction.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the wearable audio pickup device further includes a speaker connected to the processor. In some embodiments, audio signals picked up by the microphone array can be output in real time through the speaker to achieve a directional enhancement of sound for hearing aids.

[0016] Secondly, this technical solution provides a sound pickup method applied to a wearable sound pickup device as described in any of the above claims. The sound pickup method includes: in response to an input sound pickup command, acquiring audio signals in the environment through a microphone array, the audio signals including a first audio signal and a second audio signal, the direction of the first audio signal being consistent with the extension direction of the extension component, and the direction of the second audio signal being inconsistent with the extension direction of the extension component; amplifying the first audio signal and suppressing the second audio signal through a processor to obtain a processed audio signal, and outputting the processed audio signal. In some embodiments, this application first uses a microphone array to acquire a first audio signal consistent with the extension direction of the extension component and a second audio signal inconsistent with the extension direction of the extension component in the environment. Then, the processor amplifies the first audio signal and suppresses the second audio signal, making the first audio signal in a specific direction stronger and the second audio signal in other directions weaker in the output audio signal, thereby achieving directional sound pickup in the environment.

[0017] In conjunction with the second aspect, in some implementations of the second aspect, the step of outputting the processed audio signal includes: parsing the pickup instruction by the processor to obtain a pickup task category, wherein the pickup task category includes a hearing aid task; if the pickup task category is a hearing aid task, then the processed audio signal is output to a speaker. In some embodiments, the pickup task category of the pickup instruction may include a hearing aid task, that is, in this case, the wearable pickup device can be regarded as a far-field hearing aid, capable of picking up and amplifying sounds beyond the human hearing distance so that the wearer can clearly hear distant sounds. When the pickup task category of the pickup instruction is a hearing aid task, the processed audio signal can be output to a speaker.

[0018] In conjunction with the second aspect, in some implementations of the second aspect, the sound pickup task category includes a sound pickup auxiliary task, and the method further includes: if the sound pickup task category is a sound pickup auxiliary task, then the processor parses the sound pickup instruction to obtain the target device; and the processed audio signal is output to the target device through a wireless communication module. In some embodiments, the sound pickup task category of the sound pickup instruction may also include a sound pickup auxiliary task. When the sound pickup task category of the sound pickup instruction is a sound pickup auxiliary task, the processed audio signal can be output to the target device. That is, at this time, the wearable sound pickup device can be regarded as a long-range auxiliary sound pickup, which can assist video recording devices such as mobile phones and cameras in sound pickup, thereby improving the sound pickup effect. [Attached Image Description]

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of a smart glasses according to this application;

[0021] Figure 2 This is a schematic diagram of a hearing aid according to this application;

[0022] Figure 3 This is a schematic diagram of the structure of a wearable audio pickup device 100 provided in an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of the structure of another wearable audio pickup device 100 provided in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the structure of a smart glasses provided in an embodiment of this application;

[0025] Figure 6 for Figure 5 A cross-sectional view of the internal structure of the temple of a smart pair of glasses is provided.

[0026] Figure 7 for Figure 5 A cross-sectional view of the temples of a smart glasses provided in the first section;

[0027] Figure 8 for Figure 5 A diagram showing the internal structure of the temple of a pair of smart glasses in a second cross-section;

[0028] Figure 9 for Figure 5 A cross-sectional view of the temples of a type of smart glasses in the second section;

[0029] Figure 10 A vertical cross-sectional view of a nose bridge connector in smart glasses provided in an embodiment of this application;

[0030] Figure 11 This is a schematic diagram of the structure of a wireless earphone provided in an embodiment of this application;

[0031] Figure 12 This is a schematic diagram illustrating long-distance sound pickup using the wireless earphones provided in the embodiments of this application;

[0032] Figure 13 A flowchart illustrating a sound pickup method provided in an embodiment of this application;

[0033] Figure 14 This is a schematic diagram of an audio signal processing method provided in an embodiment of this application;

[0034] Figure 15 This is a schematic diagram illustrating the process of using a wearable audio pickup device for audio pickup assistance, as provided in an embodiment of this application.

[0035] Figure 16 This is a schematic diagram illustrating a scenario where smart glasses are selected for sound pickup assistance, as provided in an embodiment of this application.

[0036] Figure 17 This is a schematic diagram illustrating a scenario where smart glasses are used for sound pickup assistance, as provided in an embodiment of this application.

[0037] Figure 18 This is a schematic diagram illustrating a scenario where hearing aids are provided through smart glasses, as provided in an embodiment of this application.

Detailed Implementation Methods

[0038] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0039] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0040] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0041] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0042] In normal scenarios, the volume of normal conversation between people is about 40 decibels, and the hearing distance of the human ear is about 5 meters. However, existing smart glasses and hearing aids typically have 2 to 3 microphones, which are mainly designed to pick up sounds from the wearer or nearby, and it is difficult to provide the ability to pick up sounds beyond the hearing distance of the human ear.

[0043] For example, please refer to Figure 1 This is a schematic diagram of a smart glasses according to this application. Two omnidirectional microphones are arranged on each temple of the glasses to enable sound pickup, allowing them to be used as hearing aids or headphones. However, these smart glasses primarily function as hearing aids, possessing the ability to pick up ambient and forward sounds, but with a limited pickup range. The hearing aid capabilities of typical hearing aids and glasses are roughly similar to those of a person with normal hearing, and they cannot provide hearing assistance beyond the range of normal hearing.

[0044] For example, please refer to Figure 2 The image shown is a schematic diagram of a hearing aid according to this application. Existing hearing aids are generally designed as single-sided single-pickup or single-sided dual-pickup, with the hearing aid hanging on the ear and the sound tube inserted into the ear canal. However, the pickup range of such hearing aids is limited, generally within 2m to 3m, and this type of external hearing aid needs to be inserted into the ear canal, which is uncomfortable to wear and greatly affects the user experience.

[0045] Therefore, this application provides a wearable sound pickup device to solve the above problems.

[0046] Please refer to Figure 3 This is a schematic diagram of the structure of a wearable audio pickup device 100 provided in an embodiment of this application.

[0047] like Figure 3 As shown, the wearable microphone 100 includes: a wearable microphone body 110, the wearable microphone body 110 including an extension component; a microphone array 120 installed inside the wearable microphone body 110 for long-distance microphone pickup, the microphone array 120 including at least two microphones 121 and 122, the arrangement direction of each microphone in the microphone array 120 being consistent with the extension direction of the extension component; and a processor 130 connected to the microphone array 102.

[0048] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the wearable microphone 100. In other embodiments of this application, the wearable microphone 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0049] The wearable audio pickup device 110 may include, but is not limited to, smart glasses, wireless headphones, VR (Virtual Reality) glasses, AR (Augmented Reality) glasses, etc. It can collect the voice of the wearer, ambient sound and / or the sound emitted by a designated target through the microphone array 120 installed inside the wearable audio pickup device 110, and process the collected sound through the processor 130 connected to the microphone array 120.

[0050] The microphone array 120 is installed inside the wearable microphone device body 110. By aligning the arrangement direction of each microphone in the array with the extension direction of the extension component, the microphone array 120 can achieve a sound pickup capability beyond the human ear's hearing distance. The microphone array 120 may include two microphones or multiple microphones, the number of which can be determined based on the desired sound pickup effect and the internal volume of the wearable microphone device body 110.

[0051] The pickup array 120 can be composed entirely of omnidirectional pickups, or entirely of directional pickups, or a mixture of omnidirectional and directional pickups. The specific composition of the pickup array 120 can be determined according to the desired pickup effect and the desired pickup direction.

[0052] The processor 130 processes the acquired sound.

[0053] In some embodiments, please refer to Figure 4 This is a schematic diagram of another wearable audio pickup device 100 provided in the embodiments of this application.

[0054] like Figure 4 As shown, the wearable microphone device 100 may include a processor 130, a sensor module 140 (e.g., which can be used to acquire the user's posture), a microphone array 120, a button 150, an input / output interface 160, a communication module 170, a camera 180, a battery 190, an optical display module 1100, an eye-tracking module 1200, a memory 1300, etc.

[0055] Processor 130 is typically used to control the overall operation of wearable audio pickup device 100 and may include one or more processing units. For example, processor 130 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a video processing unit (VPU) controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0056] The processor 130 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 130 is a cache memory. This memory can store instructions or data that the processor 130 has just used or that are used repeatedly. If the processor 130 needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces the waiting time of the processor 130, and thus improves the efficiency of the system.

[0057] In some embodiments of this specification, the processor 130 can be used to control the optical power of the wearable audio pickup device 100. For example, the processor 130 can be used to control the optical power of the optical display module 1100, thereby adjusting the optical power of the head-mounted display device 100. For instance, the processor 130 can adjust the relative positions of the various optical components (such as lenses) in the optical display module 1100, thereby adjusting the optical power of the optical display module 1100, and consequently adjusting the position of the virtual image plane when the optical display module 1100 images onto the human eye. This achieves the effect of controlling the optical power of the head-mounted display device 100.

[0058] In some embodiments, the processor 130 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, a serial peripheral interface (SPI) interface, etc.

[0059] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 130 may include multiple I2C buses.

[0060] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 130 and the communication module 170. For example, the processor 130 communicates with the Bluetooth module in the communication module 170 via the UART interface to implement Bluetooth functionality.

[0061] The MIPI interface can be used to connect the processor 130 to peripheral devices such as the display device and camera 180 in the optical display module 1100.

[0062] The GPIO interface is configurable via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 130 to the camera 180, the display device in the optical display module 1100, the communication module 170, the sensor module 140, the microphone array 120, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc. In some embodiments, the camera 180 can capture images including real objects, and the processor 130 can fuse the captured images with virtual objects, displaying the fused image through the optical display module 1100. In some embodiments, the camera 180 can also capture images including human eyes. The processor 130 uses these images for eye tracking.

[0063] The USB interface conforms to the USB standard specification and can be a Mini USB interface, Micro USB interface, USB Type-C interface, etc. The USB interface can be used to connect a charger to charge the wearable audio pickup device 100, and can also be used for data transfer between the wearable audio pickup device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as mobile phones. The USB interface can be USB 3.0, used for compatibility with high-speed display port (DP) signal transmission, enabling the transmission of high-speed audio and video data.

[0064] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the head-mounted display device 100. In other embodiments of this specification, the head-mounted display device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0065] Additionally, the wearable microphone 100 may include wireless communication functionality. For example, the wearable microphone 100 may receive and display images from other electronic devices (such as VR headsets), or it may directly acquire data from base stations or other sites. The communication module 170 may include a wireless communication module and a mobile communication module. The wireless communication functionality may be implemented using an antenna (not shown), a mobile communication module (not shown), a modem processor (not shown), and a baseband processor (not shown). The antenna is used to transmit and receive electromagnetic wave signals. The wearable microphone 100 may include multiple antennas, each of which can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in conjunction with a tuning switch.

[0066] The mobile communication module can provide solutions for wireless communication applications in wearable audio pickup devices 100, including 2G, 3G, 4G, 5G, and 6G networks. The mobile communication module may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module can receive electromagnetic waves via an antenna, filter and amplify the received electromagnetic waves, and transmit them to a modem processor for demodulation. The mobile communication module can also amplify the signal modulated by the modem processor and radiate it as electromagnetic waves via the antenna. In some embodiments, at least some functional modules of the mobile communication module may be housed in the processor 130. In some embodiments, at least some functional modules of the mobile communication module and at least some modules of the processor 130 may be housed in the same device.

[0067] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to a speaker), or displays images or videos through a display device in the optical display module 1100. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 130 and may be housed in the same device as the mobile communication module or other functional modules.

[0068] The wireless communication module can provide solutions for wireless communication applications in the wearable audio pickup device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module can be one or more devices integrating at least one communication processing module. The wireless communication module receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signal, and sends the processed signal to the processor 130. The wireless communication module can also receive signals to be transmitted from the processor 130, modulate and amplify them, and then convert them into electromagnetic waves for radiation via the antenna.

[0069] In some embodiments, the antenna of the wearable microphone 100 is coupled to the mobile communication module, enabling the wearable microphone 100 to communicate with networks and other devices via wireless communication technologies. These wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), 5G, 6G, BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. GNSS can include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0070] The wearable audio pickup device 100 implements display functions through a GPU, an optical display module 1100, and an application processor. The GPU is a microprocessor for image processing, connecting the optical display module 1100 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 130 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0071] The memory 1300 can be used to store computer executable program code, which includes instructions. The processor 130 executes various functional applications and data processing of the wearable audio pickup device 100 by running the instructions stored in the memory 1300. The memory 1300 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the head-mounted display device 100 (such as audio data, phonebook, etc.). Furthermore, the memory 1300 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0072] The wearable audio pickup device 100 can implement audio functions through an audio module, speaker, microphone array 120, headphone jack, and application processor. Examples include music playback and recording. The audio module is used to convert digital audio information into analog audio signal output and to convert analog audio input into digital audio signals. The audio module can also be used for encoding and decoding audio signals. In some embodiments, the audio module can be located in the processor 130, or some functional modules of the audio module can be located in the processor 130. The speaker, also called a "loudspeaker," is used to convert audio electrical signals into sound signals. The head-mounted display device 100 can listen to music or make hands-free calls through the speaker.

[0073] A microphone array 120 is used to convert sound signals into electrical signals. The wearable microphone device 100 may have at least one microphone array 120. In some embodiments, the wearable microphone device 100 may have two microphone arrays 120, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, the wearable microphone device 100 may also have three, four, or more microphone arrays 120, which can collect sound signals, reduce noise, identify sound sources, and perform directional recording, etc.

[0074] The headphone jack is used to connect wired headphones. The headphone jack can be a USB interface or a 3.5 mm Open Mobile Terminal Platform (OMTP) standard interface, or a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0075] In some embodiments, the wearable microphone 100 may include one or more buttons 150 that can control the wearable microphone and provide users with the ability to interact with the wearable microphone 100. The buttons 150 may take the form of buttons, switches, dials, and touch or proximity sensing devices (such as touch sensors). Specifically, for example, a user can turn on the optical display module 1100 of the wearable microphone 100 by pressing a button. Buttons 150 may include a power button, volume buttons, etc. Buttons 150 may be mechanical buttons or touch-sensitive buttons. The head-mounted display 100 can receive button input and generate key signal inputs related to user settings and function control of the head-mounted display 100.

[0076] In some embodiments, the wearable audio pickup device 100 may include an input / output interface 160, which can connect other devices to the wearable audio pickup device 100 via suitable components. Components may include, for example, audio / video jacks, data connectors, etc.

[0077] The optical display module 1100, under the control of the processor 130, presents images to the user. The optical display module 1100 can use one or more optical devices, such as mirrors, transmissive mirrors, or optical waveguides, to convert real-pixel images into near-eye projection virtual images, enabling virtual interactive experiences or a combination of virtual and real interactive experiences. For example, the optical display module 1100 receives image data information sent by the processor 130 and presents the corresponding image to the user.

[0078] For example, the optical display module 1100 may include two display devices, namely display device 110 and display device 120. Alternatively, the optical display module 1100 may include display module 1 and display module 2, where display module 1 includes display device 110 and optical device 130, and display module 2 includes display device 120 and optical device 140.

[0079] In some embodiments, the wearable audio pickup device 100 may further include an eye-tracking module 1200, which tracks the movement of the human eye to determine the gaze point. For example, image processing technology can be used to locate the pupil position, obtain the pupil center coordinates, and then calculate the person's gaze point. In some embodiments, the eye-tracking system can determine the user's gaze point position (or determine the user's gaze direction) using methods such as video eye diagrams, photodiode response methods, or pupil-corneal reflection methods, thereby achieving eye tracking.

[0080] In some embodiments, the user's gaze direction is determined using the pupillary corneal reflex method. The eye-tracking system may include one or more near-infrared light-emitting diodes (LEDs) and one or more near-infrared cameras. The near-infrared LEDs and near-infrared cameras are not... Figure 4 As shown in the diagram. In different examples, the near-infrared LED can be positioned around the optics to provide comprehensive illumination of the human eye. In some embodiments, the center wavelength of the near-infrared LED can be 850 nm or 940 nm. The eye-tracking system can obtain the user's gaze direction by illuminating the human eye with a near-infrared LED, capturing an image of the eyeball with a near-infrared camera, and then determining the optical axis direction of the eyeball based on the position of the reflective point of the near-infrared LED on the cornea and the center of the pupil in the eyeball image, thereby obtaining the user's gaze direction.

[0081] It should be noted that in some embodiments of this specification, separate eye-tracking systems can be set up for each of the user's eyes to perform eye tracking synchronously or asynchronously. In other embodiments of this specification, an eye-tracking system can be set up only near one of the user's eyes. The eye-tracking system obtains the gaze direction of the corresponding eye, and based on the relationship between the fixation points of the two eyes (e.g., when a user observes an object through both eyes, the fixation points of the two eyes are generally close or the same), combined with the user's interocular distance, the gaze direction or fixation point position of the user's other eye can be determined.

[0082] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the wearable microphone 100. In other embodiments of this specification, the wearable microphone 100 may include... Figure 4 The embodiments of this application do not limit the number of components, the combination of certain components, the separation of certain components, or the different arrangement of components.

[0083] In some embodiments, when the wearable microphone 100 is a pair of smart glasses, the extension component is the temple of the smart glasses. For each microphone in the microphone array 120, the temple is provided with a microphone pickup channel that matches the microphone. The microphone pickup channel has at least one sound inlet hole. The microphone pickup channel is the channel through which the microphone picks up sound; external sounds enter the microphone pickup channel through the sound inlet hole and are picked up by the microphone. The function of the microphone pickup channel is to change the pickup direction of the microphone and enhance the pickup effect.

[0084] In some embodiments, the microphone array may consist entirely of omnidirectional microphones, entirely of directional microphones, or a mixture of both. The specific composition of the microphone array can be determined based on the desired sound pickup effect and desired pickup direction. An omnidirectional microphone is one that has the same sensitivity to all angles, allowing it to pick up sound evenly from all directions. A directional microphone, on the other hand, has high sensitivity to a specific angle, allowing it to pick up sound from that specific direction.

[0085] For example, a microphone array can have four omnidirectional microphones to pick up sound from all directions around the wearer. A microphone array can also have four directional microphones to pick up sound from all directions around the wearer. Alternatively, a microphone array can be composed of a combination of two omnidirectional and two directional microphones, where the first and third microphones can be omnidirectional, and the second and fourth microphones can be directional, thus picking up sound from all directions around the wearer.

[0086] In some embodiments, taking the unbent portion of the temple of the smart glasses as a cuboid as an example, when a user wears the smart glasses, the temple has an inner surface that directly contacts the wearer's skin, and three outer surfaces that do not directly contact the wearer's skin, including two outer surfaces adjacent to the inner surface (top and bottom), and one outer surface not adjacent to the inner surface. The sound inlet of the pickup channel can be located on any one of the outer surfaces; the sound inlets of different pickup channels can be located on the same outer surface or on different outer surfaces. Furthermore, when the temple of the smart glasses is a cylinder or other polyhedron, the portion directly in contact with the wearer's skin can be considered the inner surface, and the portion not directly in contact with the wearer's skin can be considered the outer surface.

[0087] In some embodiments, the sound pickup channel may have one or more sound inlets. For example, when there is only one sound inlet, sound from every direction can be picked up by the microphone, thus enabling the microphone to pick up all sounds around the wearer in all directions, increasing the pickup range. When the smart glasses are used as a hearing aid or other hearing assistance device, they can better help the wearer hear surrounding sounds. And when the smart glasses are used as a microphone or other sound pickup assistive device, they can better collect ambient sound when recording video with a mobile phone.

[0088] Furthermore, when there are two sound inlets, the lines connecting the two sound inlets have a direction. If the direction of the lines connecting the two sound inlets is the same as the direction of the lines connecting the temples, when the user wears the smart glasses, sounds from above and below the user will cancel each other out because they reach the two sound inlets at the same distance. This prevents the microphone from picking up the sound, thus blocking sounds from above and below the user. Sounds from directions other than those in front and behind the user will also be weakened because their components above and below the user are canceled out.

[0089] When a user wears smart glasses, the distance difference between the sounds coming from in front of and behind the user is greatest due to the two sound inlets. At this distance, the interference between the sounds entering the pickup channel from the two inlets is minimized, allowing the microphone to pick up the least amount of interfered sound. This results in the best sound pickup performance when capturing sounds from both directions. By designing two sound inlets for the microphone's pickup channel, the microphone gains directional sound pickup capabilities.

[0090] Furthermore, the direction of the two sound inlets of the pickup channel of the microphone can be different from the direction of the direction of the temple of the glasses; this application does not specifically limit this. For example, the direction of the two sound inlets of the pickup channel of one or more microphones in the microphone array can be set to a vertical direction to achieve directional sound pickup above and below the wearer. Alternatively, the direction of the two sound inlets of the pickup channel of one or more microphones in the microphone array can be set to a specific direction (such as tilted upwards or downwards at a specific angle) to achieve directional sound pickup from a specific direction.

[0091] In some embodiments, the pickup channel of the microphone may also have three or more pickup holes. When all the pickup holes are located on the same straight line, the microphone will have the ability to pick up sound in a directional manner, and the pickup direction is the direction of the straight line where all the pickup holes are located.

[0092] In some embodiments, the number of entrance holes for the corresponding pickup channel of each pickup in the pickup array may be the same or different, and this application does not specifically limit this. For example, the pickup array may have four pickups, wherein the pickup channels of the first and third pickups may have one entrance hole for omnidirectional pickup. The pickup channels of the second and fourth pickups may have two entrance holes for directional pickup.

[0093] Furthermore, in some embodiments, when each microphone in the microphone array has two sound inlets for its corresponding sound pickup channel, the sound inlets of the sound pickup channels for different microphones can be on the same straight line or on different straight lines; this application does not specifically limit this. For example, the microphone array can have four microphones, where the two sound inlets of the sound pickup channel of the first microphone and the two sound inlets of the sound pickup channel of the second microphone can be on the same straight line, with the direction consistent with the direction of the temple, for directional sound pickup in the front-back direction of the wearer. The straight line containing the two sound inlets of the sound pickup channel of the third microphone can be parallel to the straight line containing the sound inlets of the sound pickup channels of the first and second microphones, also for directional sound pickup in the front-back direction of the wearer. The straight line containing the two sound inlets of the sound pickup channel of the fourth microphone can point vertically, for directional sound pickup in the up-down direction of the wearer.

[0094] Based on the previous embodiment, in some embodiments, the microphone array 102 can be installed inside the temple of the smart glasses, which provides directional sound pickup capability while avoiding discomfort or affecting the wearer's image due to installation on the outside of the smart glasses.

[0095] Building upon the previous embodiment, in some embodiments, the wearable audio pickup device 100 may further include a microphone connected to the processor 130 and installed inside the nose bridge connector of the smart glasses. In this embodiment, the microphone installed inside the nose bridge connector of the smart glasses is specifically used to pick up the wearer's voice, so as to perform audio enhancement processing on the wearer's voice during calls or recordings.

[0096] In some embodiments, the microphone's pickup channel, installed inside the nose bridge connector of the smart glasses, may also have one or more sound inlets. When there is only one sound inlet, sound from every direction can be picked up by the microphone, thus enabling it to capture all sounds around the wearer from all directions, increasing the pickup range. When the smart glasses are used as a hearing aid or other hearing assistance device, they can better help the wearer hear surrounding sounds. And when the smart glasses are used as a microphone or other sound pickup auxiliary device, they can better collect ambient sound when recording video with a mobile phone.

[0097] Furthermore, when the microphone installed inside the nose bridge connector of the smart glasses has two sound inlets, when the user wears the smart glasses, sounds from the front and back of the user will cancel each other out because the distances to the two sound inlets are the same. This prevents the microphone from picking up the sound, thus effectively blocking sounds from the front and back of the user. Sounds from other directions besides the front and back of the user will also be weakened because their components in front of and behind the user are canceled out.

[0098] When a user wears smart glasses, the distance difference between the sounds arriving at the two sound inlets is greatest when they are above and below the user. At this distance, the interference between the sounds entering the pickup channel from these two inlets is minimized, allowing the microphone to pick up the least amount of interfered sound. This results in the best possible sound pickup performance when capturing sounds from above and below the user. By designing two sound inlets for the microphone's pickup channel, the microphone can directionally pick up the user's voice.

[0099] For example, please refer to Figure 5 , Figure 6 , Figure 7 , Figure 8 , Figure 9 and Figure 10 , Figure 5 This is a schematic diagram of the structure of a smart glasses provided in an embodiment of this application; Figure 6 for Figure 5 A cross-sectional view of the internal structure of the temple of a smart pair of glasses is provided. Figure 7 for Figure 5 A cross-sectional view of the temples of a smart glasses provided in the first section; Figure 8 for Figure 5 A diagram showing the internal structure of the temple of a pair of smart glasses in a second cross-section; Figure 9 for Figure 5 A cross-sectional view of the temples of a type of smart glasses in the second section; Figure 10 This is a vertical cross-sectional view of a nose bridge connector in smart glasses provided in an embodiment of this application.

[0100] The smart glasses provided in this embodiment have a total of nine directional microphones, eight of which are installed inside the two temples. The direction of the sound inlet holes of the microphone channels is consistent with the direction of the straight line of the temples of the smart glasses. The openings of the microphone channels at the temples are as follows: Figure 5As shown, 501 is the first microphone, 502 is the second microphone, 503 is the third microphone, and 504 is the fourth microphone. The other temple also houses four microphones, with the same arrangement of sound inlets, which will not be described further here. The first cross-section is the plane containing the outer surface of the temple (the plane where the sound inlets are located in the figure), and the second cross-section is the plane containing the upper surface of the temple.

[0101] like Figure 5 As shown, the connecting lines of each sound inlet hole in the pickup channel of the four microphones are located on the same straight line, and the direction of this straight line is consistent with the extension direction of the temple. That is, the eight sound holes of the four microphones on this temple are located on the same straight line, and this straight line points in the front and back direction of the wearer. When the user wears smart glasses, the sound from above and below the user will cancel each other out because the distance to the two sound inlets is the same. As a result, the microphones cannot pick up the sound, thus blocking the sound from above and below the user. Sounds from other directions besides the front and back direction of the user will also be weakened because the sound components existing above and below the user are canceled out.

[0102] When a user wears smart glasses, the distance difference between the sounds coming from in front of and behind the user is greatest due to the two sound inlets. At this point, the interference between the sounds entering the pickup channel from the two inlets is minimized, allowing the microphone to pick up the least amount of interfered sound. This results in the best sound pickup performance when capturing sounds from both directions. By aligning the four microphones (a total of eight sound holes) on the temple of the glasses in a straight line pointing in both the front and rear directions of the wearer, the smart glasses gain directional sound pickup capabilities, enabling them to pick up signals from a considerable distance.

[0103] In some embodiments, the purpose of setting at least two microphones in the microphone array is that the audio signals picked up by multiple microphones can be merged into one enhanced audio signal after being processed by the processor. Combined with the above embodiment of setting two sound inlet holes for the microphone pickup channel, the microphone has the ability to enhance the pickup direction, enabling the wearable microphone device to obtain the ability to pick up sound beyond the human ear's hearing distance, making it convenient for the wearer to hear distant sounds or collect audio signals from a distance, thus improving the user experience.

[0104] In some embodiments, the processor performs audio enhancement processing on the audio signals picked up by multiple microphones. Specifically, this can be achieved by performing super-directional beamforming on the audio signals picked up by the microphone array. This super-directional beamforming can be implemented using an optimized beamformer that controls white noise amplification. Its function is to perform maximum directional beamforming under the constraint of minimal or no amplification of white noise. The signal after beamforming changes from 4 channels to 1 channel. At this time, the input signal-to-noise ratio of the 1-channel audio signal is higher than that of any channel in the original 4-channel signal, making the acquired long-distance audio signal clearer and further improving the user experience.

[0105] In some embodiments, the microphone includes two back-to-back microphone ports, and the microphone channel has two inlet holes located on the outer surface of the temple of the smart glasses. Each inlet hole corresponds to one microphone port, and the microphone channel connects the corresponding inlet hole and microphone port. The straight line containing the two inlet holes is aligned with the direction of the straight line containing the temple of the smart glasses.

[0106] Specifically, considering the potential irregular distribution and uneven size of the internal space of the temples, this embodiment provides two structural designs for microphones. For example... Figure 6 , Figure 7 , Figure 8 and Figure 9 As shown, 601 is a directional pickup, which can be considered as a hexahedron in shape, with a sound hole on each of two opposing faces, and these holes are through holes. The directional pickup has two sound holes at both ends of the diaphragm, one on each side. The vibration of the diaphragm depends on the pressure difference between the two ends according to the phase relationship. A fine acoustic filter is placed at the front end of the rear sound hole to delay the sound. This allows the sound coming from the rear to reach the diaphragm simultaneously from both the front and rear sound holes and cancel each other out, resulting in high sensitivity to sound at a specific angle. 604 is a structural component attached to the temple of the glasses, used to form a sound pickup channel suitable for the directional pickup 601. 602 are the two sound ports of the pickup 601 itself, and 603 is the entrance hole of the sound pickup channel of the pickup 601. The two sound ports 602 of the pickup 601 itself correspond to the two entrance holes 603 of the sound pickup channel, and the sound pickup channel connects the entrance holes 603 and the sound ports 602.

[0107] Using the direction of the temple line as the x-axis, the direction of the lines containing the left and right edges of the lens as the y-axis, and the direction of the lines containing the top and bottom edges of the lens as the z-axis, the microphone 601 can be described as follows: Figure 6 , Figure 7 , Figure 8 and Figure 9In the left-to-right arrangement, the first and third microphones are mounted inside the temple parallel to the xy plane (the plane where the x-axis and y-axis are located). The first and third microphones are laid flat in the xy plane. The pickup channel matching the third microphone consists of the sound inlet 603 on the temple, the microphone 601, and the structural component 604 attached to the temple. The two sound inlets 602 of the third microphone 601 correspond to the two sound inlets 603 of the pickup channel. The sound is picked up by the microphone 601 through the sound inlets 602 and 603 on the temple.

[0108] Furthermore, the pickup 601 can also be like... Figure 6 , Figure 7 , Figure 8 and Figure 9 In the left-to-right arrangement, the second and fourth microphones are mounted inside the temple parallel to the yz plane (the plane where the y-axis and z-axis are located) of the lens. It should be noted that this application does not specifically limit the installation method of the microphone 601 inside the temple. Users or those skilled in the art can freely choose the appropriate installation method to adapt to irregular or uneven spatial distribution inside the temple.

[0109] like Figure 10 As shown, the ninth directional microphone 1001 can be installed inside the nose bridge connector of the smart glasses. The directional microphone 1001 can be installed inside the nose bridge connector of the smart glasses parallel to the xz plane (the plane where the x-axis and z-axis are the same). Its main function is to pick up the wearer's voice and assist in enhancing the wearer's voice.

[0110] In some embodiments, when the wearable audio pickup device 100 is a wireless earphone, the microphone array 102 can be installed inside the ear stem of the wireless earphone. This provides audio pickup capability that exceeds the human ear's hearing distance while avoiding the discomfort of wearing it or the impact on the wearer's image caused by installing it outside the wireless earphone.

[0111] Based on the previous embodiment, in some embodiments, please refer to Figure 11 and Figure 12 , Figure 11 This is a schematic diagram of the structure of a wireless earphone provided in an embodiment of this application; Figure 12 This is a schematic diagram illustrating long-distance sound pickup using the wireless earphone provided in the embodiments of this application.

[0112] like Figure 11As shown, for each microphone in the microphone array, the pickup channel matched with the microphone has two sound inlets 1101 located on the outer surface of the ear stem of the wireless earphone. The straight line of the two sound inlets 1101 is aligned with the straight line of the ear stem of the wireless earphone. When the user wears the wireless earphone, the sounds from in front of and behind the user will cancel each other out because the distance to the two sound inlets 1101 is the same. Therefore, the microphone cannot pick up the sound, allowing the microphone to block the sounds from in front of and behind the user. Sounds from directions other than the user's vertical direction will also be attenuated because the sound components present in front of and behind the user are canceled out.

[0113] The sound from above and below the user will have the greatest distance difference to reach the two sound inlets. At this time, the sound entering the pickup channel from the two sound inlets one after the other will have the least interference with each other. As a result, the microphone can pick up the least amount of interfered sound, which makes the wireless headphones able to pick up the wearer's voice more clearly and accurately.

[0114] like Figure 12 As shown, when it's necessary to pick up sound from the front and rear directions, the position of one earphone can be adjusted so that the stem points forward. In this case, the connection between each inlet hole 1201 points towards the wearer's front and rear direction. Sounds from above and below the user will cancel each other out because the distances to the two inlet holes 1201 are equal. This allows the microphone to block sounds from above and below the user. Sounds from directions other than those above and below the user will also be attenuated because their components above and below the user are canceled out. Sounds from the front and rear of the user will have the greatest distance difference to the two inlet holes 1201. This minimizes interference between the sounds entering the microphone channel from the two inlet holes 1201 one after the other, allowing the microphone to pick up the least amount of interfered sound and achieve the best pickup effect when picking up sound from the front and rear directions. This allows the microphone to achieve the best sound pickup effect when picking up sound from the front and rear directions, thus enabling the wireless headphones to have a sound pickup capability that exceeds the human ear's hearing distance.

[0115] Please refer to Figure 13 , Figure 13 A flowchart of a sound pickup method provided in this application embodiment can be applied to the wearable sound pickup device in the above embodiments. The method may include, but is not limited to, the following steps:

[0116] Step S01: In response to the input pickup command, audio signals from the environment are acquired through a pickup array.

[0117] In some embodiments, the audio signal includes a first audio signal and a second audio signal, wherein the direction of the first audio signal is consistent with the extension direction of the extension member, and the direction of the second audio signal is inconsistent with the extension direction of the extension member.

[0118] In some embodiments, the voice pickup command can be input by the wearer pressing a designated button on the wearable voice pickup device or touching a touch area on the wearable voice pickup device. For example, the wearer can trigger the voice pickup command by double-clicking a designated button.

[0119] The voice pickup command can also be input by the wearer through a terminal or mobile terminal that is wirelessly connected to the wearable voice pickup device. For example, the wearer can send the voice pickup command to the wearable voice pickup device by clicking on the voice pickup command on the mobile phone. This application does not specifically limit the input method of the voice pickup command.

[0120] Step S02: The processor enhances the first audio signal and suppresses the second audio signal to obtain a processed third audio signal, and then outputs the third audio signal.

[0121] In some embodiments, this application first uses a microphone array to collect a first audio signal in the environment that is aligned with the extension direction of the extension component and a second audio signal that is not aligned with the extension direction of the extension component. Then, a processor amplifies the first audio signal and suppresses the second audio signal, so that the first audio signal in a specific direction is stronger in the output audio signal, while the second audio signal in other directions is weaker, thereby achieving directional sound pickup in the environment.

[0122] In some embodiments, the processor performs audio enhancement processing on the audio signals picked up by multiple microphones. Specifically, this can be achieved by performing super-directional beamforming on the audio signals picked up by the microphone array. The super-directional beamforming can be implemented based on an optimized beamformer that controls white noise amplification.

[0123] Based on the above embodiments, in some embodiments, an enhanced signal in the end-fire direction can be obtained through super-directional beamforming, and then the enhanced signal after beamforming can be used as a guide to feed all paths back to a preset neural network for audio enhancement, so as to further improve the sound pickup effect.

[0124] For example, please refer to Figure 14 , Figure 14 This is a schematic diagram of an audio signal processing method provided in an embodiment of this application.

[0125] For a single temple, four directional microphones can be placed along the temple at equal or variable intervals. The signal from the single-sided microphone array first undergoes super-directional beamforming. Super-directional beamforming is an optimized beamformer that controls white noise amplification. Its function is to achieve maximum directional beamforming under the constraint of minimal or no white noise amplification. After beamforming, the signal changes from four channels to one channel. At this point, the input signal-to-noise ratio of the single-channel audio signal is higher than that of any channel in the original four-channel signal, making the acquired long-distance audio signal clearer and further improving the user experience.

[0126] and Figure 14 The post-processing module, from the "encoder" module to the "decoder," constitutes the entire neural network architecture. The beamformed signal and the target signal are used to train the neural network. A complex temporal convolutional network is used to learn the spectrum, thereby obtaining a neural network capable of extracting the target sound source. It should be noted that this application does not specifically limit the type of neural network, as long as the neural network can achieve the purpose of extracting the target sound source.

[0127] In some embodiments, after the mobile phone sends control commands to the smart glasses, the smart glasses will pick up and process the sound from the target to be enhanced and then send back a single-channel and / or dual-channel audio stream to the mobile phone. The processed signal can be processed separately for each temple to form a dual-channel audio stream (e.g., Figure 14 The single-sided signal 1 and single-sided signal 2 shown can be processed together to form a single-channel audio stream, and the audio stream can be wirelessly transmitted to the mobile phone and stored by the mobile phone.

[0128] In some embodiments, the pickup task category of the pickup command may include a hearing aid task, meaning that the wearable pickup device can be considered a distance hearing aid, capable of picking up and amplifying sounds beyond the human hearing distance so that the wearer can clearly hear distant sounds. When the pickup task category of the pickup command is a hearing aid task, the processed audio signal can be output to a speaker.

[0129] The third audio signal is output, specifically as follows:

[0130] The processor parses the pickup instruction to obtain the pickup task category, which includes hearing aid tasks.

[0131] If the sound pickup task is classified as a hearing aid task, then the third audio signal is output to the speaker.

[0132] In some embodiments, the method further includes:

[0133] In response to a first determination instruction input by the user, the first determination instruction is executed to determine a first target to be enhanced;

[0134] The processor amplifies the fourth audio signal emitted by the first target to be amplified and suppresses the fifth audio signal to obtain a sixth audio signal; wherein the fifth audio signal is any audio signal other than the fourth audio signal.

[0135] The sixth audio signal is output to the speaker in real time.

[0136] In some embodiments, the sound pickup task category of the sound pickup command may also include a sound pickup auxiliary task. When the sound pickup task category of the sound pickup command is a sound pickup auxiliary task, the processed audio signal can be output to the target device. That is, at this time, the wearable sound pickup device can be regarded as a long-distance auxiliary sound pickup, which can assist video recording devices such as mobile phones and cameras in sound pickup, thereby improving the sound pickup effect.

[0137] The sound pickup task category includes sound pickup assistance tasks, and the method further includes:

[0138] If the sound pickup task category is a sound pickup auxiliary task, then the target device is obtained by parsing the sound pickup instruction through the processor;

[0139] The third audio signal is output to the target device via a wireless communication module.

[0140] In some embodiments, the method further includes:

[0141] Receive a second determination instruction sent by the target device, and execute the second determination instruction to determine a second target to be enhanced;

[0142] The processor amplifies the seventh audio signal emitted by the second target to be amplified and suppresses the eighth audio signal to obtain a ninth audio signal; wherein the eighth audio signal is an audio signal other than the seventh audio signal.

[0143] The ninth audio signal is output to the target device via a wireless communication module.

[0144] For example, please refer to Figure 15 , Figure 15 This is a schematic diagram illustrating the process of using a wearable audio pickup device for audio pickup assistance, as provided in an embodiment of this application.

[0145] like Figure 15As shown, when a wearable audio pickup device detects a connection request from another device, it can decide whether to establish a connection based on user input. After establishing a connection, the wearable audio pickup device can decide whether to perform multi-device recording enhancement processing based on default settings or user input. For example, when a user's mobile phone and smart glasses establish a wireless connection via Bluetooth, if the default setting is to automatically enable the multi-device recording enhancement function after connection, the smart glasses can detect in real time whether the mobile phone has activated its own audio pickup function (e.g., whether the mobile phone has activated its recording function, video recording function, voice sending function, voice call function, video call function, etc.). If the mobile phone's own audio pickup function is detected, the sound is picked up through the microphone array and the obtained audio signal is sent to the mobile phone in real time for storage.

[0146] In the future, the mobile phone can further process the audio signal picked up by the smart glasses. For example, the mobile phone can use the ambient sound picked up by itself as the background and superimpose the audio signal picked up by the smart glasses onto the ambient sound. This will achieve the goal of both preserving the ambient sound and enhancing the target sound signal. Furthermore, the left and right temples can generate the enhancement signals for the left and right channels, respectively.

[0147] After generating the default playback sound signal, users can manually adjust the parameter settings to further enhance the target signal or retain more ambient sound signals.

[0148] Based on the above embodiments, in some embodiments, when the sound pickup task category of the sound pickup command is a sound pickup assistance task, the wearer can also select the target to be enhanced by video recording devices such as mobile phones and cameras, so that the wearable sound pickup device can perform audio enhancement processing specifically for the sound of the target to be enhanced, thereby further improving the sound pickup effect.

[0149] For example, please refer to Figure 16 and Figure 17 , Figure 16 This is a schematic diagram illustrating a scenario where smart glasses are selected for sound pickup assistance, as provided in an embodiment of this application. Figure 17 This is a schematic diagram illustrating a scenario where smart glasses are used for sound pickup assistance, as provided in an embodiment of this application.

[0150] like Figure 16 As shown, in a scenario where smart glasses are used for audio pickup assistance, if a user needs to record a video and wants to enhance the audio of a specific target through the smart glasses, they can first click the recording function on their phone and select "Yes" in the pop-up "Do you want to activate the audio pickup assistance function?" page. The phone will then output devices that can be used for audio enhancement processing, such as smart glasses or wireless headphones, which the user can select. For example, the user can select "Smart Glasses 2" as the device for audio enhancement processing.

[0151] like Figure 17 As shown, after determining that the device for audio enhancement processing is smart glasses, the mobile phone detects the user's selection of a target to be enhanced on the screen and sends a confirmation command containing the target to be enhanced to the smart glasses. The smart glasses then execute this confirmation command through their processor to determine the target to be enhanced. When the mobile phone detects the user's selection to start recording on the screen, it sends a sound pickup command to the smart glasses and simultaneously starts recording the target area. After receiving the sound pickup command, the smart glasses acquire the audio signal of the target to be enhanced through a microphone array.

[0152] In some embodiments, after the user wears the smart glasses, they face the target to be photographed. After the target to be enhanced is determined, the smart glasses can acquire the audio signal in the environment through the microphone array. At this time, the direction of the audio signal of the target to be enhanced is consistent with the extension direction of the smart glasses. The audio signal of the target to be enhanced will be enhanced, while the audio signal in other directions will be weakened.

[0153] It is understood that this application provides a mode of sound pickup assisted by smart glasses, allowing the mobile phone to switch its desired sound pickup mode, such as omnidirectional or directional, based on the user's actual sound pickup needs when the user has video recording requirements. Specifically, when the user selects directional sound pickup mode, the mobile phone can detect the user's convenient selection operation of a certain area within the viewfinder, identify the user's target to be enhanced, and thus selectively pick up sound signals from the target of interest during video recording, while blocking sound signals from other targets. Based on the sound pickup method provided by this application, a more user-friendly sound pickup experience can be offered.

[0154] For example, suppose a user is recording video or making a video call on their phone during a concert. If the user selects directional audio pickup mode and chooses a target subject to be filmed, the phone can use the audio pickup glasses to selectively pick up audio signals from the target (such as a musician's instrument, a presenter, etc.) while blocking out noise signals such as audience cheers and staff movement. For instance, the phone can use the audio pickup glasses to control the microphones located in the vicinity of the performer, presenter, or other target to collect audio signals within that area, while controlling the microphones in the vicinity of the stands to not collect audio signals.

[0155] In some embodiments, after the mobile phone enables the recording function, it can only acquire video data, while all audio signals are acquired through smart glasses, thereby saving mobile phone storage resources.

[0156] Furthermore, after enabling the recording function, the phone can acquire both video and audio signals. The audio signal acquired by the phone can be superimposed on the audio signal acquired by the smart glasses to enhance or reduce ambient noise. In this way, the audio output scheme can be adjusted as needed when the user reviews the recording, providing a more user-friendly audio pickup experience. For example, as one possible solution, the user can play only the sound of the target subject in the focus area; as another, the user can choose to play the sounds of any number of target subjects in the recording.

[0157] In some embodiments, users can also input a switching command on a mobile phone or smart glasses. After detecting the switching command, the mobile phone or smart glasses can determine a new target based on the switching command, switch the target to be photographed to the new target, and continue to pick up sound from the new target.

[0158] In some embodiments, the sound emitted by the target to be enhanced is amplified, while other sounds besides the target to be enhanced are suppressed. This can be achieved by modifying the compensation value for the sound.

[0159] For example, corresponding audio enhancement parameters can be generated based on the positional relationship between the target to be enhanced and the wearable audio pickup device. The audio signal can be enhanced based on the second audio enhancement parameters to obtain an audio enhancement signal, and the audio enhancement signal can be output to the target device through a wireless communication module.

[0160] Based on the above embodiments, in some embodiments, an enhanced signal in the end-fire direction can be obtained through super-directional beamforming, and then the enhanced signal after beamforming can be used as a guide to feed all paths back to a preset neural network for audio enhancement, so as to further improve the sound pickup effect.

[0161] For example, super-pointing beamforming can be performed on the audio signals collected by each microphone to obtain super-pointing beam signals; then, a preset neural network can be used to perform audio enhancement processing on the super-pointing beam signals according to audio enhancement parameters to obtain audio enhancement signals.

[0162] Based on the above embodiments, in some embodiments, after obtaining the audio enhancement signal, the following steps may also be performed:

[0163] The ambient audio signal is acquired, and the ambient audio signal and the second audio enhancement signal are fused using a preset fusion algorithm to obtain an audio fused signal;

[0164] The audio fusion signal is output to the target device via a wireless communication module.

[0165] For example, after the smart glasses receive the second audio enhancement signal, the second audio enhancement signal can be superimposed on the ambient audio signal as a background, so as to both preserve the ambient sound and enhance the target sound signal.

[0166] It should be noted that the above embodiments of this application only use a mobile phone in a recording scenario as an example. For scenarios such as video calls, similar methods for directional audio pickup can also be used. For example, when the mobile phone displays the shooting interface in a video call scenario, if the mobile phone detects that the user selects one or more target subjects on the video call interface, the mobile phone can use smart glasses to only acquire audio from those one or more target subjects and block audio from other target subjects. Alternatively, the mobile phone can use smart glasses to amplify the audio from those one or more target subjects and reduce the audio from other target subjects. This provides users with a more user-friendly video call experience.

[0167] It should be noted that the sound pickup method provided in this embodiment can be implemented by a wearable sound pickup device in which the arrangement direction of the microphones is consistent with the extension direction of the extension component, as provided in the above embodiment. Alternatively, it can be implemented by other types of wearable sound pickup devices in which the arrangement direction of the microphones is inconsistent with the extension direction of the extension component. This application does not make any specific limitation in this regard.

[0168] In some embodiments, please refer to Figure 18 , Figure 18 This is a schematic diagram illustrating a scenario where hearing aids are provided through smart glasses, as provided in an embodiment of this application.

[0169] like Figure 18 As shown, suppose A, as the meeting recorder, needs to record the meeting content. However, B and C, who are in the middle of the meeting, are far away from A, and D and E, who are talking and working next to A, cannot hear the conversation between B and C, making it impossible for A to record the meeting properly.

[0170] To avoid this situation, in scenarios where hearing aids are used with smart glasses, if A is wearing smart glasses, they can select a directional sound pickup mode on the smart glasses and choose the target they want to enhance. The smart glasses can then selectively pick up the sound signals from the targets B and C, while blocking out noise signals such as conversations between D and E, and the movement of other staff. For example, the smart glasses can control the microphones located in the direction of B and C to collect audio signals within that direction, while controlling the microphones located in the direction of D and E not to collect audio signals. Finally, the collected audio signals are output in real time through the smart glasses' speakers, allowing A to take meeting notes based on what they hear.

[0171] For example, a mobile phone can also collect all audio signals through the microphone array of smart glasses, then determine the audio signals at the locations of B and C as the first audio signal, and the audio signals at the locations of D and E as the second audio signal. Finally, the processor enhances the first audio signal and suppresses the second audio signal to obtain the processed audio signal. The processed audio signal is then output in real time through the speaker of the smart glasses. At this time, A can take meeting notes based on the meeting content heard.

[0172] In some embodiments, sound pickup can also be achieved using AR glasses with "super hearing" capabilities. For example, image target recognition can be achieved through voice interaction. In a scenario where hearing aids are provided through smart glasses, if A is wearing AR glasses, the AR glasses can first perform person tag recognition, identifying B and C with "conference tag," and D and E with "communication tag / noise tag."

[0173] At this point, user A can input statements into the intelligent voice assistant to target specific individuals. For example, user A can input "enhance users with meeting tags", "block users with noise tags", or "enter meeting target pickup mode". The AR glasses will confirm the target based on the input statement and can then selectively pick up the sound signals from users' focus targets B and C, while blocking the conversation between D and E, noise signals from other staff members walking around, and other noise signals.

[0174] It should be noted that the above embodiments of this application only use a mobile phone in a recording scenario as an example. For scenarios such as video calls, similar methods for directional audio pickup can also be used. For example, when the mobile phone displays the shooting interface in a video call scenario, if the mobile phone detects that the user selects one or more target subjects on the video call interface, the mobile phone can use smart glasses to only acquire audio from those one or more target subjects and block audio from other target subjects. Alternatively, the mobile phone can use smart glasses to amplify the audio from those one or more target subjects and reduce the audio from other target subjects. This provides users with a more user-friendly video call experience.

[0175] In addition, the sound pickup method provided in this embodiment can be implemented by a wearable sound pickup device in which the arrangement direction of the microphones is consistent with the extension direction of the extension component, as provided in the above embodiment. Alternatively, it can be implemented by other types of wearable sound pickup devices in which the arrangement direction of the microphones is inconsistent with the extension direction of the extension component. This application does not make any specific limitation in this regard.

[0176] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A wearable audio pickup device, characterized in that, include: A wearable microphone body, the wearable microphone body including an extension component; A microphone array installed inside the wearable microphone device body for long-distance sound pickup, the microphone array including at least two microphones, the arrangement direction of each microphone in the microphone array being consistent with the extension direction of the extension component; The processor connected to the microphone array; The wearable microphone is specifically a smart glasses, and the extension component is the temple of the smart glasses. For each microphone in the microphone array, the temple is provided with a microphone channel that matches the microphone, and the microphone channel has at least one sound inlet hole. The microphone includes two back-to-back sound inlets, and a diaphragm is provided in the microphone. The two back-to-back sound inlets are located on opposite sides of the microphone. The microphone channel has two sound inlets located on the outer surface of the temple of the smart glasses. Each sound inlet hole corresponds to one sound inlet hole. The microphone channel connects the corresponding sound inlet hole and the sound inlet hole. The two microphone channels corresponding to the two back-to-back sound inlets are two independent microphone channels. The straight line where the two sound inlets are located is consistent with the direction of the straight line where the temple of the smart glasses is located. The wearable microphone also includes a microphone connected to the processor and installed inside the nose bridge connector of the smart glasses; The microphone installed inside the nose bridge connector of the smart glasses has two sound inlets located on the outer surface of the nose bridge connector of the smart glasses, and the direction of the line where the two sound inlets are located is vertically downward.

2. The wearable audio pickup device according to claim 1, characterized in that, The microphone array is installed inside the temples of the smart glasses.

3. The wearable audio pickup device according to any one of claims 1-2, characterized in that, Each of the pickups in the pickup array is specifically a directional pickup.

4. The wearable audio pickup device according to claim 3, characterized in that, The wearable audio pickup device also includes a speaker connected to the processor.

5. A sound pickup method, characterized in that, Applied to the wearable audio pickup device as described in any one of claims 1-4, the audio pickup method includes: In response to an input pickup command, an audio signal in the environment is acquired through a microphone array. The audio signal includes a first audio signal and a second audio signal. The direction of the first audio signal is consistent with the extension direction of the extension component, and the direction of the second audio signal is inconsistent with the extension direction of the extension component. The processor enhances the first audio signal and suppresses the second audio signal to obtain a processed third audio signal, which is then output.

6. The sound pickup method according to claim 5, characterized in that, The step of outputting the third audio signal includes: The processor parses the pickup instruction to obtain the pickup task category, which includes hearing aid tasks. If the sound pickup task is classified as a hearing aid task, then the third audio signal is output to the speaker.

7. The sound pickup method according to claim 6, characterized in that, The method further includes: In response to a first determination instruction input by the user, the first determination instruction is executed to determine a first target to be enhanced; The processor amplifies the fourth audio signal emitted by the first target to be amplified and suppresses the fifth audio signal to obtain a sixth audio signal; wherein the fifth audio signal is any audio signal other than the fourth audio signal. The sixth audio signal is output to the speaker in real time.

8. The sound pickup method according to claim 6, characterized in that, The sound pickup task category includes sound pickup assistance tasks, and the method further includes: If the sound pickup task category is a sound pickup auxiliary task, then the target device is obtained by parsing the sound pickup instruction through the processor; The third audio signal is output to the target device via a wireless communication module.

9. The sound pickup method according to claim 8, characterized in that, The method further includes: Receive a second determination instruction sent by the target device, and execute the second determination instruction to determine a second target to be enhanced; The processor amplifies the seventh audio signal emitted by the second target to be amplified and suppresses the eighth audio signal to obtain a ninth audio signal; wherein the eighth audio signal is an audio signal other than the seventh audio signal. The ninth audio signal is output to the target device via a wireless communication module.

Images