Intelligent audio-visual auxiliary glasses, audio-visual auxiliary method, electronic equipment and storage medium

Abstract

The invention relates to the field of audio-visual conversion, and provides intelligent audio-visual auxiliary glasses, an audio-visual auxiliary method, electronic equipment and a storage medium, and the intelligent audio-visual auxiliary glasses comprise a glasses frame which comprises a microphone and lenses; the character conversion unit is used for receiving external sound information through the microphone, converting the sound information into characters and displaying the characters on the lens; and the voice conversion unit is used for receiving external visual information through the visual sensor, converting the visual information into voice and playing the voice. The invention provides an integrated audio-visual conversion device, which is used for solving the defect of lack of audio-visual conversion devices for visually impaired people and hearing impaired people in the prior art, can help the visually impaired people to obtain front visual information and can also help the hearing impaired people to obtain sound information, so that the audio-visual conversion effect is improved. And bidirectional enhancement of visual information and auditory information is realized.

Description

Technical Field

[0001] This invention relates to the field of audiovisual conversion technology, and in particular to a smart audiovisual assistive glasses, an audiovisual assistive method, an electronic device, and a storage medium. Background Technology

[0002] There are many visually and hearing impaired people in society. Their access to information from the outside world is limited, which affects their lives.

[0003] Most existing visual assistive devices rely on cameras and voice broadcasting devices, but they generally lack systematic integration, cannot perceive the external physical environment, and are difficult to meet the information perception needs in complex environments. For hearing-impaired individuals, the current main reliance is on hearing aids and subtitle recognition systems, but these devices are usually bulky, inconvenient to use, and cannot reproduce physical scenes. They also lack integrated noise suppression mechanisms, affecting the accuracy of information transmission and user experience. Summary of the Invention

[0004] This invention provides intelligent audiovisual assistive glasses, an audiovisual assistive method, an electronic device, and a storage medium to address the deficiency in related technologies of lacking audiovisual conversion devices for visually and hearing impaired individuals. The solution of this application provides an integrated audiovisual conversion device that can help visually impaired individuals obtain visual information in front of them and help hearing impaired individuals obtain sound information, thereby achieving bidirectional enhancement of visual and auditory information.

[0005] This invention provides intelligent audiovisual assistive glasses, comprising:

[0006] Eyeglasses frame, including microphone and lenses;

[0007] A text conversion unit is used to receive external sound information through the microphone, convert the sound information into text, and display it on the lens.

[0008] The voice conversion unit is used to receive external visual information through a visual sensor, and to convert the visual information into voice and play it.

[0009] The intelligent audiovisual assistive glasses provided by the present invention also include a noise control unit;

[0010] The noise control unit is used to receive ambient noise information through the microphone and generate an anti-noise signal based on the ambient noise information;

[0011] The anti-noise signal is played through the speaker of the eyeglasses frame to suppress the environmental noise information.

[0012] According to the intelligent audiovisual assistive glasses provided by the present invention, the text conversion unit is further configured to:

[0013] The phoneme sequence of the external sound information is acquired through a visual sensor;

[0014] Speech features are extracted from the external sound information received by the microphone;

[0015] The phoneme sequence and the speech features are input-aligned and jointly modeled to obtain the audio signal;

[0016] The audio signal is used to convert text, and the converted text is then displayed on the lens.

[0017] According to the intelligent audiovisual assistive glasses provided by the present invention, the voice conversion unit is further used for:

[0018] The visual sensors acquire images of the scene in front of the user, and the visual sensors include at least two, which are respectively disposed on both sides of the eyeglass frame;

[0019] Convert the pixels in the scene image in front of the user into three-dimensional coordinate values ​​to realize the target 3D of the scene in front of the user;

[0020] The 3D rendering of the scene in front of the user is then broadcast to the user via voice.

[0021] According to the intelligent audiovisual assistive glasses provided by the present invention, the noise control unit is further configured to:

[0022] The acoustic path is determined based on the environmental noise signal;

[0023] The anti-noise signal is played through the speaker of the eyeglass frame based on the acoustic path.

[0024] According to the intelligent audiovisual assistive glasses provided by the present invention, the noise control unit is further configured to:

[0025] The environmental noise signal is input into a pre-built noise prediction model to obtain the acoustic path output by the noise prediction model;

[0026] The noise prediction model is trained using a database of different noise types and their corresponding acoustic paths.

[0027] The present invention also provides an intelligent audiovisual assistance method, applied to any of the above-mentioned intelligent audiovisual assistance glasses, comprising:

[0028] Receive external sound information, and convert the sound information into text and display it;

[0029] And / or,

[0030] It receives external visual information and converts the visual information into speech and plays it.

[0031] The intelligent audiovisual assistance method provided by the present invention further includes:

[0032] Receive environmental noise information and generate an anti-noise signal based on the environmental noise information;

[0033] The anti-noise signal is played to suppress the environmental noise information.

[0034] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the intelligent audiovisual assistance methods described above.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the intelligent audiovisual assistance methods described above.

[0036] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the intelligent audiovisual assistance methods described above.

[0037] The intelligent audiovisual assistive glasses provided by this invention may include an eyeglass frame. Thus, the intelligent audiovisual assistive glasses can be worn on the user's face like normal eyeglasses. During use, the text conversion unit can convert external sound information into text and display it on the lens so that hearing-impaired people can perceive sound information. The speech conversion unit can also convert visual information into speech and play it so that visually impaired people can perceive visual information in front of them. This achieves bidirectional enhancement of visual and auditory information, providing strong technical support for multimodal information interaction and adaptive environmental perception. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0039] Figure 1 This is one of the structural schematic diagrams of the intelligent audiovisual assistive glasses provided in the embodiments of the present invention;

[0040] Figure 2 This is a second structural schematic diagram of the intelligent audiovisual assistive glasses provided in an embodiment of the present invention;

[0041] Figure 3This is the third structural schematic diagram of the intelligent audiovisual assistive glasses provided in the embodiments of the present invention;

[0042] Figure 4 This is the fourth structural schematic diagram of the intelligent audiovisual auxiliary glasses provided in the embodiments of the present invention;

[0043] Figure 5 This is a flowchart illustrating the intelligent audiovisual assistance method provided in an embodiment of the present invention;

[0044] Figure 6 This is a schematic diagram of the audio-visual conversion process provided in an embodiment of the present invention;

[0045] Figure 7 This is a schematic diagram of the audiovisual conversion process provided in an embodiment of the present invention;

[0046] Figure 8 This is a schematic diagram of the physical structure of the electronic device provided in an embodiment of the present invention.

[0047] in:

[0048] 1-First sound module; 2-Second sound module; 3-First front-facing camera module;

[0049] 4-Second front-facing camera module; 5-Front-facing microphone acquisition module;

[0050] 6-First rear microphone acquisition module; 7-Second rear microphone acquisition module;

[0051] 8-Speech conversion unit; 9-Text conversion unit; 10-Power supply and communication unit;

[0052] 11-Noise control unit; 12-Signal preprocessing unit; 13-Lens. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0054] Figure 1 This is one of the structural schematic diagrams of the intelligent audiovisual auxiliary glasses provided in the embodiments of the present invention.

[0055] Figure 2 This is the second structural schematic diagram of the intelligent audiovisual auxiliary glasses provided in the embodiments of the present invention.

[0056] Figure 3This is the third structural schematic diagram of the intelligent audiovisual auxiliary glasses provided in the embodiments of the present invention.

[0057] Figure 4 This is the fourth structural schematic diagram of the intelligent audiovisual auxiliary glasses provided in the embodiments of the present invention.

[0058] like Figures 1 to 4 As shown, this embodiment provides a smart audiovisual assistive glasses, including:

[0059] Eyeglasses frame, including microphone and lenses;

[0060] A text conversion unit is used to receive external sound information through the microphone, convert the sound information into text, and display it on the lens.

[0061] The voice conversion unit is used to receive external visual information through a visual sensor, and to convert the visual information into voice and play it.

[0062] In practical applications, the sound-generating module can include two modules: a first sound-generating module 1 and a second sound-generating module 2, which are mounted on the two temples of the eyeglasses frame. These modules can be loudspeakers or bone conduction devices, allowing the wearer to both play speech and retain environmental auditory perception. By positioning them on the temples, they are close to the user's ears, ensuring clear perception of the sound information. A first front-facing camera module 3 and a second front-facing camera module 4 are positioned directly above the lenses of the eyeglasses frame. These modules feature wide-angle shooting and night vision capabilities, capturing images of the scene in front of the user, and are respectively positioned on the left and right ends of the eyeglasses frame. It can acquire the scene in front of the user, and because it has two cameras, it can realize the 3D representation of the target in front of the user based on binocular vision, providing the user with more visual information; front microphone acquisition module 5, first rear microphone acquisition module 6, and second rear microphone acquisition module 7, wherein the front microphone acquisition module is located in the middle of the frame, and uses beamforming technology to acquire directional speech and environmental noise; the first rear microphone acquisition module 6 and the second rear microphone acquisition module 7 are symmetrically distributed on both sides of the glasses frame, and can be used to acquire surrounding environmental noise for subsequent generation of anti-noise signals; speech conversion unit 8, which integrates target detection algorithm... The system includes a speech synthesis system and a fast image matching algorithm; the text conversion unit 9 integrates an automatic speech recognition system and a speech enhancement system; the power supply and communication unit 10 contains a rechargeable battery pack and a wireless transmission chip (such as Wi-Fi / Bluetooth) to ensure module connectivity and remote interaction, directly transmitting image information captured by the first front-facing camera module 3 and the second front-facing camera module 4, as well as noise information collected by the microphone, through interfaces such as Type-C and Lightning. This module can also power internal sensors and circuits; the noise control unit 11 includes feedforward and feedback control paths, forming a multi-channel FxLMS active noise reduction system. The noise reduction system generates an inverse signal in real time to cancel noise. Specifically, it can be used to receive ambient noise information through the microphone, generate an anti-noise signal based on the ambient noise information, and play the anti-noise signal through the speaker of the eyeglass frame to suppress the ambient noise information. The signal preprocessing unit 12 includes circuits for signal filtering, signal amplification, and signal feature extraction. It compresses and filters image information and extracts features, amplifies and filters noise information and extracts features, amplifies the anti-noise signal and drives the speaker. The lens 13 uses a transparent OLED or AR display screen to display voice recognition text, system information and notifications.

[0063] In practical applications, the first sound module 1 and the second sound module 2 serve a dual output function: conveying voice information to visually impaired users and releasing actively noise-canceling inverse sound waves into the environment. The modules employ bone conduction speakers or open-back speaker units, ensuring that users do not completely block out ambient noise while listening to the system's output, thus improving safety. The bone conduction speakers contact the temporal bone directly through the frame of the glasses, eliminating the need for in-ear wearing and effectively transmitting voice or scene announcements to the user's inner ear, adapting to users with varying hearing abilities. Simultaneously, the module incorporates a low-frequency enhancement circuit to ensure clear and full output voice and accurate and stable inverse signals.

[0064] In practical applications, the front microphone acquisition module 5, the first rear microphone acquisition module 6, and the second rear microphone acquisition module 7 can all be used to collect noise. The front microphone acquisition module is positioned at the front of the eyeglass frame, while the first and second rear microphone acquisition modules 6 and 7 are positioned near the ear canal. This arrangement, combining the directionality and spatial distribution characteristics of the sound source, enables 360-degree environmental sound perception and near-ear residual noise capture. The microphone units are manufactured using MEMS technology, possessing characteristics such as high sensitivity, wide bandwidth, and low noise. To enhance spatial resolution, this module employs a beamforming algorithm to focus and extract the target sound source, while also supporting sound source localization and multi-source separation, providing a clean signal foundation for subsequent active noise reduction and speech recognition.

[0065] In an exemplary embodiment, the text-to-text unit displays the audio information heard by the glasses microphone in text form on the lenses. Hearing-impaired individuals can quickly understand the meaning of the target speech by "seeing," thus enabling effective communication. Specifically, the text-to-text unit can collect audio signals from the target environment through a microphone array, align the phoneme sequence generated by the visual path with the speech features extracted from the audio path, and jointly model them. Multimodal fusion technology is used to weight the audio signals, enhancing the spatial-semantic features of the target speech. A deep learning speech enhancement network is introduced to extract pure speech. The text is then displayed on the glasses lenses through speech-to-text conversion, taking into account translation of different languages. The microphone collects external sound information, and AR technology displays the speech recognition results on the lenses in real time, enhancing the visually impaired individual's cognitive perception of the real-world scene.

[0066] In an exemplary embodiment, the voice conversion unit can convert the scene information seen by the glasses' visual sensor into speech and play it to the visually impaired person, enabling them to understand the surrounding scene through hearing and react accordingly. Specifically, the voice conversion unit can form a binocular vision function through a first front-facing camera module 3 and a second front-facing camera module 4, which are simultaneously positioned on the left and right sides of the glasses' front end, respectively. A stereo matching algorithm is used to perform pixel-level alignment of the left and right images to obtain a depth map. Combining the intrinsic and extrinsic parameters of the cameras, each pixel in the image can be restored to three-dimensional coordinates, enabling the 3D representation of targets in the visual scene. The 3D objects are compared with a database of real objects to determine their names, and the shape, position, size, color, and other information of the objects are converted into speech and played to the visually impaired person through a speaker, allowing them to understand their surroundings. The system can support user-defined playback frequency, range, and language style (such as concise mode, detailed mode, navigation voice, etc.) to adapt to different user needs.

[0067] In an exemplary embodiment, the noise control unit can acquire speech information and target noise through the front microphone acquisition module 5, and acquire environmental noise and monitor the real-time actual sound level at the ear through the first rear microphone acquisition module 6 and the second rear microphone acquisition module 7. The noise signal is preprocessed by the signal preprocessing module and used as input to the neural network. After training with the speech and noise dataset, the optimal active noise control parameters for different noise types are obtained, stored, and a deep neural network controller parameter database is established. During audiovisual or audio-visual conversion, the speech signal and noise type acquired by the microphone are matched to select the optimal acoustic path to adapt to dynamic environmental changes (such as walking, talking, wind noise, etc.). The generated inverse signal is output through the first sound generation module 1 and the second sound generation module 2, interfering with the noise signal to achieve rapid noise reduction at the ear position. Directional noise reduction enhancement is achieved by combining the speaker's direction with a visual sensor and fusing sound source localization information. This method differs from traditional noise reduction methods based on adaptive filtering; it has low computational complexity, high efficiency, strong robustness, and high noise reduction level.

[0068] In an exemplary embodiment, the lenses of the smart audiovisual assistive glasses can be embedded in the lens structure using a transparent OLED display or a waveguide micro-projection device. The AR display area has high light transmittance and environmental adaptability, enabling high-contrast content presentation under different lighting conditions without interfering with the wearer's normal vision. The system supports functions such as adaptive content position adjustment, font scaling, color inversion, and information occlusion optimization to adapt to different visual conditions and usage scenarios.

[0069] In an exemplary embodiment, the power supply and communication module incorporates a high-density rechargeable lithium polymer battery, featuring small size, long battery life, and fast charging. It can be recharged via a Type-C interface or a wireless charging dock, meeting the needs of extended continuous operation. To enable high-speed data exchange between system modules and interconnection with external devices, the module integrates low-power Bluetooth and dual-band Wi-Fi communication chips, supporting wireless pairing, data synchronization, and remote settings with smart terminals. Furthermore, the module includes a power management unit (PMU) for dynamically regulating the power supply to each submodule, improving overall energy efficiency and thermal management performance.

[0070] In the exemplary embodiment, the signal preprocessing module is located between the microphone input and the intelligent control core. Its main functions include speech enhancement, noise suppression, feature extraction, and data compression. The module embeds multiple acoustic preprocessing algorithms, such as spectral subtraction, Wiener filtering, and minimum mean square error estimation, to perform real-time noise reduction on the input speech signal, improving the signal-to-noise ratio. For image signals, the module simultaneously performs brightness enhancement, edge detection, and target region cropping, providing high-quality input for visual recognition and speech generation. The processed data is encoded and compressed before being transmitted to the central processing unit module to reduce communication latency and resource consumption, thereby improving the overall system real-time performance.

[0071] In practical applications, the smart audiovisual assistive glasses provided in this embodiment can include at least the following three functions:

[0072] (1) Audiovisual conversion function: The glasses capture surrounding visual information in real time through the front camera, extract key scene elements such as obstacles, people, signs, etc. using image recognition technology, and convert the image content into natural language audio output through speech synthesis technology, so that visually impaired people can "hear" the environment and improve their navigation and life independence.

[0073] (2) Audio-visual conversion function: Using a high-sensitivity microphone array to collect voice information around the user, perform real-time voice recognition and text transcription, and display the text on the AR lens to help hearing-impaired people "see" the dialogue content in noisy or unclear environments, thereby enhancing communication efficiency and participation.

[0074] (3) Active noise cancellation function: The system has a built-in multi-channel microphone and speaker, and generates a reverse noise signal in real time through the FxLMS active noise cancellation algorithm with feedforward and feedback structure, which effectively eliminates environmental noise and residual noise in the ear, improves the accuracy of voice calls and command input, and is especially suitable for a clear hearing experience in noisy scenes.

[0075] The intelligent audiovisual assistance method provided by the present invention is described below. The intelligent audiovisual assistance method described below can be referred to in correspondence with the intelligent audiovisual assistance glasses described above.

[0076] Figure 5 This is a flowchart illustrating the intelligent audiovisual assistance method provided in an embodiment of the present invention.

[0077] Figure 6 This is a schematic diagram of the audio-visual conversion process provided in an embodiment of the present invention.

[0078] Figure 7 This is a schematic diagram of the audiovisual conversion process provided in an embodiment of the present invention.

[0079] like Figure 5 As shown, the intelligent audiovisual assistance method provided in this embodiment may include:

[0080] Receive external sound information, and convert the sound information into text and display it;

[0081] And / or,

[0082] It receives external visual information and converts the visual information into speech and plays it.

[0083] In practical applications, such as Figure 6 As shown, when using the intelligent audiovisual assistive glasses provided by this application for audio-visual conversion, the audio signal collected by a multi-channel microphone array is first preprocessed using speech enhancement and sound source localization algorithms to improve the accuracy of speech recognition. Subsequently, the speech content is transcribed in real time using an automatic speech recognition system. After text summarization and noise word filtering, the transcription result is transmitted to the lens display terminal and presented as text overlaid within the user's field of vision. This module supports multilingual recognition and real-time subtitle display, significantly improving the user's information acquisition ability in dialogue scenarios. Furthermore, the system introduces a context-aware and user state modeling mechanism. Based on information such as ambient sound intensity, direction of voice source, and user gaze target, it intelligently determines the current interaction scenario and dynamically adjusts the activation strategy and broadcast content for audio-visual and audiovisual conversion. For example, in traffic scenarios, vehicle and road condition information is prioritized; in dialogue scenarios, the focus is on near-field speech recognition subtitle output, thereby improving the contextual adaptability of the assisted interaction and the user experience.

[0084] like Figure 7 As shown, when providing smart audiovisual assistive glasses for audiovisual conversion using the solution of this application, it may include:

[0085] (1) Image capture and preprocessing

[0086] The visual sensor acquires video streams of the target face and objects at a fixed frequency. The edge computing device performs face detection, object shape detection, key point extraction (such as lip contours and object boundary points), lip shape extraction, and object region extraction. The region is then subjected to temporal normalization and enhancement processing.

[0087] (2) Lip recognition and phoneme estimation

[0088] The lip-shape image sequence is input into a trained visual speech recognition model (such as LipNet or LipFormer), and the corresponding phoneme sequence is output. The model extracts spatial features through a convolutional network and then captures temporal dynamics through a recurrent or Transformer module. The system can also access a local lip-shape-phoneme mapping database to enhance confidence and correct errors in the recognition results, ensuring the accuracy of the output phoneme sequence.

[0089] (3) Audio Acquisition and Speech Analysis

[0090] Raw audio signals from the target environment are acquired using a microphone array, and time-frequency maps are generated based on Short-Time Fourier Transform (STFT) or Mel-Spectrogram. A speech activity detection algorithm is employed to exclude silent and non-speech segments, improving subsequent processing efficiency.

[0091] (4) Multimodal information alignment and fusion

[0092] The phoneme sequence generated by the visual path and the speech features extracted by the audio path are input into the synchronization alignment module. An attention-based alignment method is used to jointly model the visual and audio modalities. The processor infers the speech time period based on the visual phoneme sequence and modally weights the audio signal, thereby enhancing the spatial-semantic features of the target speech and suppressing non-target speech.

[0093] (5) Target speech enhancement and clean speech extraction

[0094] A deep learning-based speech enhancement network is introduced, taking a sequence of target phonemes provided by a mixture of audio signals and visual paths as input. The model output is clean speech stripped of background noise and non-target speakers.

[0095] (6) Real-time scene broadcasting

[0096] It broadcasts key information such as object types, relative spatial positions, and movement trends to help visually impaired users gain real-time scene awareness. The broadcast strategy supports multiple modes, such as summary mode, detailed mode, or user-defined broadcast content.

[0097] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute an intelligent audiovisual assistance method, which includes:

[0098] Receive external sound information, and convert the sound information into text and display it;

[0099] And / or,

[0100] It receives external visual information and converts the visual information into speech and plays it.

[0101] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the intelligent audiovisual assistance method provided by the above methods, the method including:

[0103] Receive external sound information, and convert the sound information into text and display it;

[0104] And / or,

[0105] It receives external visual information and converts the visual information into speech and plays it.

[0106] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the intelligent audiovisual assistance method provided by the methods described above, the method comprising:

[0107] Receive external sound information, and convert the sound information into text and display it;

[0108] And / or,

[0109] It receives external visual information and converts the visual information into speech and plays it.

[0110] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. Intelligent audiovisual-assisted eyeglasses, characterized by the fact that, include: Eyeglasses frame, including microphone and lenses; A text conversion unit is used to receive external sound information through the microphone, convert the sound information into text, and display it on the lens. The voice conversion unit is used to receive external visual information through a visual sensor, and to convert the visual information into voice and play it.

2. The intelligent audio-visual assisted eyewear of claim 1, wherein, It also includes a noise control unit; The noise control unit is used to receive ambient noise information through the microphone and generate an anti-noise signal based on the ambient noise information; The anti-noise signal is played through the speaker of the eyeglasses frame to suppress the environmental noise information.

3. The intelligent audio-visual assisted eyewear of claim 1, wherein, The text conversion unit is also used for: The phoneme sequence of the external sound information is acquired through a visual sensor; Speech features are extracted from the external sound information received by the microphone; The phoneme sequence and the speech features are input-aligned and jointly modeled to obtain the audio signal; The audio signal is used to convert text, and the converted text is then displayed on the lens.

4. The intelligent audio-visual assisted eyewear of claim 1, wherein, The speech conversion unit is also used for: The visual sensors acquire images of the scene in front of the user, and the visual sensors include at least two, which are respectively disposed on both sides of the eyeglass frame; Convert the pixels in the scene image in front of the user into three-dimensional coordinate values ​​to realize the target 3D of the scene in front of the user; The 3D rendering of the scene in front of the user is then broadcast to the user via voice.

5. The intelligent audio-visual assisted eyewear of claim 2, wherein, The noise control unit is also used for: The acoustic path is determined based on the environmental noise signal; The anti-noise signal is played through the speaker of the eyeglass frame based on the acoustic path.

6. The intelligent audio-visual assisted eyewear of claim 5, wherein, The noise control unit is also used for: The environmental noise signal is input into a pre-built noise prediction model to obtain the acoustic path output by the noise prediction model; The noise prediction model is trained using a database of different noise types and their corresponding acoustic paths.

7. Intelligent audiovisual aid method, applied to the intelligent audiovisual aid glasses according to any one of claims 1-6, characterized in that, include: Receive external sound information, and convert the sound information into text and display it; And / or, It receives external visual information and converts the visual information into speech and plays it.

8. The intelligent audiovisual assistance method of claim 7, wherein, Also includes: Receive environmental noise information and generate an anti-noise signal based on the environmental noise information; The anti-noise signal is played to suppress the environmental noise information.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the intelligent audiovisual assistance method as described in claim 7 or 8. 10.A non-transitory computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 9. When the computer program is executed by the processor, it implements the intelligent audiovisual assistance method as described in claim 7 or 8.

Images