An intelligent glasses information processing system based on speech recognition and image interaction

Abstract

The application discloses a kind of intelligent glasses information processing systems based on speech recognition and image interaction, it is related to intelligent glasses and man-machine interaction technical field, the system includes: voice acquisition and analysis module, ultrasonic drive and coding module, ultrasonic phased array array, sound wave receiving array, acoustic signal processing module, space positioning solution module, display drive module and integrated in the display unit of intelligent glasses ontology;The application encodes the semantic information of voice instruction as the coded sound wave field of directional projection, uses the frequency, phase and flight time characteristics of reflected wave to solve the spatial relative position of entity object, realizes target positioning and interaction guidance not dependent on visual image acquisition, effectively avoids the privacy disclosure risk caused by light environment restriction and camera continuous acquisition, realizes acoustic closed-loop confirmation of interaction event by reflected wave characteristic change, and is combined with the lightweight guide mark display matched with positioning result.

Description

Technical Field

[0001] This invention relates to the field of smart glasses and human-computer interaction technology, specifically to a smart glasses information processing system based on voice recognition and image interaction. Background Technology

[0002] Smart glasses are wearable smart terminal devices that integrate sensing, computing, display, and interaction modules. They can overlay virtual information onto the user's real-world field of vision, enabling information interaction and functional assistance in augmented reality scenarios. They are one of the core terminal carriers of the next-generation human-computer interaction system. The smart glasses information processing system is the core hardware and software architecture supporting environmental perception, command parsing, information processing, and interactive output. Its performance directly determines the smart glasses' scene adaptability, interaction smoothness, and usage security, and is a core research and development direction for smart glasses technology iteration. Voice recognition and image interaction are the two most crucial interaction and perception paths for smart glasses. Voice recognition frees the user's hands, enabling contactless command input and semantic parsing, while image interaction enables environmental perception, target recognition, and visual information overlay. The integrated application of these two technologies is a key technological path to achieving natural, all-scenario human-computer interaction in smart glasses. In industrial warehousing scenarios, machine vision technology, as a core supporting technology for image interaction and environmental perception, can quickly complete the identification of building material categories and the automated counting of quantities through the image acquisition unit of smart glasses in the process of counting building materials entering the warehouse. This replaces the tedious process of manual counting, improves the efficiency of warehouse management and the accuracy of data, and is an important application direction of smart glasses image interaction technology in industrial scenarios.

[0003] However, current smart glasses information processing systems based on machine vision and image recognition generally rely on cameras for continuous acquisition of environmental images and target recognition. In practical applications, this has certain technical limitations. The image recognition process is highly sensitive to ambient light conditions; in complex environments with insufficient light, overexposure in strong light, or obstructions, the accuracy of target recognition drops significantly, making stable environmental perception and interactive guidance impossible. Furthermore, continuous image acquisition by cameras poses a serious risk of user privacy leaks, severely limiting the device's use in private, non-public settings. In addition, the indiscriminate superposition of visual information continuously obstructs the user's actual field of vision, easily interfering with the user's fine motor skills and daily walking. Moreover, the interaction method is limited to a single visual channel, lacking a multimodal interactive feedback mechanism, making it difficult to create a natural and smooth wearable interactive experience. Therefore, developing a smart glasses information processing system based on voice recognition and image interaction is of great significance. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide an information processing system for smart glasses based on voice recognition and image interaction. It can solve the technical problems of traditional information processing solutions for smart glasses based on vision recognition, such as being greatly affected by ambient light, having a high risk of privacy leakage, being prone to obstructing the user's field of vision, and having a single interaction dimension. It enables stable, safe, and natural human-computer interaction for smart glasses in all scenarios.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: an information processing system for smart glasses based on voice recognition and image interaction, the system comprising: a voice acquisition and parsing module, an ultrasonic driving and encoding module, an ultrasonic phased array array, a sound wave receiving array, an acoustic signal processing module, a spatial positioning calculation module, a display driving module, and a display unit integrated into the smart glasses body; The output of the voice acquisition and parsing module is connected to the input of the ultrasonic drive and encoding module, acquiring user voice commands and parsing and outputting encoded trigger signals to the ultrasonic drive and encoding module. The output of the ultrasonic drive and encoding module is connected to the ultrasonic phased array, and receives the encoding trigger signal to generate an ultrasonic drive signal that is transmitted to the ultrasonic phased array. The ultrasonic phased array is fixed to the frame of the smart glasses and receives ultrasonic drive signals to project a coded sound wave field to the target area. The output of the acoustic wave receiving array is connected to the input of the acoustic signal processing module, and the reflected acoustic wave field is collected, converted into an electrical signal, and output to the acoustic signal processing module. The output of the acoustic signal processing module is connected to the input of the spatial positioning solution module, preprocesses the electrical signal and extracts feature information, and outputs it to the spatial positioning solution module. The output of the spatial positioning calculation module is connected to the input of the display driver module, and the spatial relative position is calculated and the position calculation result is output to the display driver module. The output of the display driver module is connected to the display unit, and generates a guide marker driving signal that is transmitted to the display unit. The display unit receives a drive signal and overlays a guide mark at the corresponding position in the user's field of vision.

[0006] Furthermore, the voice acquisition and parsing module performs the following operations when processing voice commands: The system collects voice data emitted by the user through a built-in pickup unit, performs frame-by-frame processing on the collected voice data, and generates continuous voice frame data. Environmental noise reduction processing is performed on the generated speech frame data to filter out the environmental background noise components contained in the speech frame data; Acoustic features are extracted from the denoised speech frame data to obtain the Mel frequency cepstral coefficient features corresponding to the speech frame. The extracted acoustic features are input into the pre-trained semantic recognition model, which outputs the text content and interactive keywords corresponding to the voice commands. The identified interactive keywords are semantically categorized, and the corresponding encoded trigger signals are output.

[0007] Furthermore, the semantic recognition model built into the speech acquisition and parsing module adopts an end-to-end convolutional neural network and recurrent neural network fusion architecture. Before deployment, the semantic recognition model undergoes supervised training using a multi-scene speech dataset. This dataset includes speech samples collected in indoor, outdoor, industrial / warehouse, and public scenarios. Each speech sample is labeled with corresponding text content, interactive keywords, and semantic category labels. The semantic recognition model calculates speech feature weights through a self-attention mechanism, using the following formula: ,in, Let be the attention weight for the i-th speech feature. Let be the query vector for the i-th speech feature. Let be the key vector of the i-th speech feature. The query vector representing the j-th speech feature. Let T represent the key vector of the j-th speech feature. The superscript T represents the vector / matrix transpose, a fundamental operation in linear algebra. Where n is the dimension of the key vector, and n is the total number of speech features. The attention weights are determined by iteratively optimizing the parameters of the query vector and key vector through the backpropagation algorithm during supervised training of multi-scene speech datasets. After training, the semantic recognition model is stored in the built-in storage unit of the speech acquisition and parsing module after quantization and compression. During operation, the speech acquisition and parsing module updates the weight parameters of the semantic recognition model online at fixed intervals.

[0008] Furthermore, the ultrasonic driving and encoding module has a built-in non-volatile storage unit, which pre-stores multiple sets of acoustic coding strategies. Each set of acoustic coding strategies establishes a unique mapping relationship with the semantic category of a type of interactive keyword. Each set of acoustic coding strategies includes the carrier frequency parameters, modulation waveform parameters, frequency combination rules, and phase arrangement rules of the corresponding encoded acoustic field. After receiving the encoding trigger signal, the ultrasonic driving and encoding module reads the semantic category information carried by the encoding trigger signal, retrieves the corresponding acoustic coding strategy according to the semantic category information, generates the corresponding timing ultrasonic driving signal according to the parameters in the retrieved acoustic coding strategy, and outputs the generated ultrasonic driving signal to the corresponding array element channel of the ultrasonic phased array according to the preset channel allocation rules.

[0009] Furthermore, the ultrasonic drive and encoding module incorporates a beamforming control unit. When generating the ultrasonic drive signal, the beamforming control unit calculates the phase offset of the drive signal corresponding to each element channel in the ultrasonic phased array based on the spatial orientation of the target area. The phase offset calculation formula is as follows: ,in, Let be the phase offset of the m-th element, d be the center-to-center spacing of the ultrasonic phased array elements, and θ be the horizontal azimuth angle of the target area. Let λ be the vertical pitch angle of the target area, λ be the carrier wavelength of the encoded acoustic field, and m be the array element number. The coefficients in the formula are determined by the hardware layout parameters of the ultrasonic phased array, the carrier frequency of the encoded acoustic field, and the spatial orientation parameters of the target area. The beamforming control unit adjusts the phase of each ultrasonic drive signal according to the calculated phase offset, so that the encoded acoustic field projected by the ultrasonic phased array forms a focused beam in the target area. The beamforming control unit updates the calculation results of the phase offset at fixed time intervals and adjusts the phase parameters of each ultrasonic drive signal accordingly.

[0010] Furthermore, the acoustic signal processing module performs the following operations when processing reflected sound wave signals: The system receives multiple reflected acoustic signals from the acoustic wave receiving array, amplifies each reflected acoustic signal, and adjusts the signal amplitude to a preset sampling range. The amplified multi-channel reflected acoustic signals are processed by analog-to-digital conversion to generate a digital signal sequence corresponding to the number of channels. The generated digital signal sequence is subjected to bandpass filtering to remove noise components outside the carrier frequency range. The filtered digital signal sequence is converted from the time domain to the frequency domain to obtain the frequency domain data corresponding to the reflected sound wave signal. Based on the acquired frequency domain data and time domain digital signal sequence, the frequency characteristics, phase characteristics and time-of-flight information of the reflected sound wave are extracted, and the extracted information is encapsulated and output according to a preset data format.

[0011] Furthermore, the ultrasonic phased array and the acoustic receiving array adopt a co-integrated structure. The integrated structure is fixedly arranged at the left and right ends of the frame of the smart glasses. The ultrasonic phased array includes a multi-element ultrasonic transducer, which is linearly arranged at equal intervals along the horizontal direction of the frame. The acoustic receiving array includes a multi-element MEMS microphone, which is alternately arranged with the ultrasonic transducer along the same arrangement path. Each element channel of the ultrasonic phased array is independently connected to the corresponding output channel of the ultrasonic driving and encoding module, and each element channel of the acoustic receiving array is independently connected to the corresponding input channel of the acoustic signal processing module.

[0012] Furthermore, the spatial positioning calculation module performs the following operations when calculating the spatial relative position: Receive the reflected sound wave characteristic information output by the acoustic signal processing module, and read the flight time data, phase data and frequency data of the multiple reflected sound waves contained in the characteristic information; Based on the flight time data of multi-path reflected sound waves, the flight time difference corresponding to each reflected sound wave is calculated. Combined with the speed of sound wave propagation in the air, the straight-line distance between the physical object and the smart glasses body is calculated. Based on the phase data of multi-path reflected sound waves, the phase difference between different receiving array element channels is calculated. Combined with the array element arrangement parameters of the receiving array, the horizontal azimuth angle and vertical pitch angle of the physical object relative to the smart glasses body are calculated. By combining straight-line distance, horizontal azimuth, and vertical pitch angle, spatial relative position data is generated using a three-dimensional coordinate formula, which is: ,in, Let L be the three-dimensional spatial coordinates of the entity object, L be the straight-line distance between the entity object and the smart glasses, and θ be the horizontal azimuth angle. The vertical pitch angle is determined by the speed of sound in the air and the sampling clock period of the sound wave receiving array. The generated three-dimensional spatial coordinate data is output as the position calculation result.

[0013] Furthermore, the display driving module incorporates a coordinate mapping unit and a marker generation unit. The coordinate mapping unit receives the position calculation results output by the spatial positioning calculation module and converts the spatial relative position data of the entity object with respect to the smart glasses body into pixel coordinate data corresponding to the display coordinate system of the display unit. The marker generation unit reads the converted pixel coordinate data, retrieves pre-stored guide marker vector material, aligns the anchor point coordinates of the guide marker vector material with the converted pixel coordinate data, generates display screen data containing guide markers, and converts the display screen data into a driving signal recognizable by the display unit and outputs it to the display unit. The display unit adopts an optical waveguide display structure, which is attached to the inner surface of the lens of the smart glasses body.

[0014] Furthermore, the acoustic signal processing module has a built-in feature monitoring unit. The feature monitoring unit continuously reads the frequency characteristics and sound pressure amplitude data of the extracted reflected sound waves, compares the real-time read frequency characteristics and sound pressure amplitude data with a preset benchmark feature threshold, and when the comparison result meets the preset interaction triggering conditions, the feature monitoring unit generates an interaction confirmation signal and outputs the generated interaction confirmation signal to the display driving module. After receiving the interaction confirmation signal, the display driving module retrieves the marker material corresponding to the pre-stored interaction confirmation state, updates the display screen data of the guide marker, and generates a driving signal corresponding to the state and outputs it to the display unit.

[0015] Compared with existing technologies, this smart glasses information processing system based on voice recognition and image interaction has the following advantages: This invention encodes the semantic information of voice commands into a directionally projected encoded sound wave field. By utilizing the frequency, phase, and time-of-flight characteristics of the reflected waves, it calculates the spatial relative position of the entity object, achieving target localization and interactive guidance without relying on visual image acquisition. This effectively avoids the privacy risks caused by lighting conditions and continuous camera acquisition. The acoustic closed-loop confirmation of interactive events is achieved through changes in reflected wave characteristics. Combined with lightweight guidance markers that match the localization results, it eliminates the obstruction of the user's field of vision caused by the superposition of full visual information. At the same time, it constructs a multimodal interaction mechanism that integrates vision and acoustics, significantly improving the scene adaptability, interaction naturalness, and usage safety of smart glasses.

[0016] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0018] Figure 1 This is a schematic diagram of the structure of an intelligent glasses information processing system based on voice recognition and image interaction; Figure 2 This is a flowchart of a smart glasses information processing system based on voice recognition and image interaction. Figure 3 This is a flowchart of the voice acquisition and parsing module processing voice commands. Detailed Implementation

[0019] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0020] This invention provides an information processing system for smart glasses based on speech recognition and image interaction. The system uses voice commands as the core interaction interface. A voice acquisition and parsing module collects, frames, reduces noise, and extracts acoustic features from user voice data. A pre-trained fusion neural network model enables semantic recognition and keyword classification, converting semantic information into coded trigger signals that are transmitted to the ultrasonic drive and encoding module. This module, based on a preset acoustic coding strategy and beamforming algorithm, calculates phase shift parameters according to the target's spatial orientation to generate a directional focusing ultrasonic drive signal. This signal drives the ultrasonic phased array on the smart glasses frame to project a coded acoustic wave field towards the target area. The coded acoustic wave field reflects off the target, forming a reflected sound wave. This reflected sound wave is collected by an integrated acoustic receiving array and converted into an electrical signal. The acoustic signal processing module then amplifies, converts the signal to digital, filters, and extracts features, obtaining the frequency, phase, and time-of-flight core features of the reflected wave. Based on the aforementioned feature data, the spatial positioning and calculation module calculates the target's three-dimensional spatial coordinates by combining the sound wave propagation laws and array arrangement parameters. The display driving module maps these spatial coordinates to display pixel coordinates and overlays lightweight guidance markers onto the waveguide display unit of the smart glasses. Simultaneously, closed-loop confirmation of interactive events is achieved through acoustic signal feature monitoring. The entire system abandons the traditional camera image acquisition perception method, using voice-acoustic fusion technology to achieve target positioning and interactive guidance, adapting to all usage scenarios and solving technical pain points such as light constraints, privacy leaks, field-of-view obstruction, and limited interaction. A detailed description is provided below with specific embodiments.

[0021] This embodiment takes the warehousing, searching, and counting of building materials in an industrial warehousing scenario as its application background. The intelligent glasses information processing system is deployed throughout the entire process in the intelligent glasses hardware carrier used for warehousing operations. The system hardware and software modules are arranged as follows: Figure 1 The architecture shown integrates and connects the voice acquisition and analysis module output to the ultrasonic drive and encoding module input, the ultrasonic drive and encoding module output to the ultrasonic phased array, the acoustic wave receiving array output to the acoustic signal processing module input, the acoustic signal processing module output to the spatial positioning calculation module input, the spatial positioning calculation module output to the display drive module input, and the display drive module output to the display unit. The modules form a complete interactive link of unidirectional signal flow and bidirectional feedback.

[0022] The ultrasonic phased array and the acoustic receiving array adopt an integrated structure, which is fixedly installed on the left and right ends of the frame of the smart glasses. The ultrasonic phased array consists of multi-element ultrasonic transducers, which are linearly arranged at equal intervals along the horizontal direction of the frame. The acoustic receiving array consists of multi-element MEMS microphones. The microphones and ultrasonic transducers are arranged alternately along the same path. Each ultrasonic transducer element channel is independently connected to the corresponding output channel of the ultrasonic driving and encoding module, and each MEMS microphone element channel is independently connected to the corresponding input channel of the acoustic signal processing module, ensuring the independence of the sound wave transmission and reception signals and avoiding interference between channels that may affect the positioning accuracy.

[0023] After the system starts up, it enters a real-time standby state and first executes the voice acquisition and parsing process, which strictly follows... Figure 3 The steps are as shown. The voice acquisition and parsing module collects user voice commands through a built-in high-sensitivity pickup unit. In industrial warehousing scenarios, common user voice commands include locating steel, counting cement bags, and positioning plates. The pickup unit collects voice data at a sampling rate of 16kHz. After acquisition, the system performs frame processing on the voice data, with a frame length of 25ms and a frame shift of 10ms, generating continuous voice frame data to ensure complete extraction of voice features. After frame segmentation, environmental noise reduction processing is performed. Warehouse scenarios contain background noise such as forklift operation, fan noise, and personnel conversations. The noise reduction algorithm uses a combination of adaptive filtering and spectral subtraction to filter out environmental background noise components in the voice frame data, retaining the pure user voice signal. After noise reduction processing, the system extracts acoustic features from the voice frame data, obtaining the Mel-frequency cepstral coefficient features corresponding to each voice frame. This feature can accurately characterize the acoustic properties of the voice, providing basic data for subsequent semantic recognition.

[0024] The extracted Mel-frequency cepstral coefficient features are input into a pre-trained semantic recognition model. This model employs an end-to-end convolutional neural network and recurrent neural network fusion architecture, balancing local speech feature extraction and temporal feature analysis capabilities. Before deployment, the model undergoes supervised training using a multi-scenario speech dataset. The dataset includes speech samples from four scenarios: indoor homes, outdoor streets, industrial warehouses, and public transportation. Each speech sample is labeled with corresponding text content, interactive keywords, and semantic category tags. The semantic categories are divided into three types: target search, quantity statistics, and location positioning. In the specific implementation of this embodiment, the semantic recognition model calculates speech feature weights through a self-attention mechanism, accurately focusing on key speech features to improve recognition accuracy. The self-attention weight calculation formula is as follows: , in the formula The attention weight represents the i-th speech feature. The query vector representing the i-th speech feature. The query vector representing the j-th speech feature. Let T represent the key vector of the j-th speech feature. The superscript T represents the vector / matrix transpose, a fundamental operation in linear algebra. The key vector representing the i-th speech feature. The dimension of the key vector is represented by , and 'n' represents the total number of speech features. The attention weights are determined during supervised training on multi-scene speech datasets. The system uses a backpropagation algorithm to iteratively optimize the parameters of the query vector and key vector. The training rounds are set to 100 rounds, with an initial learning rate of 0.001. The learning rate decreases by 0.1 every 10 rounds until the model's recognition accuracy reaches over 98%, at which point training stops. After training, the semantic recognition model undergoes 8-bit quantization compression, reducing its size to one-quarter of the original. It is then stored in the built-in flash memory of the speech acquisition and parsing module. During system operation, the system updates multi-scene speech data via the cloud at a fixed 24-hour cycle, updating the weight parameters of the semantic recognition model online to adapt to the speech habits of different warehousing scenarios.

[0025] After the semantic recognition model outputs the text content and interactive keywords corresponding to the voice command, the system performs semantic category classification on the interactive keywords, classifying search and location keywords into target location category and counting and statistics keywords into quantity statistics category. Subsequently, it outputs the encoded trigger signal corresponding to the semantic category. The encoded trigger signal is transmitted to the ultrasonic drive and encoding module in digital level form to complete the conversion of voice into command.

[0026] After receiving the encoding trigger signal, the ultrasonic drive and encoding module enters the acoustic wave encoding and drive generation process. The module has a built-in non-volatile storage unit that pre-stores three sets of acoustic wave encoding strategies, corresponding to three semantic categories: target localization, quantity statistics, and location localization. Each set of acoustic wave encoding strategies establishes a unique mapping relationship with the semantic category of an interactive keyword. The encoding strategy includes the carrier frequency parameters, modulation waveform parameters, frequency combination rules, and phase arrangement rules of the encoded acoustic wave field. In this embodiment, the carrier frequency corresponding to the target localization category is 40kHz, the modulation waveform is a sine wave, the frequency combination is a single-frequency fixed output, and the phase arrangement is uniform. The carrier frequency corresponding to the quantity statistics category is 45kHz, the modulation waveform is a square wave, the frequency combination is a dual-frequency alternating output, and the phase arrangement is a gradient arrangement. After reading the semantic category information carried by the encoding trigger signal, the module automatically retrieves the matching acoustic wave encoding strategy, generates the corresponding timing ultrasonic drive signal according to the parameters within the strategy, and outputs the drive signal to the corresponding array element channel of the ultrasonic phased array according to the preset channel allocation rules.

[0027] The ultrasonic drive and encoding module incorporates a beamforming control unit, responsible for directional focusing of the encoded acoustic field, thereby improving positioning accuracy. In this specific implementation, the beamforming control unit calculates the phase offset of the drive signal corresponding to each element channel in the ultrasonic phased array based on the spatial orientation of the target area in the industrial warehousing scenario. The phase offset calculation formula is as follows: , in the formula Let d represent the phase offset of the m-th element, d represent the center-to-center spacing of the ultrasonic phased array elements, θ represent the horizontal azimuth angle of the target area, φ represent the vertical elevation angle of the target area, λ represent the carrier wavelength of the encoded acoustic field, and m represent the element number. The coefficients in the formula are determined by the hardware layout parameters of the ultrasonic phased array, the carrier frequency of the encoded acoustic field, and the spatial orientation parameters of the target area. In this embodiment, the center-to-center spacing d of the ultrasonic phased array elements is set to 2 mm, the carrier frequency of the encoded acoustic field is 40 kHz, the corresponding carrier wavelength λ is calculated to be 8.5 mm, the horizontal azimuth angle θ of the target area ranges from -60° to 60°, and the vertical elevation angle... The value range is -30° to 30°, and the array element number m increases sequentially starting from 1. The beamforming control unit collects the target orientation change caused by the user's head rotation in real time at fixed time intervals of 10ms, recalculates the phase offset, and adjusts the phase parameters of each ultrasonic drive signal to ensure that the coded acoustic field is always focused on the target area of ​​interest to the user, avoiding positioning failure due to orientation offset.

[0028] After receiving the phase-adjusted ultrasonic drive signal, the ultrasonic phased array directionally projects an coded acoustic field onto the target area of ​​building materials in the industrial warehousing scenario. The coded acoustic field propagates in the air in the form of ultrasonic waves and is reflected by the surfaces of solid building materials such as steel, cement bags, and boards, forming a reflected acoustic field carrying target location information. The MEMS microphone of the acoustic receiving array synchronously collects the reflected acoustic field, converts the acoustic wave signal into an analog electrical signal, and transmits multiple analog electrical signals synchronously to the acoustic signal processing module to complete the acquisition and conversion of the acoustic signal.

[0029] After receiving multiple reflected acoustic wave analog electrical signals, the acoustic signal processing module initiates the signal preprocessing and feature extraction process. First, each reflected acoustic wave electrical signal is amplified using a programmable gain amplifier to adjust the signal amplitude to a preset sampling range of 0-5V, preventing sampling distortion due to excessively low signal strength or module overload due to excessively high signal strength. After amplification, analog-to-digital conversion is performed using a 12-bit analog-to-digital converter to convert the analog electrical signal into a digital signal sequence, with a sampling rate set to 100kHz to ensure conversion accuracy. Subsequently, the digital signal sequence undergoes bandpass filtering, with the passband range set to 35-50kHz to filter out clutter components outside the carrier frequency range and eliminate interference from other acoustic waves in the environment. After filtering, the digital signal sequence undergoes time-domain and frequency-domain transformation, using a fast Fourier transform algorithm to obtain the frequency domain data corresponding to the reflected acoustic wave signal, separating the signal's time-domain and frequency-domain characteristics.

[0030] In the specific implementation of this embodiment, the acoustic signal processing module extracts the frequency characteristics, phase characteristics, and time-of-flight information of the reflected sound wave based on the acquired frequency domain data and time domain digital signal sequence. The frequency feature extraction accuracy is 1Hz, the phase feature extraction accuracy is 1°, and the time-of-flight extraction accuracy is 1μs. The extracted information is encapsulated in a 32-bit data format and transmitted to the spatial positioning calculation module. Simultaneously, the acoustic signal processing module has a built-in feature monitoring unit that continuously reads the extracted frequency characteristics and sound pressure amplitude data of the reflected sound wave in real time. The real-time data is compared with preset reference feature thresholds. In this embodiment, the reference frequency feature threshold is set to 40kHz±1kHz, and the reference sound pressure amplitude threshold is set to 0.1Pa. When both the real-time frequency and sound pressure amplitude meet the threshold range, it is determined that the interaction trigger condition is met. The feature monitoring unit immediately generates an interaction confirmation signal and transmits the signal to the display driver module, completing the closed-loop confirmation of the acoustic interaction.

[0031] After receiving the reflected sound wave feature information, the spatial positioning calculation module initiates the three-dimensional spatial position calculation process. First, it reads the flight time data, phase data, and frequency data of the multiple reflected sound waves contained in the feature information, removing outliers and retaining valid feature values. Based on the flight time data of the multiple reflected sound waves, it calculates the flight time difference for each reflected sound wave. Combining this with the standard speed of sound in air (340 m / s), it calculates the straight-line distance L between the physical building material and the smart glasses using a distance calculation formula, with the distance calculation accuracy controlled within ±1 cm. Subsequently, based on the phase data of the multiple reflected sound waves, it calculates the phase difference between different receiving array element channels. Combining this with the array element arrangement parameters of the sound wave receiving array, it obtains the horizontal azimuth angle θ and vertical pitch angle of the physical building material relative to the smart glasses using an azimuth angle calculation formula. The azimuth angle calculation accuracy is controlled within ±1°.

[0032] In the specific implementation of this embodiment, the spatial positioning calculation module combines straight-line distance, horizontal azimuth angle, and vertical pitch angle to generate spatial relative position data of the physical building materials using a three-dimensional coordinate formula. The three-dimensional coordinate calculation formula is as follows: , in the formula The coordinates of the entity object are represented by L, where L represents the straight-line distance between the entity object and the smart glasses, and θ represents the horizontal azimuth angle. This represents the vertical pitch angle. The coefficients in the formula are determined by the speed of sound in air and the sampling clock period of the sound wave receiving array. In this embodiment, the speed of sound is 340 m / s, and the sampling clock period of the sound wave receiving array is set to 1 μs. The coefficients are determined by linear fitting, and the fitting function is a linear function to ensure the stability and accuracy of the coordinate calculation. The calculated three-dimensional spatial coordinate data is used as the position calculation result and is output to the display driver module in real time to provide positional basis for visual display.

[0033] After receiving the spatial position calculation results, the display driver module executes the coordinate mapping and guide marker generation process. The module has a built-in coordinate mapping unit and a marker generation unit. The coordinate mapping unit first receives the three-dimensional spatial coordinate data and converts the spatial relative position data of the physical building material with respect to the smart glasses body into pixel coordinate data corresponding to the display unit's display coordinate system. In this embodiment, the display unit adopts an optical waveguide display structure, which is attached to the inner surface of the lens of the smart glasses body. The display resolution is 1920×1080. The coordinate mapping adopts a linear mapping relationship: the spatial X-axis coordinate corresponds to the horizontal pixel coordinate of the display unit, the spatial Y-axis coordinate corresponds to the vertical pixel coordinate of the display unit, and the spatial Z-axis coordinate corresponds to the display size level of the guide marker. The larger the Z-axis value, the larger the marker display size. The mapping function is a continuous linear function with no pixel offset or distortion.

[0034] The marker generation unit reads the converted pixel coordinate data and retrieves pre-stored guide marker vector material. In this embodiment, target positioning markers are highlighted circular boxes, quantity statistics markers are numerical subscripts, and location positioning markers are arrow indicators. The marker generation unit precisely aligns the anchor point coordinates of the guide marker vector material with the converted pixel coordinate data, generates display screen data containing the guide markers, converts the display screen data into LVDS drive signals recognizable by the display unit, and transmits them to the display unit. When the display drive module receives an interaction confirmation signal from the acoustic signal processing module, the marker generation unit immediately retrieves the marker material corresponding to the pre-stored interaction confirmation state, switches the color of the guide marker from blue to green, sets the flashing frequency to 2Hz, updates the display screen data, and outputs the drive signal. The display unit overlays the updated guide markers at the corresponding position in the user's field of vision, completing the visual interaction feedback.

[0035] The entire system's workflow follows Figure 2 The logic loop shown operates in a closed loop, from voice command acquisition and parsing to ultrasonic wave transmission and reception, and then to spatial positioning calculation and visual guidance display. The entire response time is controlled within 50ms, meeting the real-time operation requirements of industrial warehousing scenarios. The modules work together to achieve intelligent interaction without camera dependence.

[0036] In summary, this embodiment achieves a stable and efficient smart glasses interactive application in industrial warehousing and building materials management scenarios by deeply integrating voice recognition technology with ultrasonic acoustic positioning technology. The system relies on voice commands to establish the interactive entry point, freeing users' hands to adapt to warehousing operation scenarios. It achieves target spatial positioning through encoded ultrasonic sound wave fields, completely eliminating the constraints of complex environments such as insufficient light, overexposure in strong light, and obstruction, maintaining a positioning accuracy of over 99% even in dimly lit warehousing environments. The entire process involves no camera image acquisition, fundamentally avoiding the risks of user privacy and warehousing data leakage, meeting the data security requirements of industrial scenarios. Lightweight guide markers are only superimposed at the target location, not obstructing the user's overall field of vision and avoiding interference with delicate warehousing and handling operations. The combination of acoustic feature monitoring for interactive closed-loop confirmation and optical waveguide display for visual feedback constructs a multimodal interactive system of voice, acoustics, and vision, significantly improving the naturalness and fluency of the interaction. Meanwhile, the parameters of each module of the system can be flexibly adjusted according to the application scenario, adapting to multiple scenarios such as indoor, outdoor, public, and industrial use. It has strong scenario adaptability and technical scalability, effectively making up for the technical defects of traditional smart glasses information processing systems and promoting the large-scale application of smart glasses in the field of industrial wearable devices.

[0037] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. An intelligent glasses information processing system based on voice recognition and image interaction, characterized in that, The system includes: a voice acquisition and analysis module, an ultrasonic driving and encoding module, an ultrasonic phased array, a sound wave receiving array, an acoustic signal processing module, a spatial positioning calculation module, a display driving module, and a display unit integrated into the smart glasses body; The output of the voice acquisition and parsing module is connected to the input of the ultrasonic drive and encoding module, acquiring user voice commands and parsing and outputting encoded trigger signals to the ultrasonic drive and encoding module. The output of the ultrasonic drive and encoding module is connected to the ultrasonic phased array, and receives the encoding trigger signal to generate an ultrasonic drive signal that is transmitted to the ultrasonic phased array. The ultrasonic phased array is fixed to the frame of the smart glasses and receives ultrasonic drive signals to project a coded sound wave field to the target area. The output of the acoustic wave receiving array is connected to the input of the acoustic signal processing module, and the reflected acoustic wave field is collected, converted into an electrical signal, and output to the acoustic signal processing module. The output of the acoustic signal processing module is connected to the input of the spatial positioning solution module, preprocesses the electrical signal and extracts feature information, and outputs it to the spatial positioning solution module. The output of the spatial positioning calculation module is connected to the input of the display driver module, and the spatial relative position is calculated and the position calculation result is output to the display driver module. The output of the display driver module is connected to the display unit, and generates a guide marker driving signal that is transmitted to the display unit. The display unit receives a drive signal and overlays a guide mark at the corresponding position in the user's field of vision. 2.The intelligent glasses information processing system based on voice recognition and image interaction of claim 1, wherein, The voice acquisition and parsing module performs the following operations when processing voice commands: The system collects voice data emitted by the user through a built-in pickup unit, performs frame-by-frame processing on the collected voice data, and generates continuous voice frame data. Environmental noise reduction processing is performed on the generated speech frame data to filter out the environmental background noise components contained in the speech frame data; Acoustic features are extracted from the denoised speech frame data to obtain the Mel frequency cepstral coefficient features corresponding to the speech frame. The extracted acoustic features are input into the pre-trained semantic recognition model, which outputs the text content and interactive keywords corresponding to the voice commands. The identified interactive keywords are semantically categorized, and the corresponding encoded trigger signals are output.

3. The intelligent glasses information processing system based on voice recognition and image interaction according to claim 2, characterized in that, The semantic recognition model built into the speech acquisition and parsing module adopts an end-to-end convolutional neural network and recurrent neural network fusion architecture. Before deployment, the semantic recognition model undergoes supervised training using a multi-scene speech dataset. This dataset includes speech samples collected in indoor, outdoor, industrial / warehouse, and public scenarios. Each speech sample is labeled with corresponding text content, interactive keywords, and semantic category labels. The semantic recognition model calculates speech feature weights using a self-attention mechanism, with the following formula: ,in, Let be the attention weight for the i-th speech feature. Let be the query vector for the i-th speech feature. Let be the key vector of the i-th speech feature. The query vector representing the j-th speech feature. Let T represent the key vector of the j-th speech feature. The superscript T represents the vector / matrix transpose, a fundamental operation in linear algebra. Where n is the dimension of the key vector and n is the total number of speech features. After training, the semantic recognition model is stored in the built-in storage unit of the speech acquisition and parsing module after quantization and compression. During operation, the speech acquisition and parsing module updates the weight parameters of the semantic recognition model online at fixed intervals.

4. The intelligent glasses information processing system based on voice recognition and image interaction according to claim 1, characterized in that, The ultrasonic driving and encoding module has a built-in non-volatile storage unit, which pre-stores multiple sets of acoustic coding strategies. Each set of acoustic coding strategies establishes a unique mapping relationship with the semantic category of a type of interactive keyword. Each set of acoustic coding strategies includes the carrier frequency parameters, modulation waveform parameters, frequency combination rules, and phase arrangement rules of the corresponding encoded acoustic field. After receiving the encoding trigger signal, the ultrasonic driving and encoding module reads the semantic category information carried by the encoding trigger signal, retrieves the corresponding acoustic coding strategy according to the semantic category information, generates the corresponding timing ultrasonic driving signal according to the parameters in the retrieved acoustic coding strategy, and outputs the generated ultrasonic driving signal to the corresponding array element channel of the ultrasonic phased array according to the preset channel allocation rules.

5. The intelligent glasses information processing system based on voice recognition and image interaction according to claim 4, characterized in that, The ultrasonic drive and encoding module incorporates a beamforming control unit. When generating the ultrasonic drive signal, the beamforming control unit calculates the phase offset of the drive signal corresponding to each element channel in the ultrasonic phased array based on the spatial orientation of the target area. The phase offset calculation formula is as follows: ,in, Let be the phase offset of the m-th element, d be the center-to-center spacing of the ultrasonic phased array elements, and θ be the horizontal azimuth angle of the target area. Let λ be the vertical pitch angle of the target area, λ be the carrier wavelength of the encoded acoustic field, and m be the array element number. The beamforming control unit adjusts the phase of each ultrasonic drive signal according to the calculated phase offset, so that the encoded acoustic field projected by the ultrasonic phased array forms a focused beam in the target area. The beamforming control unit updates the calculation results of the phase offset at fixed time intervals and adjusts the phase parameters of each ultrasonic drive signal accordingly.

6. The intelligent glasses information processing system based on voice recognition and image interaction according to claim 1, characterized in that, The acoustic signal processing module performs the following operations when processing reflected sound wave signals: The system receives multiple reflected acoustic signals from the acoustic wave receiving array, amplifies each reflected acoustic signal, and adjusts the signal amplitude to a preset sampling range. The amplified multi-channel reflected acoustic signals are processed by analog-to-digital conversion to generate a digital signal sequence corresponding to the number of channels. The generated digital signal sequence is subjected to bandpass filtering to remove noise components outside the carrier frequency range. The filtered digital signal sequence is converted from the time domain to the frequency domain to obtain the frequency domain data corresponding to the reflected sound wave signal. Based on the acquired frequency domain data and time domain digital signal sequence, the frequency characteristics, phase characteristics and time-of-flight information of the reflected sound wave are extracted, and the extracted information is encapsulated and output according to a preset data format.

7. The intelligent glasses information processing system based on voice recognition and image interaction according to claim 1, characterized in that, The ultrasonic phased array and the acoustic receiving array adopt an integrated structure. The integrated structure is fixedly arranged on the left and right ends of the frame of the smart glasses. The ultrasonic phased array includes a multi-element ultrasonic transducer, which is linearly arranged at equal intervals along the horizontal direction of the frame. The acoustic receiving array includes a multi-element MEMS microphone, which is alternately arranged with the ultrasonic transducer along the same arrangement path. Each element channel of the ultrasonic phased array is independently connected to the corresponding output channel of the ultrasonic driving and encoding module. Each element channel of the acoustic receiving array is independently connected to the corresponding input channel of the acoustic signal processing module.

8. The intelligent glasses information processing system based on voice recognition and image interaction according to claim 1, characterized in that, The spatial positioning calculation module performs the following operations when calculating spatial relative positions: Receive the reflected sound wave characteristic information output by the acoustic signal processing module, and read the flight time data, phase data and frequency data of the multiple reflected sound waves contained in the characteristic information; Based on the flight time data of multi-path reflected sound waves, the flight time difference corresponding to each reflected sound wave is calculated. Combined with the speed of sound wave propagation in the air, the straight-line distance between the physical object and the smart glasses body is calculated. Based on the phase data of multi-path reflected sound waves, the phase difference between different receiving array element channels is calculated. Combined with the array element arrangement parameters of the receiving array, the horizontal azimuth angle and vertical pitch angle of the physical object relative to the smart glasses body are calculated. By combining straight-line distance, horizontal azimuth, and vertical pitch angle, spatial relative position data is generated using a three-dimensional coordinate formula, which is: ,in, Let L be the three-dimensional spatial coordinates of the entity object, L be the straight-line distance between the entity object and the smart glasses, and θ be the horizontal azimuth angle. The vertical pitch angle is used to output the generated three-dimensional spatial coordinate data as the position calculation result.

9. The intelligent glasses information processing system based on voice recognition and image interaction according to claim 1, characterized in that, The display driving module has a built-in coordinate mapping unit and a marker generation unit. The coordinate mapping unit receives the position calculation result output by the spatial positioning calculation module and converts the spatial relative position data of the entity object with respect to the smart glasses body into pixel coordinate data corresponding to the display coordinate system of the display unit. The marker generation unit reads the converted pixel coordinate data, retrieves the pre-stored guide marker vector material, aligns the anchor point coordinates of the guide marker vector material with the converted pixel coordinate data, generates display screen data containing guide markers, and converts the display screen data into a driving signal that can be recognized by the display unit and outputs it to the display unit. The display unit adopts an optical waveguide display structure, which is attached to the inner surface of the lens of the smart glasses body.

10. The intelligent glasses information processing system based on voice recognition and image interaction according to claim 1, characterized in that, The acoustic signal processing module has a built-in feature monitoring unit. The feature monitoring unit continuously reads the frequency characteristics and sound pressure amplitude data of the extracted reflected sound waves, compares the real-time read frequency characteristics and sound pressure amplitude data with a preset benchmark feature threshold, and when the comparison result meets the preset interaction triggering conditions, the feature monitoring unit generates an interaction confirmation signal and outputs the generated interaction confirmation signal to the display driving module. After receiving the interaction confirmation signal, the display driving module retrieves the marker material corresponding to the pre-stored interaction confirmation state, updates the display screen data of the guide marker, and generates a driving signal for the corresponding state and outputs it to the display unit.

Images