Intelligent glasses voice interaction method and system based on large language model

Abstract

The application relates to the technical field of wearable smart devices, smart voice and natural language processing, and discloses a smart glasses voice interaction method and system based on a large language model. The method freezes to-be-continued-broadcast data when a detachable smart foot cover is disconnected, generates detection subframes with the same amplitude spectrum and different peak-to-average ratios from to-be-continued-broadcast voice data after reconnection, synchronously collects air sound responses and structural vibration responses, removes linear components and loudspeaker homologous nonlinear components, and forms a current interface nonlinear fingerprint; the abnormal intermodulation frequency points are reversely determined to determine a parent frequency band combination, the to-be-continued-broadcast voice is divided into a coincidence domain and a migratable domain, text coincidence and cross-assembly linear compensation are respectively performed, and the to-be-continued-broadcast voice is fused to form continued-broadcast voice. The application can improve voice adaptability and continued-broadcast continuity after reassembly.

Description

Technical Field

[0001] This invention relates to the fields of wearable smart devices, artificial intelligence-generated content, natural language processing, intelligent voice, human-computer interaction, speech analysis and synthesis, and electronic digital data processing, and particularly to a voice interaction method and system for smart glasses based on a large language model. Background Technology

[0002] With the development of artificial intelligence voice processing technology, smart glasses are gradually acquiring voice acquisition, wireless communication, natural language processing, and voice playback functions, enabling them to provide users with question-and-answer, translation, and information prompt services via voice. For example, patent document CN119002054A (its family publication number WO2024235182A1), entitled "Smart Glasses, System, and Control Method Based on Generative Artificial Intelligence Large Language Model," discloses a smart glasses comprising a frame, temples, microphone, speaker, processor, and memory. It can acquire user voice, obtain responses through a generative artificial intelligence large language model, convert the responses into speech, and play them through a speaker. This document also discloses a technical solution for transmitting voice, text, and model response data via Bluetooth components, smart mobile terminals, or model servers.

[0003] Existing smart glasses typically have fixed temple structures and sound propagation conditions for setting voice playback parameters. When the detachable components carrying the sound output, voice acquisition, and processing components are removed from the original temple and reconnected to another temple or reassembled back into the original temple, the connection status, contact level, and sound propagation conditions may change. If the large language model response continues to be output based on the voice data and sound processing parameters before disassembly, the actual playback effect may be inconsistent with the original playback state. If all the unplayed response content is re-synthesized, it will increase the processing load and playback waiting time. Therefore, how to balance the continuous output of the large language model response and voice playback adaptability after reconnection of the detachable smart temples becomes a problem that needs to be solved. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a voice interaction method and system for smart glasses based on a large language model. It aims to solve the technical problem that after the detachable smart earpiece separates and reconnects during the large language model's response to voice output, the original cached voice no longer perfectly matches the sound propagation characteristics of the current connection state. This leads to potential distortion when directly resuming the cached voice, while completely re-performing text-to-speech synthesis increases processing load and playback latency. Furthermore, this invention also addresses the problem that structural vibration responses simultaneously contain nonlinear components from the sound output component itself and the detachable connection interface, making it difficult to accurately distinguish abnormal responses caused by changes in the connection interface; and the lack of a clear correspondence between abnormal intermodulation frequency points and the actual affected audio bands to be resumed, making it difficult to determine the specific range of cached voice that needs to be resynthesized.

[0005] The present invention specifically adopts the following scheme: a voice interaction method for smart glasses based on a large language model, including a response caching step, a migration freezing step, a phase detection step, an interface extraction step, and a migration resumption step. Wherein: The response caching step involves: acquiring the large language model response text, caching the text to be played, phoneme timing, and synthesized speech spectrum by phoneme, and saving the pre-separation airborne acoustic linear model, structural vibration linear model, loudspeaker homology mapping, and interface nonlinear baseline. Specifically, the response caching step establishes the semantic data, timing data, speech data, and reference model required for subsequent assembly migration before the sound output component begins outputting the large language model response. After receiving the large language model response text, the wireless communication component first converts punctuation marks, numbers, English abbreviations, and special symbols into directly readable standard text, and then converts the standard text into continuous phonemes according to the corresponding phoneme conversion rules. For Chinese text, characters can be converted into phoneme sequences containing initials, finals, and tones; for foreign language text, the corresponding language's phoneme conversion model can be called to generate phoneme sequences. Each phoneme records its start time, end time, and duration, thus forming a phoneme timing sequence. The local processing component generates synthesized speech based on the phoneme sequence and timing sequence, and performs a short-time Fourier transform on the synthesized speech to form a synthesized speech spectrum arranged by time frame and frequency band. The text to be played, the phoneme timing sequence, and the synthesized speech spectrum are indexed according to the same response sequence number and phoneme sequence number, so that the corresponding text and speech spectrum can be obtained later based on any phoneme position.

[0006] The pre-separation airborne acoustic linear model represents the linear propagation relationship of synthesized speech output by the sound output component through the air to the speech acquisition component. The structural vibration linear model represents the linear propagation relationship of synthesized speech output by the sound output component through the detachable smart earpiece end, connection position, and temple to the inertial detection component. These two linear models can be calculated using the synthesized speech spectrum, airborne acoustic response, and structural vibration response when the detachable smart earpiece end is properly connected and the wearing posture is stable. To reduce the influence of random noise, the spectra corresponding to multiple consecutive sound elements can be averaged, and the model coefficients for the corresponding frequency band can be stopped from being updated when the input frequency band energy is lower than the input energy threshold.

[0007] The speaker source mapping relationship is used to describe the correspondence between the nonlinear distortion generated by the sound output component itself in the airborne acoustic channel and the structural vibration channel. When establishing the speaker source mapping relationship, calibration speech can be output with the detachable smart temple unconnected to the temple, and the airborne acoustic response and structural vibration response can be acquired simultaneously. At this time, there is no interface nonlinear component in the structural vibration response propagating through the temple connection interface. Therefore, nonlinear components with the same frequency, order, and stable phase relationship in both the airborne acoustic channel and the structural vibration channel can be used to establish the speaker source mapping relationship. The interface nonlinear baseline is established when the detachable smart temple is stably connected to the corresponding temple, and is used to record the intermodulation amplitude, intermodulation phase, and peak-to-average power ratio (PAPR) response slope at each intermodulation frequency point under normal assembly conditions.

[0008] Migration Freeze Step: When the detachable smart foot cover is detected to be separated during voice output, the data to be resumed is frozen; specifically, the migration freeze step is performed when the assembly state detection component outputs a separation state signal. The local processing component first determines the phoneme currently being output by the sound output component and identifies the last complete phoneme that has been output as the resume boundary. Phonemes that are in the output process but have not yet been fully output can be reincorporated into the data to be resumed, and output can be restarted from that phoneme to avoid discontinuous sound caused by resuming playback from the middle of a phoneme. The freeze process includes stopping sound output, stopping the deletion of already played buffered data, and setting the text to be resumed, phoneme timing, and synthesized speech spectrum after the resume boundary to an un-overwriteable state.

[0009] Phase detection step: After reconnection, a first detection subframe and a second detection subframe with the same amplitude spectrum and average energy but different sub-band phase and peak-to-average power ratio are generated from the first voice element to be played. The preset intermodulation frequency point falls into the low-energy frequency band of the two detection subframes. The two detection subframes are continuously output, and the airborne sound response and structural vibration response are simultaneously acquired. Specifically, the phase detection step is performed after the assembly status detection component outputs the connection status signal. The local processing component selects at least two sub-bands with effective energy from the first voice element after the replay boundary. The sum or difference frequency points corresponding to the two sub-bands should be located in the frequency band with lower energy of the original voice element, so that the intermodulation components generated by the interface are not easily masked by the original speech spectrum. The local processing component adjusts the sub-band phase while keeping the amplitude spectrum and average energy unchanged, generating the first and second detection subframes. The two detection subframes have different peak-to-average power ratios, thus allowing them to act on the connection interface with different instantaneous excitation intensities.

[0010] Interface extraction steps: Based on the two linear models, linear components of the two types of responses are removed to form a cross-assembly linear compensation relationship. The structural vibration nonlinear residual is corrected using the airborne acoustic nonlinear residual based on the speaker homogeneous mapping relationship. The current interface nonlinear fingerprint is formed based on the change in peak-to-average power ratio (PAPR) of the corrected intermodulation components. Specifically, in the interface extraction step, the local processing component uses the pre-separation airborne acoustic linear model and the structural vibration linear model to remove components that can be explained by linear propagation from the airborne acoustic response and structural vibration response, respectively, to obtain the airborne acoustic nonlinear residual and the structural vibration nonlinear residual. The local processing component then maps the airborne acoustic nonlinear residual to the structural vibration channel based on the speaker homogeneous mapping relationship to obtain the homogeneous nonlinear components generated by the sound output component itself, and subtracts the homogeneous nonlinear components from the structural vibration nonlinear residual, thereby obtaining the interface nonlinear residual. The interface fingerprint module calculates the change in intermodulation components with PAPR based on the interface nonlinear residuals corresponding to the first and second detection subframes, forming the current interface nonlinear fingerprint.

[0011] The migration and replay process involves the following steps: Based on the abnormal intermodulation frequency points of the interface nonlinear fingerprint relative to the interface nonlinear baseline, the mother band combination is determined in reverse. The time-frequency units involved in the mother band combination are assigned to the re-synthesis domain, and the rest are assigned to the transferable domain. Text re-synthesis and cross-assembly linear compensation are then performed to form the replay speech. Specifically, in the migration and replay process, the interface fingerprint module compares the current interface nonlinear fingerprint with the interface nonlinear baseline. Intermodulation frequency points exceeding an abnormal threshold are identified as abnormal intermodulation frequency points. The mother band backtracking module calculates the mother band combination that can generate the abnormal intermodulation frequency point based on its frequency value and intermodulation order. The migration partitioning module assigns the time-frequency units containing the corresponding mother band energy in the replay speech to the re-synthesis domain, and the remaining time-frequency units to the replay domain. The local synthesis module regenerates the phoneme speech based on the text to be continued and the phoneme timing corresponding to the resynthesis domain. The local processing component processes the cached speech corresponding to the transferable domain using the cross-assembly linear compensation relationship. The continued playback fusion module merges the two parts of speech according to the original phoneme timing to form the continued playback speech.

[0012] In a further embodiment of the present invention, the interface extraction step further includes the following sub-steps: The separation calibration sub-step involves the following steps: When the detachable smart temple tip is separated from the temple, calibration voice is output through the detachable smart temple tip, simultaneously acquiring the airborne sound response and structural vibration response in the separated state. Specifically, this separation calibration sub-step is performed when the detachable smart temple tip is not connected to the temple. The detachable smart temple tip can be placed on a soft supporting surface to avoid external rigid objects forming additional structural vibration paths. The sound output component outputs calibration voice, and the voice acquisition component and inertial detection component simultaneously acquire the airborne sound response and structural vibration response in the separated state under the control of a common time-scale component. The calibration voice can be normal speech or a multi-tone speech segment with the same audio frequency as normal speech.

[0013] The mapping establishment sub-step involves extracting harmonic components and intermodulation components of the same order from the separated state airborne acoustic response and the separated state structural vibration response, respectively. The loudspeaker source mapping relationship is then established based on the amplitude ratio and phase difference between the corresponding components. Specifically, in this sub-step, the local processing component extracts harmonic components and intermodulation components of the same order from the separated state airborne acoustic response and the separated state structural vibration response, respectively. If a nonlinear frequency component in the airborne acoustic channel has a corresponding component in the structural vibration channel, the amplitude ratio and phase difference between the two are calculated. After repeated calculations for multiple calibration voice samples, the amplitude ratio and phase difference at the same frequency position can be averaged to form the loudspeaker source mapping relationship. The loudspeaker source mapping relationship can be stored as complex mapping coefficients arranged by frequency.

[0014] Assembly verification sub-step: After reconnection at the detachable smart foot sleeve end, identify the homologous components satisfying the amplitude ratio and phase difference from the airborne acoustic nonlinear residual and the structural vibration nonlinear residual. Only use the homologous components satisfying the loudspeaker homologous mapping relationship to perform the homologous correction sub-step; specifically, the assembly verification sub-step is performed after reconnection at the detachable smart foot sleeve end. After the nonlinear components in the airborne acoustic nonlinear residual are mapped to the structural vibration channel according to the loudspeaker homologous mapping relationship, they should be matched with the corresponding components in the structural vibration nonlinear residual. Only components whose frequency, amplitude ratio, and phase difference all satisfy the homologous condition are subtracted from the structural vibration nonlinear residual. For structural vibration nonlinear components generated due to changes in the connection interface but for which there is no corresponding component in the airborne acoustic channel, they should be retained in the interface nonlinear residual.

[0015] Polarity configuration sub-step: Generate positive and negative polarity detection frames with the same amplitude spectrum and opposite overall polarity from the first or second detection sub-frame; specifically, the polarity configuration sub-step is used to further distinguish between even-order and odd-order nonlinear components in the interface nonlinear residual. The positive and negative polarity detection frames have the same amplitude spectrum, and the time-domain sample value of the negative polarity detection frame is the opposite of the time-domain sample value of the positive polarity detection frame. The two detection frames should maintain the same duration and average energy, and be set within the same original phoneme duration.

[0016] Polarity output sub-step: While maintaining the duration of the original phoneme corresponding to the sound element, output the positive polarity detection frame and the negative polarity detection frame, and respectively acquire the corresponding airborne sound response and structural vibration response; specifically, in the polarity output sub-step, the sound output component outputs the positive polarity detection frame and the negative polarity detection frame sequentially. A brief phase transition interval is set between the two detection frames, and the voice acquisition component and the inertial detection component respectively acquire the corresponding airborne sound response and structural vibration response. The local processing component obtains the nonlinear residuals of the positive polarity detection frame interface and the negative polarity detection frame interface according to the aforementioned linear elimination and homogeneous correction process. If the system response can be decomposed into linear terms, even-order nonlinear terms, and odd-order nonlinear terms, then the even-order nonlinear terms of the positive polarity input and the negative polarity input have the same direction, and the odd-order nonlinear terms have opposite directions. Therefore, the even-order interface nonlinear components and the odd-order interface nonlinear components can be determined according to the following formula: ; ; Even-order interface nonlinear components Represents frequency The components formed by even-order interface nonlinearity; the odd-order interface nonlinearity components. Represents frequency The components formed by the odd-order interface nonlinearity; positive polarity interface nonlinear residuals This represents the interface nonlinear residual corresponding to the positive polarity detection frame; the negative polarity interface nonlinear residual... This represents the interface nonlinear residual corresponding to the negative polarity detection frame.

[0017] Order separation sub-step: Perform summation and difference processing on the interface nonlinear residuals corresponding to the positive polarity detection frame and the negative polarity detection frame to extract the even-order interface nonlinear components and the odd-order interface nonlinear components respectively; The fingerprint verification sub-step involves verifying the current interface nonlinear fingerprint based on the even-order and odd-order interface nonlinear components, and deleting abnormal intermodulation frequency points that fail the order consistency verification. Specifically, in this sub-step, the interface fingerprint module determines whether the corresponding abnormal component should belong to an even-order or odd-order component based on the intermodulation order of the abnormal intermodulation frequency point. Second-order sum frequencies and second-order difference frequencies should primarily appear in even-order interface nonlinear components, while third-order intermodulation components should primarily appear in odd-order interface nonlinear components. If the actual order distribution of the abnormal intermodulation frequency point is inconsistent with the theoretical order, the abnormal intermodulation frequency point is deleted, or a new detection is performed to reduce the probability of noise frequencies being misidentified as interface nonlinear frequencies.

[0018] In a further embodiment of the present invention, the migration and replay step further includes the following sub-steps: Anomaly identification sub-step: The intermodulation increment, intermodulation phase, and peak-to-average ratio (PARR) response slope of the current interface nonlinear fingerprint and the interface nonlinear baseline are compared at each intermodulation frequency point. Any intermodulation frequency point whose comparison result reaches the corresponding anomaly threshold is identified as the anomalous intermodulation frequency point. Specifically, the anomaly identification sub-step compares the current interface nonlinear fingerprint with the interface nonlinear baseline frequency by frequency point. The comparison includes the intermodulation increment, intermodulation phase difference, and PARR response slope difference. The anomaly threshold can be determined based on the average value and standard deviation of the interface nonlinear baseline. If any comparison result reaches the corresponding anomaly threshold, the corresponding intermodulation frequency point is identified as the anomalous intermodulation frequency point. To avoid misjudgment due to fluctuations in a single measurement, the same intermodulation frequency point can be required to meet the anomaly conditions in two consecutive detections, or at least two of the intermodulation increment, intermodulation phase difference, and PARR response slope difference can be required to simultaneously meet the anomaly conditions.

[0019] The mother frequency backtracking sub-step involves determining, in reverse, at least one mother frequency band combination capable of generating the corresponding abnormal intermodulation frequency point based on the frequency value and intermodulation order of each abnormal intermodulation frequency point. Specifically, in the mother frequency backtracking sub-step, the mother frequency backtracking module establishes candidate mother frequency relationships based on the abnormal intermodulation frequency point and intermodulation order. For second-order sum frequencies, the sum of the first and second mother frequencies equals the abnormal intermodulation frequency; for second-order difference frequencies, the absolute value of the difference between the first and second mother frequencies equals the abnormal intermodulation frequency; for third-order intermodulation, the abnormal intermodulation frequency is the difference between the two first mother frequencies and the second mother frequency, or the difference between the two second mother frequencies and the first mother frequency. Candidate mother frequencies must also be located within the effective frequency band of the speech to be played. If multiple candidate mother frequency band combinations exist, the combination with effective energy in both the first and second detection subframes is preferentially retained.

[0020] Unit marking sub-step: Mark the time-frequency units to be re-enhanced as time-frequency units whose spectral energy is located within any of the aforementioned mother frequency band combinations, and mark the remaining time-frequency units to be re-enhanced as time-frequency units to be compensated; specifically, in the unit marking sub-step, the migration partitioning module divides the time-frequency units to be re-enhanced according to the phoneme timing sequence and the preset frequency band boundary. The preset frequency band width can be set to 100Hz to 500Hz, and the time length can be the same as the analysis window length of the short-time Fourier transform. If the center frequency or main spectral energy of a time-frequency unit to be re-enhanced is located within any abnormal mother frequency band combination, then the time-frequency unit to be re-enhanced is marked as a time-frequency unit to be re-enhanced; the remaining time-frequency units to be re-enhanced are marked as time-frequency units to be compensated.

[0021] The region generation sub-step involves generating the recombining domain based on the time-continuous and frequency-adjacent time-frequency units to be recombined, and generating the transferable domain based on the time-frequency units to be compensated. Specifically, in the region generation sub-step, time-continuous and frequency-adjacent time-frequency units to be recombined are merged into a recombining domain. The time-frequency units to be compensated form a transferable domain. To avoid overly fragmented recombining domain boundaries, a neighborhood expansion can be performed on the recombining domain, including a time-frequency unit adjacent to the recombining domain with an energy higher than the neighborhood energy threshold.

[0022] The interval mapping sub-step: Based on the phoneme timing, the resynthesis domain is mapped to the smallest continuous phoneme interval covering the resynthesis domain; specifically, the interval mapping sub-step is used to convert the spectral-level resynthesis domain into a phoneme interval that can be directly processed by the text-to-speech process. The local processing component determines which phonemes the resynthesis domain covers based on the phoneme timing. If the resynthesis domain only covers a part of a certain phoneme, to avoid re-concatenating speech generated by two different synthesis processes within the same phoneme, the resynthesis domain can be extended to cover the full duration of that phoneme. If the resynthesis domain spans multiple phonemes, the smallest continuous phoneme interval that can completely cover the resynthesis domain is selected.

[0023] Boundary extraction sub-step: Extracting the boundary fundamental frequency, boundary spectral envelope, boundary phase, and boundary energy from the transferable domain before and after the minimum continuous phoneme interval; specifically, in the boundary extraction sub-step, the local processing component extracts the boundary fundamental frequency, boundary spectral envelope, boundary phase, and boundary energy from the transferable domain before and after the minimum continuous phoneme interval. The preceding boundary parameter can be the average parameter within the last 5ms to 20ms of the preceding phoneme, and the following boundary parameter can be the average parameter within the first 5ms to 20ms of the following phoneme. If the minimum continuous phoneme interval is located at the beginning of the speech to be continued, the process is regenerated using the following boundary parameter and the original phoneme acoustic features as constraints; if it is located at the end of the speech to be continued, the preceding boundary parameter and the original phoneme acoustic features are used for constraints.

[0024] The local synthesis sub-step involves regenerating the phoneme speech based on the text to be played and the original phoneme timing corresponding to the minimum continuous phoneme interval, and then using the boundary fundamental frequency, the boundary spectral envelope, the boundary phase, and the boundary energy constraints to regenerate the phoneme speech. Specifically, in the local synthesis sub-step, the local synthesis module reads the text to be played, the original phoneme order, and the original phoneme duration corresponding to the minimum continuous phoneme interval, and inputs the boundary fundamental frequency, boundary spectral envelope, boundary phase, and boundary energy as additional conditions into the speech synthesis model. The generated result should maintain the original text content and the original phoneme order unchanged, and the deviation of each phoneme duration from the original phoneme duration can be limited to within 5%.

[0025] The continuous fusion sub-step involves performing phase-continuous fusion between the regenerated phoneme speech and the transferable domain speech processed according to the cross-assembly linear compensation relationship to form the continued playback speech. Specifically, in the continuous fusion sub-step, an overlap interval is set at the boundary between the regenerated phoneme speech and the transferable domain speech processed by cross-assembly linear compensation. Within the overlap interval, a cosine window can be used for weighted fusion, gradually decreasing the weight of the preceding speech component and gradually increasing the weight of the following speech component. After fusion, the phase jump variable and energy jump variable on both sides of the boundary are checked. If the phase jump variable or energy jump variable exceeds the boundary threshold, the overlap interval length is increased and fusion is performed again.

[0026] Another solution: a smart glasses voice interaction system based on a large language model, including: The frame includes lenses; temples connected to the frame; detachable smart temple tips detachably connected to the ends of the temples; an assembly status detection component located within the detachable smart temple tips for outputting separation and connection status signals; a voice acquisition component located within the detachable smart temple tips for acquiring airborne sound response; an inertial detection component located within the detachable smart temple tips and adjacent to the connection point for acquiring structural vibration response; a sound output component located within the detachable smart temple tips for outputting synthesized speech; a wireless communication component located within the detachable smart temple tips for acquiring large language model response text; and a data storage component located within the detachable smart temple tips for storing text to be played, phoneme timing, synthesized speech spectrum, airborne sound linear model, structural vibration linear model, speaker homology mapping relationship, and interface nonlinear baseline; and more. A local processing component, located within the detachable smart foot sleeve, is connected to each of the aforementioned components. After reconnection, it generates two detection subframes with the same amplitude spectrum but different peak-to-average ratios from the audio elements to be played. Based on the two types of responses and the speaker homology mapping relationship, it extracts the interface nonlinear fingerprint, reversely determines the mother band combination based on the abnormal intermodulation frequency points, performs text resynthesis on the time-frequency units to be played involving the mother band combination, and performs buffered spectrum compensation on the remaining time-frequency units to be played.

[0027] The beneficial effects of this invention are: By pre-saving the text to be played back, phoneme timing, synthesized speech spectrum, pre-separation airborne acoustic linear model, structural vibration linear model, speaker homology mapping, and interface nonlinear baseline through a response caching step, the semantic content of the large language model response, playback order, and sound propagation state before detachment can be preserved when the detachable smart foot sleeve is separated, providing a complete data foundation for speech recovery after reconnection. The migration freeze step freezes the unoutput data to be played back based on the phoneme timing, avoiding discontinuities in speech boundaries caused by direct interruption or resumption of playback in the middle of phonemes.

[0028] After reconnection, the phase detection step generates a first detection subframe and a second detection subframe from the first voice element to be played, which have the same amplitude spectrum and average energy but different sub-band phase and peak-to-average power ratio. Since the two detection subframes are derived from normal voice to be played, the detection process can be combined with the large language model response playback process without the need to insert additional dedicated detection sounds unrelated to the response content; the amplitude spectrum and average energy remain consistent, and the main difference between the two detection subframes is concentrated on the instantaneous excitation level, providing stable conditions for judging the nonlinear response of the connection interface to different instantaneous excitations.

[0029] The interface extraction step utilizes the pre-separation airborne acoustic linear model and structural vibration linear model to remove the linear propagation components from the airborne acoustic response and structural vibration response, respectively, to obtain the airborne acoustic nonlinear residual and the structural vibration nonlinear residual. Subsequently, using the speaker homologous mapping relationship, the portion of the airborne acoustic nonlinear residual reflecting the distortion of the sound output component itself is mapped to the homologous nonlinear component in the structural vibration channel and subtracted from the structural vibration nonlinear residual. This makes the remaining interface nonlinear residual more concentrated in reflecting the nonlinear effect of the connection interface between the detachable smart temple and the temple, reducing the interference of the sound output component's own distortion on the connection status judgment.

[0030] The interface fingerprint module generates a current interface nonlinear fingerprint based on the changes in intermodulation components and peak-to-average power ratio (PAPR) corresponding to the first and second detection subframes, and compares the current interface nonlinear fingerprint with the interface nonlinear baseline. This processing does not merely determine whether the sound amplitude has changed, but rather uses the law of intermodulation component changes with the instantaneous excitation level to characterize the current connection interface, making it easier to identify abnormal nonlinear responses caused by loose connections, changes in contact state, or changes in local damping.

[0031] After identifying anomalous intermodulation frequency points, the mother frequency backtracking module reversely determines the mother frequency band combination based on the frequency value and intermodulation order of the anomalous intermodulation frequency points, so that the detected anomalous intermodulation results can be mapped to the original speech frequency band that generated the anomalous intermodulation. The migration partitioning module then assigns the time-frequency units to be re-encoded involving the mother frequency band combination to the resynthesis domain, and assigns the remaining time-frequency units to the migrateable domain, thereby avoiding the erroneous direct use of the anomalous intermodulation frequency points themselves as the speech resynthesis range.

[0032] For the resynthesis domain, the local synthesis module regenerates the phoneme speech based on the corresponding text to be resumed and the original phoneme timing. For the transferable domain, the local processing component processes the cached speech according to the cross-assembly linear compensation relationship. The resume fusion module then continuously fuses the regenerated phoneme speech with the compensated cached speech according to the original phoneme timing. This allows for the resynthesis of parts affected by the current connection interface anomaly, while preserving and compensating for still usable cached speech, reducing the computational load and resume playback waiting time caused by complete resynthesis, and minimizing playback distortion caused by directly reusing all cached speech.

[0033] The assembly status detection component, voice acquisition component, inertial detection component, common time-stamp component, and assembly identification component work together to establish data correlation between assembly status, airborne acoustic response, structural vibration response, and corresponding temple. The common time-stamp component provides a unified time reference for the detection subframe, airborne acoustic response, and structural vibration response, while the assembly identification component enables the interface nonlinear baseline to correspond to a specific temple, thereby improving the accuracy of the correspondence between detection results and reference data when migrating between different temples.

[0034] Through the above processing, the present invention can identify the nonlinear changes generated by the current connection interface after the detachable smart foot cover is reassembled, and further use the identification results to determine the range of cached speech compensation and the range of text resynthesis. While ensuring the continuity of the semantic and phoneme timing of the large language model response, it improves the adaptability of the continued playback speech to the current assembly state. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the structure of the smart glasses of the present invention.

[0036] Figure 2 This is a signal flowchart illustrating the execution steps of the method of the present invention.

[0037] Figure 3 This is a schematic diagram of the overall system structure of the present invention.

[0038] Figure 4 This is a flowchart of the current interface nonlinear fingerprint formation process of the present invention.

[0039] Figure 5 This is a flowchart of the migration partition and resumed voice generation process of the present invention.

[0040] The reference numerals in the figures include: 1—Lens, 2—Frame, 21—Template, 22—Removable smart temple tips. Detailed Implementation

[0041] The technical solutions in the embodiments of the present invention will now be clearly and completely described in conjunction with the accompanying drawings.

[0042] This embodiment provides a voice interaction method and system for smart glasses based on a large language model. The smart glasses include a frame 2, lenses 1 mounted on the frame 2, temples 21 connected to the frame 2, and detachable smart temple tips 22 detachably connected to the ends of the temples 21. The detachable smart temple tips 22 can be removed from the ends of the temples 21 of one pair of glasses and installed on the ends of another temple 21 with a matching connection structure, so that the voice interaction function set in the detachable smart temple tips 22 is transferred along with the detachable smart temple tips 22.

[0043] The detachable smart foot cover 22 is equipped with an assembly status detection component, a voice acquisition component, an inertial detection component, a sound output component, a wireless communication component, a data storage component, a common time stamp component, an assembly identification component, and a local processing component. The local processing component includes a phase detection module, a linear elimination module, a homogeneous source correction module, an interface fingerprint module, a master frequency backtracking module, a migration partitioning module, a local synthesis module, and a replay fusion module.

[0044] The wireless communication component is used to receive the large language model response text. The large language model response text can be generated by a smart terminal that wirelessly communicates with the detachable smart shoe end 22, or it can be generated by a cloud-based large language model processing node. The local processing component divides the large language model response text into multiple continuous phonemes, performs speech synthesis on the continuous phonemes, and saves the corresponding text to be played, phoneme timing, and synthesized speech spectrum to the data storage component.

[0045] The sound output component can be a miniature speaker, a directional speaker, or a bone conduction transducer. In this embodiment, the sound output component is a miniature speaker disposed within the detachable smart temple end 22. The voice acquisition component employs at least one microphone to acquire the user's voice and the airborne sound response generated after the voice is output by the sound output component. The inertial detection component employs an accelerometer capable of acquiring high-frequency vibration components to acquire the structural vibration response propagating through the connection point between the detachable smart temple end 22 and the temple 21 when the sound output component is operating.

[0046] The assembly status detection component can be a Hall effect sensor, a mechanical contact sensor, a pressure sensor, or an impedance sensor. In this embodiment, the assembly status detection component outputs a separation status signal or a connection status signal based on the on / off state of the contacts at the connection position. The assembly marking component is used to output an assembly marking corresponding to the current temple 21. The assembly marking component can be a resistor encoder, a magnetic encoder, an RFID tag, or a memory chip storing the marking information of the temple 21.

[0047] After the detachable smart temple tip 22 is removed from the original temple 21 and installed on another temple 21, the tightness, contact area, material hardness, and local damping of the detachable connection may change. Even if the sound output component outputs the same voice, the airborne sound response and the structural vibration response transmitted through the temple 21 may also change.

[0048] The airborne acoustic response mainly includes the sound generated by the sound output component itself, the acoustic effects generated by the housing of the detachable smart temple tip 22, and the acoustic effects generated by the external wearing environment. The structural vibration response includes the vibration of the sound output component itself, as well as the vibration propagation effect generated by the connection interface between the detachable smart temple tip 22 and the temple 21.

[0049] This embodiment does not directly identify all nonlinear components in the structural vibration response as nonlinear components generated by the connection interface, but instead simultaneously collects the airborne acoustic response and the structural vibration response. The local processing component first determines the homologous nonlinear components generated by the sound output component based on the airborne acoustic response, and then subtracts the homologous nonlinear components from the structural vibration nonlinear residual to obtain the interface nonlinear residual mainly generated by the detachable connection interface.

[0050] After obtaining the interface nonlinear residual, the local processing component extracts the sum frequency point, difference frequency point, and corresponding intermodulation components, and forms the current interface nonlinear fingerprint based on the change in peak-to-average power ratio and the change in intermodulation components between the two detection subframes. The current interface nonlinear fingerprint is used to characterize the variation law of the nonlinear intermodulation response generated by the connection interface between the detachable smart temple end 22 and the temple 21 under different instantaneous excitation intensities.

[0051] When an abnormal intermodulation frequency point appears in the current interface nonlinear fingerprint relative to the interface nonlinear baseline, the mother frequency backtracking module determines the mother frequency band combination of the abnormal intermodulation frequency point in reverse based on the abnormal intermodulation frequency point and the intermodulation order. The migration partitioning module assigns the time-frequency units to be resumed playback involving the mother frequency band combination to the resynthesis domain, and assigns the remaining time-frequency units to be resumed playback to the migrateable domain. The local synthesis module resynthesizes the text to be resumed playback corresponding to the resynthesis domain, the local processing component performs cross-assembly linear compensation on the cached speech corresponding to the migrateable domain, and the resumed playback fusion module fuses the two parts of speech into resumed playback speech.

[0052] Therefore, this embodiment does not require re-performing speech synthesis on all text to be played, nor does it directly use all cached speech that may no longer be compatible with the current connection interface. Instead, it partitions and migrates the cached speech according to the mother frequency band range involved in the abnormal interface inter-call.

[0053] like Figure 2 As shown, the method in this embodiment includes, in sequence, a response caching step, a migration freezing step, a phase detection step, an interface extraction step, and a migration replay step.

[0054] In the response caching step, the wireless communication component acquires the response text of the large language model, and the local processing component performs text normalization, phoneme conversion and speech synthesis on the response text of the large language model to obtain the text to be played, the phoneme timing sequence and the synthesized speech spectrum.

[0055] The phoneme timing sequence includes the start time, end time, and duration of each phoneme. The synthesized speech spectrum can be obtained by performing a short-time Fourier transform on the synthesized speech.

[0056] Perform a short-time Fourier transform on the synthesized speech to generate a spectrum of the synthesized speech arranged by time frames and frequencies. The formula for calculating the short-time Fourier transform is as follows: ; Among them, synthesized speech signals Represents the time-domain synthesized speech to be cached; continuous time variables Indicates sampling time; analysis window function Indicates time shift Centered analysis window; time shift Indicates the time offset of the analysis window relative to the start position of the synthesized speech; angular frequency. Represents the angular frequency corresponding to the spectral analysis; imaginary unit. satisfy Synthetic speech time-frequency distribution This represents the complex spectrum of the synthesized speech signal at the corresponding time and frequency positions.

[0057] In one embodiment, the sampling frequency is set to 16kHz or 24kHz, the analysis window length is set to 20ms to 40ms, and the overlap ratio between adjacent analysis windows is set to 50% to 75%. This embodiment uses a 24kHz sampling frequency, a 32ms analysis window length, and a 75% overlap ratio.

[0058] While the detachable smart temple end 22 is being installed normally and outputting historical synthesized speech, the local processing component establishes a linear model of the airborne sound before separation based on the historical airborne sound response, a linear model of the structural vibration based on the historical structural vibration response, and establishes the speaker homology mapping relationship and interface nonlinear baseline. The above data is then correlated with the current assembly identifier of the temple 21 and saved to the data storage component.

[0059] The pre-separation airborne acoustic linear model describes the linear correspondence between the synthesized speech spectrum and the airborne acoustic response. The structural vibration linear model describes the linear correspondence between the synthesized speech spectrum and the structural vibration response. The linear models can be represented using frequency domain transfer functions.

[0060] The pre-separation airborne acoustic linear model is used to describe the linear transfer relationship between the synthesized speech spectrum and the airborne acoustic response. The calculation formula for the airborne acoustic linear model is as follows: ; Among them, the airborne acoustic linear model Represents frequency Linear transfer relationship from synthesized speech to airborne sound response; frequency Indicates the frequency position corresponding to the spectrum analysis; synthesized speech self-power spectrum Indicates the frequency of synthesized speech Average power; cross-power spectrum of synthesized speech and airborne sound response Indicates the frequency response of synthesized speech and airy sound. The following correlations; airborne sound regularization factor This is used to prevent the low-energy frequency bands of synthesized speech from generating excessively large model coefficients.

[0061] The linear model of structural vibration is used to describe the linear transfer relationship between the synthesized speech spectrum and the structural vibration response. The calculation formula for the linear model of structural vibration is as follows: ; Among them, the linear model of structural vibration Represents frequency Linear transfer relationship from synthesized speech to structural vibration response; cross-power spectrum of synthesized speech and structural vibration response. This indicates the frequency response of synthesized speech and structural vibration. The correlation between the following; the power spectrum of synthesized speech Indicates the frequency of synthesized speech Average power; structural vibration regularization factor Used to suppress model calculation fluctuations in low-energy frequency bands; frequency This indicates the frequency position corresponding to the spectrum analysis.

[0062] During the migration freeze step, the assembly status detection component detects the separation of the detachable smart foot sleeve 22 and outputs a separation status signal. In response to the separation status signal, the local processing component stops the sound output component from continuing to output the incomplete response speech and freezes the text to be played, phoneme timing, and synthesized speech spectrum corresponding to the currently incomplete output portion.

[0063] Freezing data to be played back does not mean deleting data that has already been output. Instead, it records the current output interruption point and retains the data that has not yet been output after the interruption point. For example, if a large language model response text includes five response statements, and the first three response statements have been output, and the fourth response statement is separated at the second phoneme, the local processing component records the position after the second phoneme of the fourth response statement as the starting point for continued playback, and freezes the text to be played back, the phoneme timing, and the synthesized speech spectrum after this starting point.

[0064] During the phase detection step, the assembly status detection component detects that the detachable smart foot sleeve end 22 has been reconnected and outputs a connection status signal. The local processing component reads the assembly identifier output by the assembly identifier component based on the connection status signal, and reads the pre-separation airborne acoustic linear model, structural vibration linear model, loudspeaker homology mapping relationship, and interface nonlinear baseline corresponding to the assembly identifier from the data storage component.

[0065] The local processing component selects a first and a second master frequency band from the first audio element to be played. The first and second master frequency bands should have a frequency interval sufficient to form a detectable sum or difference frequency, and the corresponding sum or difference frequency point should be located in the frequency band where the original audio element has lower energy.

[0066] For example, if the first audio element to be played has strong energy in the 900Hz-1100Hz and 1700Hz-1900Hz ranges, and lower energy in the 2600Hz-3000Hz range, then 900Hz-1100Hz can be designated as the first mother frequency band, and 1700Hz-1900Hz as the second mother frequency band. The sum frequency components of the first and second mother frequency bands are located in the 2600Hz-3000Hz range, and intermodulation components generated at the interface can be detected within this low-energy frequency band.

[0067] While keeping the amplitude spectrum and average energy of the first audio element to be played unchanged, the local processing component generates a first detection subframe and a second detection subframe by changing the initial phase of each sub-band, so that the first detection subframe and the second detection subframe have different peak-to-average power ratios.

[0068] The first and second detection subframes have the same amplitude spectrum and average energy, but different peak-to-average power ratios (PAPRs) are formed through different sub-band phase configurations. The formula for calculating the PAPR is as follows: ; Among them, peak-to-average ratio This represents the ratio between the peak power and the average power of the detected subframe; the discrete signal of the detected subframe. Indicates the first detection subframe or the second detection subframe in the... Discrete sampled values ​​at each sampling location; discrete sample index Indicates the sampling position within the detection subframe; number of sampling points in the detection subframe. Indicates the total number of sampling points contained in the detected subframe; maximum value operation. Indicates the maximum instantaneous power within the detected subframe; average power This represents the average power of all sampling points in the detected subframe.

[0069] The first and second detection subframes have the same amplitude spectrum, therefore, they should have essentially the same frequency band output energy in a linear system. Because the sub-band phases of the first and second detection subframes are different, their instantaneous peak value and peak-to-average power ratio are different. The nonlinear response of the interface changes with the instantaneous excitation intensity; therefore, the two detection subframes can detect the degree of nonlinearity of the interface without altering the original sound element amplitude spectrum.

[0070] The first and second detection subframes are encapsulated within the original duration of the first voiced sound to be played. For example, if the first voiced sound to be played lasts 120ms, the first and second detection subframes can be set to 50ms each, and a 20ms phase transition interval can be set between the two detection subframes so that the total duration of the two detection subframes does not exceed 120ms.

[0071] The sound output component continuously outputs the first and second detection subframes. The common time-stamp component provides the same sampling clock to the sound output component, voice acquisition component, inertial detection component, and local processing component, so that the first detection subframe, the second detection subframe, the airborne sound response, and the structural vibration response have a unified time reference.

[0072] In the interface extraction step, the local processing component first predicts the airborne acoustic linear response corresponding to the first and second detection subframes based on the airborne acoustic linear model before separation, and then subtracts the corresponding airborne acoustic linear response from the measured airborne acoustic response to obtain the airborne acoustic nonlinear residual.

[0073] The local processing component predicts the airborne acoustic linear response corresponding to the detected subframe based on the airborne acoustic linear model before separation, and subtracts the predicted airborne acoustic linear response from the measured airborne acoustic response to obtain the airborne acoustic nonlinear residual. The formula for calculating the airborne acoustic nonlinear residual is as follows: ; Among them, airborne acoustic nonlinear residual Indicates the first Each detection subframe in frequency The remaining nonlinear components after removing the linear component of airborne sound; detecting the subframe sequence number. Used to distinguish between the first detection subframe and the second detection subframe; measured airborne acoustic response. This indicates the first voice acquisition component to acquire. The airborne acoustic response spectrum corresponding to each detection subframe; the airborne acoustic linear model Represents the linear transfer relationship from synthesized speech to airborne sound response before separation; detects subframe spectra. Indicates the first Each detection subframe in frequency Complex spectrum under; prediction of airborne acoustic linear response Indicates the linear model of airborne sound for the first... The prediction results generated by each detection subframe.

[0074] The local processing component predicts the linear response of structural vibration corresponding to the first and second detection subframes based on the linear model of structural vibration, and then subtracts the corresponding linear response of structural vibration from the measured structural vibration response to obtain the nonlinear residual of structural vibration.

[0075] The formula for calculating the nonlinear residual of structural vibration is as follows: ; Among them, structural vibration nonlinear residuals Indicates the first Each detection subframe in frequency The remaining nonlinear components after removing the linear components of structural vibration; measured structural vibration response. This indicates the number of data collected by the inertial detection component. Structural vibration response spectrum corresponding to each detection subframe; linear model of structural vibration Represents the linear transfer relationship from synthesized speech to structural vibration response before separation; detects subframe spectra. Indicates the first Each detection subframe in frequency Complex spectrum under; predicting linear response of structural vibration The linear model of structural vibration is represented by the first... The prediction results generated by each detection subframe.

[0076] The loudspeaker homogeneous mapping relationship is used to map the harmonics and intermodulation components generated by the sound output component itself in the nonlinear residual of airborne sound to the corresponding homogeneous nonlinear components generated by the sound output component in the structural vibration channel.

[0077] In one embodiment, when the detachable smart temple tip 22 is separated from the temple 21, the sound output component outputs calibration voice. Since the detachable smart temple tip 22 is not connected to the temple 21, the nonlinearity of the structural vibration collected at this time mainly originates from the sound output component and the detachable smart temple tip 22 itself, and does not include the structural nonlinearity generated by the connection interface. By comparing the same-order harmonic components and same-order intermodulation components in the airborne acoustic response and the structural vibration response in the separated state, a source mapping relationship of the loudspeakers can be established.

[0078] The loudspeaker homogeneity mapping relationship is used to map the airborne acoustic nonlinear residual to homogeneous nonlinear components formed by the sound output components in the structural vibration channel. The formula for calculating the interface nonlinear residual is as follows: ; Among them, the interface nonlinear residual Indicates the first Each detection subframe in frequency The structural vibration nonlinear component remaining after deducting the nonlinear components from the same source as the sound output component; structural vibration nonlinear residual. Represents the residual obtained after removing the linear components of the structural vibration response; source mapping relationship of loudspeakers. This represents the frequency domain mapping relationship for the transformation of airborne acoustic nonlinear residuals to the same source nonlinear components of the structural vibration channel; airborne acoustic nonlinear residuals This represents the residual obtained after removing the linear airborne acoustic component from the airborne acoustic response; co-originating nonlinear components. This indicates the components that are mainly generated by the nonlinearity of the sound output component itself and mapped to the structural vibration channel.

[0079] The interface fingerprint module extracts the intermodulation components of the interface nonlinear residuals corresponding to the first and second detection subframes at the sum frequency points or difference frequency points, and forms the current interface nonlinear fingerprint based on the correspondence between the change in intermodulation components and the change in peak-to-average ratio.

[0080] The following is one formula for calculating the nonlinear fingerprint of the current interface: ; Among them, the current interface nonlinear fingerprint Indicates intermodulation frequency point The nonlinear response slope at the interface; intermodulation frequency. This indicates the sum frequency point, difference frequency point, or higher-order intermodulation frequency point generated by the first and second mother frequency bands; the nonlinear amplitude of the second detection subframe interface. This represents the interface nonlinearity amplitude of the second detection subframe at the intermodulation frequency point; the interface nonlinearity amplitude of the first detection subframe. The first detection subframe represents the amplitude of the interface nonlinearity at the intermodulation frequency point; the second detection subframe's peak-to-average power ratio (PAPR) represents the amplitude of the interface nonlinearity at the intermodulation frequency point. This represents the peak-to-average power ratio (PAPR) of the second detection subframe; the PAPR of the first detection subframe. Represents the peak-to-average power ratio (PAPR) of the first detection subframe; stability factor. This is used to prevent the denominator from approaching zero when the two peak-mean ratios are too close, which would lead to an abnormally large calculation result.

[0081] During the migration and replay process, the interface fingerprint module compares the current interface nonlinear fingerprint with the interface nonlinear baseline stored in the data storage component. The interface nonlinear baseline is a reference fingerprint formed when the detachable smart temple end 22 and the current temple 21 are in a stable assembly state.

[0082] When the intermodulation increment, intermodulation phase, or peak-to-average ratio response slope of a certain intermodulation frequency point reaches the corresponding abnormal threshold, the local processing component determines that the intermodulation frequency point as an abnormal intermodulation frequency point.

[0083] The bus frequency backtracking module determines the bus frequency band combination that generated the abnormal intermodulation frequency point by reverse-engineering based on the frequency value and intermodulation order of the abnormal intermodulation frequency point. The formula for calculating the second-order sum frequency is as follows: ; Among them, the sum frequency This represents the second-order sum frequency point generated by the combined effect of the first and second mother frequencies; the first mother frequency. Indicates the frequency within the first mother frequency band; the second mother frequency This indicates the frequency within the second mother frequency band.

[0084] The formula for calculating the second-order difference frequency is as follows: ; Among them, the difference frequency This represents the second-order difference frequency point corresponding to the absolute value of the difference between the first and second mother frequencies; the first mother frequency. Indicates the frequency within the first mother frequency band; the second mother frequency This represents the frequency within the second mother frequency band; the absolute value operation is used to keep the difference frequency a non-negative frequency.

[0085] The formula for calculating the third-order intermodulation frequency is as follows: ;or: ; Among them, the third-order intermodulation frequency This represents the intermodulation frequency point generated by the third-order nonlinear interaction between the first and second mother frequencies; the first mother frequency... Indicates the frequency within the first mother frequency band; the second mother frequency This indicates the frequency within the second mother frequency band. The mother frequency backtracking module searches for mother frequencies that satisfy the above frequency relationship within the effective frequency band of the voice to be played, and determines the frequency band containing the corresponding mother frequency as the mother frequency band combination.

[0086] If there are multiple possible combinations of mother bands, the local processing component will prioritize retaining the mother band combination that meets the following conditions: the mother band contains voice energy to be played, the mother band combination can generate the corresponding abnormal intermodulation frequency point, and the mother band combination is excited in both the first detection subframe and the second detection subframe.

[0087] The migration partitioning module divides the spectrum of the speech to be resumed into multiple time-frequency units. For example, the time-frequency units can be divided according to a 32ms time window and a 250Hz bandwidth. Time-frequency units whose spectral energy falls into abnormal mother frequency band combinations are assigned to the resynthesis domain; the remaining time-frequency units are assigned to the migrateable domain.

[0088] The local synthesis module regenerates the corresponding phoneme speech based on the text to be played back and the original phoneme timing for the resynthesis domain. For the transferable domain, the local processing component forms a cross-assembly linear compensation relationship based on the linear response after reconnection and the linear response before separation, and performs frequency domain compensation on the buffered speech.

[0089] The cross-assembly linear compensation relationship can be obtained using regularized inverse filtering, and its calculation formula is as follows: ; Among them, the linear compensation relationship across assembly Represents frequency The frequency domain compensation function is applied to the buffered speech; the synthesized linear response before separation. This indicates that before the detachable smart foot cover end 22 is separated, at the frequency The overall linear response measured below; the overall linear response is then reconnected. This indicates that after the detachable smart foot cover end 22 is reconnected, at a frequency The overall linear response was measured; the conjugate of the overall linear response was then reconnected. This represents the complex conjugate of the synthesized linear response after reconnection; the square of the magnitude of the synthesized linear response after reconnection. Indicates the energy magnitude of the synthesized linear response after reconnection; compensation regularization factor. Used to limit the compensation gain in weak response bands to prevent excessive amplification caused by cross-assembly linear compensation.

[0090] The playback fusion module combines the resynthesized resynthesized domain speech and the transferable domain speech after cross-assembly linear compensation into a playback speech according to the original phoneme timing. For the boundary between the resynthesized domain and the transferable domain, an overlap interval of 5ms to 20ms can be set, and weighted fusion is performed within the overlap interval to reduce boundary abruptness.

[0091] like Figure 3 As shown, the frame 2 is used to mount the lens 1, the temple 21 is connected to the frame 2, and the detachable smart temple tip 22 is connected to the end of the temple 21. Figure 3 The upper frame 2, lens 1, temple 21, and detachable smart temple tips 22 represent the overall mechanical composition of the smart glasses. Figure 3 The lower part shows the internal logic structure of the detachable smart foot sleeve 22. The assembly status detection component, voice acquisition component, inertial detection component, wireless communication component, data storage component, and assembly identification component respectively provide the local processing component with detection data, back-collected data, structural vibration data, large language model response text, cached data, and assembly identification.

[0092] The assembly status detection component outputs a separation status signal when the detachable smart temple tip 22 separates from the temple 21, and outputs a connection status signal when the detachable smart temple tip 22 reconnects to the temple 21. The local processing component performs a migration freeze step based on the separation status signal and initiates a phase detection step based on the connection status signal.

[0093] The voice acquisition component collects airborne sound responses. It can also collect user voice, enabling the local processing component to send the user's voice to the large language model processing node. The inertial detection component collects structural vibration responses. This component is located within the detachable smart foot sleeve 22, near the connection point, allowing it to receive structural vibrations propagating through the detachable connection.

[0094] The wireless communication component is used to receive the response text from the large language model. The wireless communication component can be a Bluetooth communication component, a wireless LAN communication component, or a mobile communication component. The data storage component is used to store the text to be played, phoneme timing, synthesized speech spectrum, pre-separation airborne acoustic linear model, structural vibration linear model, speaker source mapping relationships, interface nonlinear baselines, and assembly identification correspondences.

[0095] The assembly identification component outputs the assembly identification corresponding to the current temple 21. The local processing component reads the corresponding interface nonlinear baseline from the data storage component based on the assembly identification. If the data storage component does not contain an interface nonlinear baseline corresponding to the current assembly identification, the local processing component can establish a new interface nonlinear baseline through multiple tests after the initial stable installation is completed. The common time stamp component provides the same sampling clock and output time stamp to the voice acquisition component, inertial detection component, sound output component, and local processing component, respectively. Through the common time stamp component, the output time of the first detection subframe and the second detection subframe can correspond to the acquisition time of the airborne acoustic response and the structural vibration response, respectively.

[0096] The phase detection module within the local processing component generates a first detection subframe and a second detection subframe based on the first sound element to be played. The linear cancellation module generates airborne nonlinear residuals and structural vibration nonlinear residuals based on the pre-separation airborne acoustic linear model and structural vibration linear model, respectively. The homogeneity correction module subtracts the homogeneity nonlinear component generated by the sound output component from the structural vibration nonlinear residual based on the speaker homogeneity mapping relationship. The interface fingerprint module generates the current interface nonlinear fingerprint based on the interface nonlinear residual.

[0097] The mother frequency backtracking module determines the mother frequency band combination based on abnormal intermodulation frequency points and intermodulation orders. The migration partitioning module divides the time-frequency unit to be resumed into a resynthesis domain and a migrateable domain based on the mother frequency band combination. The local synthesis module resynthesizes the text to be resumed corresponding to the resynthesis domain. The resumed playback fusion module fuses the resynthesized phoneme speech with the buffered speech after cross-assembly linear compensation and sends the resumed playback speech to the sound output component.

[0098] Figure 3 The module connections represent the signal and data flow, and it is not required that each module be implemented by an independent chip. In one implementation, the local processing component can be implemented by a digital signal processor or a microcontroller with a digital signal processing unit, and the phase detection module, linear cancellation module, homogeneous source correction module, interface fingerprint module, bus frequency backtracking module, migration partitioning module, local synthesis module, and replay fusion module are implemented by program instructions stored in the data storage component.

[0099] like Figure 4As shown, the first sound element to be played is phase-configured to form a first detection subframe and a second detection subframe. The first and second detection subframes are output sequentially by the sound output component, generating airborne sound response and structural vibration response, respectively.

[0100] The airborne acoustic response enters the airborne acoustic processing branch. The linear cancellation module calculates the corresponding linear predicted airborne acoustic response based on the airborne acoustic linear model before separation, and subtracts the airborne acoustic response from the linear predicted airborne acoustic response to obtain the airborne acoustic nonlinear residual. The structural vibration response enters the structural vibration processing branch. The linear cancellation module calculates the corresponding linear predicted structural vibration response based on the structural vibration linear model, and subtracts the structural vibration response from the linear predicted structural vibration response to obtain the structural vibration nonlinear residual.

[0101] The nonlinear residuals of airborne sound may include harmonic distortion, intermodulation distortion, and nonlinear acoustic components caused by the housing of the detachable smart foot sleeve 22, all from the sound output component. The nonlinear residuals of structural vibration may also include nonlinear components generated by the vibration of the sound output component itself. If the nonlinear components generated by the sound output component are not corrected, the distortion of the sound output component itself may be mistaken for distortion at the detachable connection interface.

[0102] Therefore, the same-source correction module maps the airborne nonlinear residual to the same-source nonlinear component formed by the sound output component in the structural vibration channel according to the same-source mapping relationship of the loudspeaker, and subtracts the same-source nonlinear component from the structural vibration nonlinear residual to obtain the interface nonlinear residual.

[0103] The interface nonlinear residual mainly reflects the nonlinear structural response generated at the connection point between the detachable smart temple end 22 and the temple 21 under the action of two detection subframes. The interface fingerprint module extracts the interface nonlinear amplitude and interface nonlinear phase at the sum frequency point, difference frequency point, or third-order intermodulation frequency point of the first and second detection subframes, and calculates the change relationship between the two detection subframes to form the current interface nonlinear fingerprint.

[0104] For example, the peak-to-average power ratio (PAPR) of the first detection subframe is 3.2, and the PAPR of the second detection subframe is 6.0. The interface nonlinearity amplitude at the 2700Hz intermodulation frequency point in the first detection subframe is 0.015, and the interface nonlinearity amplitude at the 2700Hz intermodulation frequency point in the second detection subframe is 0.041. The interface fingerprint module calculates the interface nonlinearity response slope corresponding to the 2700Hz intermodulation frequency point based on the PAPR change and the intermodulation amplitude change, and compares it with the reference response slope at the 2700Hz intermodulation frequency point in the interface nonlinearity baseline.

[0105] If the slope of the current interface nonlinear response is significantly higher than the interface nonlinear baseline, it indicates that the current connection interface has generated an enhanced nonlinear intermodulation response under the corresponding master frequency band excitation. This enhanced response may be caused by loose connection, reduced contact area, insufficient local clamping force, or changes in the material interface.

[0106] To reduce the impact of environmental noise, two to four acquisitions can be performed on the first and second detection subframes respectively, and the median of the interface nonlinear residuals at the same intermodulation frequency point can be taken. If the voice acquisition component detects a sudden external sound or the inertial detection component detects significant head movement during a certain acquisition, the corresponding acquisition result is deleted.

[0107] like Figure 5 As shown, the abnormal intermodulation frequency point is obtained by comparing the current interface nonlinear fingerprint with the interface nonlinear baseline. The bus frequency backtracking module determines the bus frequency band combination based on the abnormal intermodulation frequency point and the intermodulation order.

[0108] For example, if a second-order sum-frequency anomaly at 2700Hz is detected, and the first voice element to be played has dominant spectral energy around 1000Hz and 1700Hz, the mother frequency backtracking module can identify 900Hz to 1100Hz and 1600Hz to 1800Hz as the mother frequency band combination. If a second-order difference-frequency anomaly at 800Hz is detected, the mother frequency backtracking module can determine the candidate mother frequency band combination based on the effective frequency bands in the spectrum of the speech to be played that differ by approximately 800Hz.

[0109] The migration partitioning module divides the time-frequency units to be played according to phoneme timing and frequency band. Figure 5 The time-frequency unit to be continued in the middle includes a transferable domain, a recombination domain and another transferable domain. Figure 5 The resynthesis domain is drawn between two transferable domains to represent the area within a single phoneme or multiple consecutive phonemes, where only a portion of the time and frequency range may be affected by abnormal mother band combinations. For the resynthesis domain, the local processing component determines the smallest consecutive phoneme interval covering the resynthesis domain and obtains the text to be played corresponding to the smallest consecutive phoneme interval. The local synthesis module performs text resynthesis based on the text to be played and the original phoneme timing.

[0110] For the transferable domain, the local processing component utilizes cross-assembly linear compensation relationships to process the buffered speech. Cross-assembly linear compensation is used to correct the amplitude and phase changes of the linear propagation path after reconnection relative to the linear propagation path before separation. The phoneme speech obtained by text resynthesis has the same original phoneme timing as the buffered speech after cross-assembly linear compensation. The continuation fusion module performs phase-continuous fusion at the boundary between the two speech parts to form the continuation speech.

[0111] Figure 5Text recombining in the recombining domain and cross-assembly linear compensation in the transferable domain are processed in parallel. Text recombining does not affect the transferable domain, and cross-assembly linear compensation does not directly replace text recombining in the recombining domain. The results of the two processes are combined in the replay fusion module according to the time and frequency position of the time-frequency unit to be replayed.

[0112] The following parameters are used to describe one executable implementation method and do not constitute a limitation on the scope of protection.

[0113] The audio sampling frequency can be set to 24kHz to acquire synthesized speech, airborne sound response, and structural vibration response. The analysis window length can be set to 32ms to form the synthesized speech spectrum and response spectrum. The analysis window overlap ratio can be set to 75% to improve time-frequency continuity near phoneme boundaries. The duration of the first detection subframe can be set to 50ms to acquire the dual-channel response under the first peak-to-average power ratio (PAPR) condition. The duration of the second detection subframe can be set to 50ms to acquire the dual-channel response under the second PAPR condition. The phase transition interval can be set to 20ms to reduce auditory abrupt changes at the boundary between the two detection subframes. The bandwidth of the time-frequency unit to be continued can be set to 250Hz to divide the time-frequency unit to be continued. The fusion overlap interval can be set to 10ms to fuse the resynthesized domain speech and the transferable domain speech. The airborne sound regularization factor can be set to 0.001 to 0.01 to suppress model fluctuations in low-energy frequency bands. The compensation regularization factor can be set to 0.005 to 0.05 to prevent excessive gain from cross-assembly linear compensation.

[0114] The abnormal threshold can be determined based on multiple measurement results under stable assembly conditions. Twenty tests can be performed when the detachable smart temple tip 22 is stably connected to the same temple 21, calculating the reference mean and reference standard deviation for each intermodulation frequency point. If the current test result deviates from the reference mean by a preset multiple of the reference standard deviation, the corresponding intermodulation frequency point is determined as an abnormal intermodulation frequency point.

[0115] When the intermodulation increment deviates from the reference mean by less than 2 times the reference standard deviation, the corresponding intermodulation frequency point can be retained as a normal intermodulation frequency point. When the intermodulation increment deviates from the reference mean by 2 to 3 times the reference standard deviation, an additional detection and reassessment can be performed. When the intermodulation increment deviates from the reference mean by 3 times the reference standard deviation, the corresponding intermodulation frequency point can be identified as an abnormal intermodulation frequency point. When multiple sets of mother frequency band combinations exist, the mother frequency band combination with effective energy for the speech to be played can be retained. When the duration corresponding to the resynthesis domain is less than the duration of a phoneme, the resynthesis domain can be expanded to cover the smallest continuous phoneme interval of the corresponding area. When the compensation gain of any frequency band exceeds the preset gain limit, the corresponding time-frequency unit to be played can be transferred to the resynthesis domain.

[0116] In a complete implementation example, a user queries a large language model for weather information via smart glasses. The wireless communication component receives the large language model's response text. The local processing component converts the large language model's response text into phoneme timing and synthesized speech, which is then played back by the sound output component.

[0117] During playback, the user removes the detachable smart temple tip 22 from the first temple 21. The assembly status detection component outputs a separation status signal. The local processing component stops playback and freezes the unfinished output of text, phoneme timing, and synthesized speech spectrum.

[0118] The user installs the detachable smart temple tip 22 onto the second temple 21. The assembly status detection component outputs a connection status signal, and the assembly identification component outputs the assembly identification corresponding to the second temple 21. The local processing component reads the pre-separation airborne acoustic linear model, structural vibration linear model, loudspeaker homology mapping relationship, and interface nonlinear baseline corresponding to the second temple 21.

[0119] The phase detection module generates a first detection subframe and a second detection subframe from the first audio element to be played. The two detection subframes have the same amplitude spectrum and average energy, but different sub-band phases and peak-to-average power ratios. The sound output component outputs the two detection subframes, and the speech acquisition component and inertial detection component synchronously acquire the airborne acoustic response and structural vibration response under the control of a common time-scale component.

[0120] The linear cancellation module obtains the airborne sound nonlinear residual and the structural vibration nonlinear residual, respectively. The homogeneous source correction module, based on the speaker homogeneous source mapping relationship, subtracts the homogeneous nonlinear component generated by the sound output component from the structural vibration nonlinear residual to obtain the interface nonlinear residual. The interface fingerprint module generates the current interface nonlinear fingerprint based on the response of the interface nonlinear residual at the intermodulation frequency point as a function of the peak-to-average power ratio (PAPR).

[0121] If the current interface nonlinear fingerprint shows an anomaly at 2700Hz relative to the interface nonlinear baseline, the mother frequency backtracking module determines 900Hz to 1100Hz and 1600Hz to 1800Hz as the corresponding mother frequency band combinations. The migration partitioning module assigns the time-frequency units in the speech to be played that involve the above two mother frequency band combinations to the recombination domain, and assigns the remaining time-frequency units to be played to the migrateable domain.

[0122] The local synthesis module generates new phoneme speech based on the text to be continued and the original phoneme timing corresponding to the resynthesis domain. The local processing component processes the buffered speech corresponding to the transferable domain according to the cross-assembly linear compensation relationship. The continuation fusion module performs phase-continuous fusion of the new phoneme speech and the compensated buffered speech to form the continuation speech, and continues to output the large language model response through the sound output component.

[0123] Through the above processing, after the detachable smart temple end 22 is moved to different temples 21, the system can use the normal voice to be played to form a detection subframe. Without inserting additional dedicated detection sounds unrelated to the large language model's response, it can identify the nonlinear changes generated by the detachable connection interface and select buffered voice compensation or text resynthesis according to the mother frequency band range involved in the abnormal intermodulation. This reduces the processing load and waiting time caused by complete resynthesis and reduces the distortion that may be caused by directly reusing the original buffered voice.

Claims

1. A voice interaction method for smart glasses based on a large language model, characterized in that, Includes the following steps: Response caching steps: Obtain the response text of the large language model, cache the text to be played, phoneme timing and synthesized speech spectrum by phoneme, and save the pre-separation air acoustic linear model, structural vibration linear model, loudspeaker homology mapping relationship and interface nonlinear baseline; Migration freeze step: When the detachable smart foot sleeve end (22) is detected to be separated during voice output, the data to be resumed is frozen; Phase detection steps: After reconnection, generate the first and second detection subframes with the same amplitude spectrum and average energy but different subband phase and peak-to-average power ratio from the first audio element to be played. Make the preset intermodulation frequency point fall into the low-energy frequency band of the two detection subframes, continuously output the two detection subframes and simultaneously collect the airborne sound response and structural vibration response. Interface extraction steps: Remove the linear components of the two types of responses according to the two linear models and form a cross-assembly linear compensation relationship; use the nonlinear residual of airborne sound to correct the nonlinear residual of structural vibration according to the source mapping relationship of the loudspeaker; and form the nonlinear fingerprint of the current interface according to the change of the corrected intermodulation components with the peak-to-average power ratio. Migration and replay steps: Based on the abnormal intermodulation frequency points of the interface nonlinear fingerprint relative to the interface nonlinear baseline, the mother band combination is determined in reverse. The time-frequency units to be replayed involving the mother band combination are assigned to the resynthesis domain, and the rest are assigned to the migratable domain. Text resynthesis and cross-assembly linear compensation are performed respectively to form the replay voice.

2. The voice interaction method for smart glasses based on a large language model according to claim 1, characterized in that, The response caching step also includes the following sub-steps: Voice filtering sub-step: Filter out voice elements from the output synthesized speech that have a fundamental frequency variation level below the fundamental frequency variation threshold, contain at least two effective sub-frequency bands, and do not overlap with the user's speech; Dual-mode update sub-step: Based on the selected sound elements, corresponding airborne sound responses, and corresponding structural vibration responses, update the pre-separation airborne sound linear model and the structural vibration linear model respectively; Baseline establishment sub-step: Select historical detection subframes with the same amplitude spectrum but different peak-to-average ratios from the selected sound elements, form the interface nonlinear baseline based on the air acoustic response and structural vibration response corresponding to the historical detection subframes, and establish a correspondence between the interface nonlinear baseline and the assembly identifier of the current temple (21).

3. The voice interaction method for smart glasses based on a large language model according to claim 2, characterized in that, The phase detection step further includes the following sub-steps: Phoneme filtering sub-step: Select from the phonemes corresponding to the text to be played that simultaneously contain the first mother frequency band and the second mother frequency band, and the sum frequency point or difference frequency point corresponding to the first mother frequency band and the second mother frequency band is located in the low energy frequency band; Phase configuration sub-step: Keeping the phoneme duration, fundamental frequency trajectory, amplitude spectrum and average energy of the sound element unchanged, configure the first sub-band phase set and the second sub-band phase set respectively, so that the first detection sub-frame and the second detection sub-frame have different peak-to-average power ratios; Intra-frame encapsulation sub-step: Encapsulate the first detection subframe and the second detection subframe within the original phoneme duration of the sound element, and perform phase continuity processing on the boundary between the two detection subframes.

4. The voice interaction method for smart glasses based on a large language model according to claim 3, characterized in that, The interface extraction step also includes the following sub-steps: Linear elimination sub-step: The linear response corresponding to the two detection subframes is predicted by the pre-separation airborne acoustic linear model and the structural vibration linear model, respectively. The difference between the measured airborne acoustic response and the predicted airborne acoustic response is determined as the airborne acoustic nonlinear residual, and the difference between the measured structural vibration response and the predicted structural vibration response is determined as the structural vibration nonlinear residual. Same-source correction step: Using the same-source mapping relationship of the loudspeaker, the airborne nonlinear residual is mapped to the same-source nonlinear component formed by the loudspeaker in the structural vibration channel. The same-source nonlinear component is subtracted from the structural vibration nonlinear residual to obtain the interface nonlinear residual. Fingerprint generation sub-step: Extract the intermodulation components of the interface nonlinear residuals corresponding to the two detection subframes at the sum frequency point or the difference frequency point, and form the current interface nonlinear fingerprint based on the correspondence between the change in intermodulation components and the change in peak-to-average ratio.

5. The voice interaction method for smart glasses based on a large language model according to claim 4, characterized in that, The interface extraction step also includes the following sub-steps: Separation calibration sub-step: When the detachable smart temple end (22) is separated from the temple (21), calibration voice is output through the detachable smart temple end (22), and the air sound response and structural vibration response in the separation state are collected simultaneously; Mapping establishment sub-step: Extract the same-order harmonic components and the same-order intermodulation components from the air acoustic response and the structural vibration response of the separated state, respectively, and establish the same-source mapping relationship of the loudspeaker based on the amplitude ratio and phase difference between the corresponding components; Assembly verification sub-step: After the reconnection is completed at the detachable smart foot sleeve end (22), identify the same source components that satisfy the amplitude ratio and the phase difference from the air acoustic nonlinear residual and the structural vibration nonlinear residual, and perform the same source correction sub-step using only the same source components that satisfy the speaker same source mapping relationship; Polarity configuration sub-step: Generate positive polarity detection frames and negative polarity detection frames with the same amplitude spectrum and opposite overall polarity from the first detection sub-frame or the second detection sub-frame; Polarity output sub-step: While keeping the duration of the original phoneme corresponding to the sound element unchanged, output the positive polarity detection frame and the negative polarity detection frame, and collect the corresponding airborne sound response and structural vibration response respectively; Order separation sub-step: Perform summation and difference processing on the interface nonlinear residuals corresponding to the positive polarity detection frame and the negative polarity detection frame to extract the even-order interface nonlinear components and the odd-order interface nonlinear components respectively; Fingerprint verification sub-step: Verify the current interface nonlinear fingerprint based on the even-order interface nonlinear components and the odd-order interface nonlinear components, and delete abnormal intermodulation frequency points that fail the order consistency verification.

6. The voice interaction method for smart glasses based on a large language model according to claim 5, characterized in that, The migration and replay step also includes the following sub-steps: Anomaly identification sub-step: Compare the intermodulation increment, intermodulation phase, and peak-to-average ratio response slope of the current interface nonlinear fingerprint and the interface nonlinear baseline at each intermodulation frequency point, and determine the intermodulation frequency point where any comparison result reaches the corresponding anomaly threshold as the abnormal intermodulation frequency point; The mother frequency backtracking sub-step involves determining, in reverse, at least one mother frequency band combination that can generate the corresponding abnormal intermodulation frequency point based on the frequency value and intermodulation order of each abnormal intermodulation frequency point. Unit marking sub-step: Mark the time-frequency units to be re-reconstituted as time-frequency units whose spectral energy is located within any of the aforementioned mother frequency band combinations; mark the remaining time-frequency units to be re-reconstituted as time-frequency units to be compensated. Region generation sub-step: Generate the recombination domain based on the time-frequency units to be recombined that are time-continuous and frequency-adjacent; generate the transferable domain based on the time-frequency units to be compensated. Interval mapping sub-step: Based on the phoneme timing, map the resynthesis domain to the smallest continuous phoneme interval covering the resynthesis domain; Boundary extraction sub-step: Extract the boundary fundamental frequency, boundary spectral envelope, boundary phase, and boundary energy from the transferable domain located before and after the minimum continuous phoneme interval; Local synthesis sub-step: Regenerate phoneme speech based on the text to be played and the original phoneme timing corresponding to the minimum continuous phoneme interval, and regenerate phoneme speech using the boundary fundamental frequency, the boundary spectral envelope, the boundary phase and the boundary energy constraints; Continuous fusion sub-step: Perform phase continuous fusion of the regenerated phoneme speech with the transferable domain speech processed according to the cross-assembly linear compensation relationship to form the continued playback speech.

7. A voice interaction system for smart glasses based on a large language model, characterized in that, include: The frame (2) is equipped with a lens (1); The temple (21) is connected to the frame (2); The detachable smart temple tip (22) is detachably connected to the end of the temple (21); An assembly status detection component is installed inside the detachable smart foot sleeve end (22) and is used to output separation status signals and connection status signals; A voice acquisition component is installed inside the detachable smart foot cover end (22) for acquiring airborne sound response; An inertial detection component is disposed inside the detachable smart foot sleeve end (22) and adjacent to the connection position for collecting structural vibration response; A sound output component is disposed within the detachable smart foot sleeve end (22) for outputting synthesized speech; A wireless communication component is disposed within the detachable smart foot sleeve end (22) for acquiring the response text of a large language model; The data storage component is located inside the detachable smart foot sleeve end (22) and is used to store the text to be played, phoneme timing, synthesized speech spectrum, airborne sound linear model, structural vibration linear model, speaker homology mapping relationship and interface nonlinear baseline; And a local processing component, which is located in the detachable smart foot sleeve end (22) and connected to each of the aforementioned components, is used to generate two detection subframes with the same amplitude spectrum but different peak-to-average ratios from the sound elements to be played after reconnection, extract the interface nonlinear fingerprint based on the two types of responses and the speaker homogeneous mapping relationship, reversely determine the mother frequency band combination based on the abnormal intermodulation frequency points, perform text resynthesis on the time-frequency units to be played involving the mother frequency band combination, and perform buffer spectrum compensation on the remaining time-frequency units to be played.

8. The intelligent glasses voice interaction system based on a large language model according to claim 7, characterized in that, The inertial detection component is disposed on the side of the detachable smart temple end (22) facing the end of the temple (21); The system also includes: A common time-stamping component, which is connected to the sound output component, the voice acquisition component, the inertial detection component, and the local processing component respectively, is used to allocate the same sampling clock for the detection subframe, the airborne sound response, and the structural vibration response; And an assembly marking component, which is set at the connection position between the detachable smart temple end (22) and the end of the temple (21), and connected to the local processing component, for outputting an assembly marking corresponding to the current temple (21), so that the local processing component reads the corresponding interface nonlinear baseline according to the assembly marking.

9. The intelligent glasses voice interaction system based on a large language model according to claim 8, characterized in that, The local processing component includes: A phase detection module, connected to the data storage component and the sound output component, is used to generate two detection subframes from the sound elements to be played, which have the same amplitude spectrum and average energy but different sub-band phase and peak-to-average ratio. A linear cancellation module is connected to the voice acquisition component, the inertial detection component, and the data storage component, respectively, and is used to generate nonlinear residuals of airborne sound and nonlinear residuals of structural vibration based on the linear model of airborne sound and the linear model of structural vibration. The same-source correction module is connected to the linear elimination module and the data storage component respectively, and is used to subtract the same-source nonlinear component of the loudspeaker from the nonlinear residual of structural vibration according to the same-source mapping relationship of the loudspeaker; And an interface fingerprint module, connected to the homogeneous correction module, is used to form the interface nonlinear fingerprint based on the change of the corrected intermodulation components with the peak-to-average power ratio.

10. The intelligent glasses voice interaction system based on a large language model according to claim 9, characterized in that, The local processing component also includes: The mother frequency backtracking module is connected to the interface fingerprint module and the data storage component, and is used to reversely determine the corresponding mother frequency band combination based on the abnormal intermodulation frequency points and intermodulation order of the interface nonlinear fingerprint relative to the interface nonlinear baseline. The migration partitioning module is connected to the mother frequency backtracking module and the data storage component, and is used to classify the time-frequency units to be continued broadcasting involving the mother frequency band combination into the recombination domain, and classify the remaining time-frequency units to be continued broadcasting into the migrationable domain. A local synthesis module, connected to the migration partition module and the data storage component, is used to regenerate phoneme speech based on the text to be played back and the original phoneme timing corresponding to the resynthesis domain; The system also includes a continuous playback fusion module, which is connected to the migration partitioning module, the local synthesis module, and the sound output component, respectively, and is used to perform phase-continuous fusion of the regenerated phoneme speech with the transferable domain speech that has undergone cross-assembly linear compensation.

Images