Intelligent hearing-aid earphone semantic interaction method and system based on multi-modal perception

Abstract

The invention discloses an intelligent hearing-aid earphone semantic interaction method and system based on multi-mode perception. The method comprises the following steps: acquiring a user voice signal and an environmental acoustic signal through a multi-mode signal acquisition unit integrated on an intelligent hearing-aid earphone body; collaborative noise reduction processing is carried out to generate an enhanced voice stream; obtaining a semantic understanding result based on the enhanced voice stream, and generating a first response set; obtaining a user interaction signal and an interaction state, and generating a second response set in cooperation with the first response set; the semantic database is modified based on the first set of responses and the second set of responses. The method has the following advantages and effects: a closed-loop system capable of deeply fusing acoustic noise reduction, context-aware semantic analysis, real-time interaction optimization based on user feedback and dynamic evolution capability of a knowledge base is provided, so that accurate, smooth and continuously evolved semantic interaction experience is realized in a complex and changeable real environment; and the user communication efficiency and life quality are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of hearing aid devices, in particular to a semantic interaction method and system of intelligent hearing aid based on multi-modal perception. BACKGROUND

[0002] Intelligent hearing aid devices are evolving from basic sound amplifiers to comprehensive platforms that integrate hearing assistance and intelligent interaction. Although the introduction of automatic speech recognition and natural language understanding technology gives the device certain semantic interaction capabilities, existing technical solutions face a fundamental system-level bottleneck in real complex scenarios: the semantic understanding and interaction feedback mechanism is difficult to maintain high accuracy and adaptability in dynamically changing acoustic environments and user intentions, resulting in low interaction efficiency and poor user experience. This is mainly reflected in the following aspects: existing noise reduction techniques often focus on improving the signal-to-noise ratio at the acoustic signal level, but fail to fully consider the destructive impact of noise characteristics (such as specific frequency band energy distribution) on the recognizability of key semantic units, resulting in "clear hearing" but not "understanding". In particular, in non-stationary high-noise environments, the loss or distortion of key speech information will directly pass to the subsequent semantic analysis module, causing understanding bias at the source. More seriously, the semantic understanding process is usually isolated and static: it lacks effective fusion and tracking of accumulated context information (such as historical interaction topics, key words) in continuous dialogue, causing the system's analysis capability to drop sharply when facing ambiguous or omitted expressions; at the same time, the system also fails to assess and adjust the confidence weight of the semantic analysis result in real time according to the dynamic characteristics of the current acoustic environment (such as the mutation of the noise spectrum), resulting in the output of incorrect but high-confidence results in interference environments. Ultimately, existing systems have significant defects in the interaction loop and knowledge evolution: user feedback on the system's initial response (whether through voice or physical interaction such as button pressing, touch operation, implicit confirmation, rejection, or inquiry intent) is not deeply analyzed as "interaction state" and used to optimize subsequent responses, and is not systematically used to continuously and finely update and optimize the internal semantic knowledge base. This rigid system, which lacks dynamic adaptation and self-learning ability, cannot learn and evolve from the user's actual interaction behavior, resulting in a long-term low-efficiency operation state and significantly increasing the user's cognitive burden. SUMMARY

[0003] The purpose of the present application is to provide a semantic interaction method and system of intelligent hearing aid based on multi-modal perception to solve the problems raised in the background art.

[0004] The above technical purpose of the present application is realized by the following technical scheme: The present application provides a semantic interaction method of intelligent hearing aid based on multi-modal perception, comprising the following steps: S100, acquiring user speech signals and environmental acoustic signals through a multi-modal signal acquisition unit integrated in a smart hearing aid earphone body; wherein the multi-modal signal acquisition unit comprises a directional microphone array, a bone conduction sensor, and an ambient noise monitoring module; S200, performing cooperative noise reduction processing on the user speech signals and the environmental acoustic signals to generate an enhanced speech stream; wherein the cooperative noise reduction processing comprises filtering to generate a primary noise reduction signal using an adaptive beamforming algorithm, and generating a frequency band compensation weight matrix using a pre-trained convolutional recurrent neural network model; S300, obtaining a semantic understanding result based on the enhanced speech stream; wherein the semantic understanding result comprises at least one candidate semantic label and a corresponding final confidence weight, the candidate semantic label corresponds to a context keyword, and the context keyword is stored in a pre-set semantic database; S400, generating a first response set based on the semantic understanding result; wherein the first response set comprises at least one candidate response content, the candidate response content is generated based on a candidate semantic label, and the first response set is used for output and display on a user interface of the smart hearing aid earphone body; S500, acquiring a user interaction signal and obtaining an interaction state based on the user interaction signal; wherein the user interaction signal is a signal emitted when the user operates an interaction module of the smart hearing aid earphone body, and the interaction state reflects the feedback type of the user on the candidate response content; S600, generating a second response set based on the interaction state and the first response set; wherein the second response set comprises at least one updated candidate response content, and the second response set is used for output and playback on an audio output unit of the smart hearing aid earphone; S700, modifying a semantic database based on the first response set and the second response set; wherein the modification process comprises adjusting the confidence weight corresponding to the semantic label or adding a new semantic label.

[0005] By using the above technical solution, multi-modal signal acquisition is achieved by integrating a directional microphone array, a bone conduction sensor, and an ambient noise monitoring module, which realizes full-dimensional capture of user speech and environmental acoustic signals, lays a data foundation for subsequent high-precision noise reduction and semantic analysis; combined with the cooperative noise reduction processing of the adaptive beamforming algorithm and the convolutional recurrent neural network, the frequency band compensation limitation of traditional noise reduction technology is broken, and the generated enhanced speech stream significantly improves speech clarity and intelligibility; the candidate response set generated based on the semantic understanding result dynamically optimizes the output content combined with the user interaction state, forms a closed-loop interactive learning mechanism, and directly improves the accuracy of user intent recognition; finally, through the dynamic modification of the semantic database, the system continuously optimizes itself, ensuring that the interaction model evolves adaptively with the use scenario.

[0006] Further settings are that the S200 specifically includes the following steps: extracting a frequency energy distribution of the environmental acoustic signal by the environmental noise monitoring module; wherein the frequency energy distribution contains sound pressure level measurement values of multiple sub-frequency bands in a preset frequency band range, and the environmental noise energy of each sub-frequency band is converted based on the frequency energy distribution; performing sub-band energy calculation on the original voice signal collected by the directional microphone array to generate a user voice signal energy distribution; based on the environmental noise energy and the user voice signal energy distribution, a signal-to-noise ratio attenuation factor of each sub-frequency band is calculated; performing spatial filtering on the original voice signal collected by the directional microphone array using an adaptive beamforming algorithm to generate a primary noise reduction signal; wherein the adaptive beamforming algorithm dynamically adjusts the filter coefficients based on the phase difference between each unit of the microphone array; inputting the signal-to-noise ratio attenuation factor into a pre-trained convolutional recurrent neural network model to generate a frequency band compensation weight matrix; wherein the convolutional recurrent neural network model contains a convolutional layer and a recurrent neural network layer, and the frequency band compensation weight matrix contains gain compensation values corresponding to multiple sub-frequency bands; According to the frequency band compensation weight matrix, the primary noise reduction signal and the voice feature vector output by the bone conduction sensor are weighted and fused to generate the enhanced voice stream.

[0007] By using the above technical solution, the environmental noise energy and the user voice signal energy distribution are calculated by frequency energy distribution, and the signal-to-noise ratio attenuation factor of each sub-frequency band is accurately quantified; the adaptive beamforming algorithm dynamically optimizes the filter coefficients based on the microphone phase difference, and the primary noise reduction signal generated by spatial filtering effectively separates the target sound source; the pre-trained convolutional recurrent neural network generates a frequency band compensation weight matrix according to the signal-to-noise ratio attenuation factor, and intelligently compensates for the acoustic damage of a specific frequency band; Finally, the weighted processing of the fusion of the bone conduction sensor voice feature vector and the primary noise reduction signal is performed, which not only retains the naturalness of the voice, but also realizes the noise suppression of the full frequency band, especially improves the voice fidelity in the sudden noise scene.

[0008] Further settings are that according to the frequency band compensation weight matrix, the primary noise reduction signal and the voice feature vector output by the bone conduction sensor are weighted and fused to generate the enhanced voice stream step: If the signal-to-noise ratio attenuation factor of the current sub-frequency band is greater than a preset signal-to-noise ratio threshold, a higher fusion weight is assigned to the voice feature vector output by the bone conduction sensor; if it is less than or equal to the preset signal-to-noise ratio threshold, a higher fusion weight is assigned to the primary noise reduction signal.

[0009] By adopting the technical scheme, based on the sub-band weight distribution mechanism of the preset signal-to-noise ratio threshold, the bone conduction signal with strong anti-interference is preferentially used in the high-noise sub-band, and the air conduction speech details are mainly reserved in the low-noise sub-band; the strategy effectively avoids the spectral distortion caused by the traditional fixed weight fusion, so that the enhanced speech stream maintains the integrity of the speech characteristics and the auditory comfort in the dynamic noise environment such as a transportation hub and a restaurant.

[0010] Further, the S300 specifically includes the following steps: inputting the enhanced speech stream into a speech recognition model for processing to obtain a text sequence; wherein the speech recognition model is an end-to-end automatic speech recognition model, and the text sequence includes multiple words or phrases; inputting the text sequence into a semantic analysis model for processing to obtain an initial semantic label set; wherein the initial semantic label set includes multiple candidate semantic labels and an initial confidence score corresponding to each candidate semantic label; performing association optimization on the initial semantic label set based on a preset context database; wherein the context database stores a context keyword set of historical interaction records, and the association optimization includes calculating an average semantic similarity score of each candidate semantic label and all context keywords in the context keyword set; performing weighted fusion calculation on the initial confidence score and the average semantic similarity score according to a preset weight ratio to generate a final confidence weight; wherein the preset weight ratio is dynamically adjusted according to the current acoustic environment; selecting a candidate semantic label with a final confidence weight greater than a preset threshold, and combining the candidate semantic label with the corresponding final confidence weight to confirm as the semantic understanding result.

[0011] By adopting the technical scheme, the end-to-end speech recognition model converts the enhanced speech stream into a high-precision text sequence; the initial semantic label output by the semantic analysis model is associated and optimized by the context database, and the average semantic similarity score of the candidate label and the historical keyword is calculated; the dynamic weight calculation model of the initial confidence score and the semantic similarity score is fused, and the optimization direction is automatically adjusted according to the acoustic environment; the label selection mechanism with a final confidence weight greater than a threshold ensures that the output semantic understanding result has timeliness and scene adaptability, especially improves the understanding robustness of dialect and terminology scenes.

[0012] Further, the preset weight ratio is dynamically adjusted according to the current acoustic environment, and specifically includes the following steps: determining an environmental stability coefficient based on the frequency energy distribution extracted from the environmental acoustic signal by the environmental noise monitoring module; and determining a semantic focus degree by the discrete degree of the initial semantic label set; Adjust the preset weight proportion based on the environmental stability coefficient and the semantic focus degree.

[0013] By adopting the technical solution, the environmental stability coefficient calculated based on the environmental noise frequency domain energy distribution quantifies the background noise time-varying characteristics; the semantic focus degree determined by the discrete degree of the initial semantic label set evaluates the concentration of user intention expression; and the fusion weight of the semantic similarity score is cooperatively regulated by the two factors, thereby strengthening the context correlation optimization in a noisy environment and focusing on semantic ontology analysis in a quiet environment, so as to realize the dual adaptation of environmental noise and user expression habits.

[0014] Further, the S400 specifically includes the following steps: The final confidence weight corresponding to each candidate semantic label in the semantic understanding result is normalized to generate a normalized weight distribution coefficient; wherein the normalized weight distribution coefficient is calculated by dividing the final confidence weight of a single candidate semantic label by the sum of the final confidence weights of all candidate semantic labels; Based on the preset response generation rule library and the candidate semantic label, a matching search is performed to obtain an initial response content set; for each candidate response content in the initial response content set: If it maps a single candidate semantic label, the normalized weight distribution coefficient of the label is taken as the content confidence score; If it maps multiple candidate semantic labels, the maximum value of the normalized weight distribution coefficients of all associated candidate semantic labels is taken to generate the content confidence score; All candidate response contents are arranged in descending order of content confidence scores; According to the preset reference response quantity, a screening operation is performed: If the total number of candidate response contents is greater than the preset reference response quantity, the top N candidate response contents after sorting are selected to form a first response set; If the total number of candidate response contents is less than or equal to the preset reference response quantity, all candidate response contents are arranged in descending order to confirm the first response set; The candidate response contents in the first response set are converted into a voice stream in a preset audio format, and visual text data is synchronously generated and displayed and output through the user interface of the intelligent hearing aid earphone body.

[0015] By adopting the technical solution, the normalized weight distribution coefficient quantifies the decision contribution of each semantic label; the label matching mechanism of the response generation rule library generates an initial response content set; the content confidence score calculation model distinguishes between single-label direct mapping and multi-label maximum value strategy; and the screening mechanism arranged in descending order of confidence scores is combined with the preset reference response quantity to intelligently control the interface information load.

[0016] Further arrangement is that the S500 specifically includes steps of: When the user operates the interaction module of the smart hearing aid earphone body, a set of interaction signal parameters is collected in real time; wherein the set of interaction signal parameters includes an interaction start timestamp, a physical button pressure value, a touch screen contact point coordinate sequence, and an interaction end timestamp, and the interaction module includes a physical button unit and a touch screen unit; Based on the set of interaction signal parameters, a key behavior index is calculated; wherein the key behavior index includes: confirming the time difference between the interaction start timestamp and the interaction end timestamp as an interaction duration; performing trajectory fitting on the touch screen contact point coordinate sequence to generate a trajectory curvature radius and a trajectory displacement amount; extracting the peak value and the pressure change gradient of the physical button pressure value; According to the key behavior index, a preset interaction rule library is matched to determine an interaction state type; The interaction state type is bound to target candidate response content currently displayed on the user interface; wherein the target candidate response content is the candidate response content corresponding to the user operation area in the first response set; Interaction state data containing the interaction state type and the target candidate response content is output.

[0017] By adopting the above technical solution, the set of interaction signal parameters completely records the physical button pressure value, the touch trajectory coordinate sequence, and the timing information; the key behavior index quantization model converts the original signal into calculable features such as interaction duration, trajectory curvature radius, and pressure change gradient; the feature matching mechanism of the preset interaction rule library accurately identifies and confirms operation intentions such as selection, rejection response, and request details; the binding mechanism of the state type and the target candidate response content establishes a mapping relationship between operation behavior and semantic content, providing a decision basis for response dynamic optimization.

[0018] Further arrangement is that the S600 specifically includes steps of: According to the interaction state type, a preset response adjustment strategy table is matched to generate a dynamic adjustment parameter; wherein the response adjustment strategy table is stored in the storage unit of the smart hearing aid earphone body; the step of generating a dynamic adjustment parameter includes: If the interaction state type is a confirmation selection operation, all candidate semantic tags associated with the target candidate response content are extracted, the final confidence weight of each candidate semantic tag is multiplied by a preset enhancement coefficient to generate a tag weight promotion value; if the interaction state type is a rejection response operation, all candidate semantic tags associated with the target candidate response content are extracted, the final confidence weight of each candidate semantic tag is multiplied by a preset suppression coefficient to generate a tag weight suppression value; if the interaction state type is a request for details operation, based on the candidate semantic tags of the target candidate response content, an associated set of extended semantic tags is retrieved from a preset semantic database, and after the set of extended semantic tags is combined with the candidate semantic tags, an extended candidate response content operation is performed; Based on the dynamic adjustment parameter, the first response set is reconstructed to generate a second response set, including: If the candidate semantic tags associated with the candidate response content exist a tag weight promotion value, the content confidence score of the candidate response content is promoted based on the tag weight promotion value; if the candidate semantic tags associated with the candidate response content exist a tag weight suppression value, the content confidence score of the candidate response content is reduced based on the tag weight suppression value; if there is a new response content identifier, a new candidate response content is generated based on the set of extended semantic tags, and an initial content confidence score is assigned to the new content; The modified candidate response content and the new candidate response content are combined, and the content confidence score is rearranged from high to low to generate an update sequence, and the top K candidate response contents in the sorted order are selected to form a second response set, which is used for output and played on the audio output unit of the intelligent hearing aid earphone.

[0019] By adopting the above technical solution, the response adjustment strategy table maps the interaction state type to the tag weight promotion value, the suppression value and the set of extended semantic tags; the content confidence score can be re-adjusted based on the tag weight promotion value and the tag weight suppression value; the reordering and filtering output of the update sequence ensure that the second response set remains updated and has timeliness.

[0020] Further, the S700 specifically includes the steps of: The semantic tag statistics are performed on the audio output records of the second response set respectively to generate an actual response semantic distribution table containing the actual response frequency values of each candidate semantic tag; the semantic tag statistics are performed on the initial candidate contents of the first response set respectively to generate a benchmark response semantic distribution table containing the benchmark response frequency values of each candidate semantic tag: The semantic database modification operation is performed by traversing the actual response semantic distribution table: If the current candidate semantic label does not exist in the reference response semantic distribution table, the current candidate semantic label is added to the semantic database, and a preset initial confidence weight is assigned; if the current candidate semantic label exists in both the reference response semantic distribution table and the actual response semantic distribution table, and the actual response frequency value is greater than the reference response frequency value, a confidence enhancement operation is performed to obtain an enhancement amplitude, and the final confidence weight of the current candidate semantic label in the semantic database is increased by the enhancement amplitude; wherein the enhancement amplitude is calculated based on a response frequency growth rate and a current acoustic environment factor; If the current candidate semantic label exists in both the reference response semantic distribution table and the actual response semantic distribution table, and the actual response frequency value is less than the reference response frequency value, a confidence suppression operation is performed to obtain a suppression amplitude, and the final confidence weight of the current candidate semantic label in the semantic database is reduced by the suppression amplitude; wherein the enhancement amplitude is calculated based on a response frequency decay rate and a current acoustic environment factor; wherein the acoustic environment factor is calculated based on the frequency energy distribution of the environmental acoustic signal; For candidate semantic labels in the reference response semantic distribution table that are not covered by the actual response semantic distribution table, an attenuation operation is performed: if the candidate semantic label exists in the reference response semantic distribution table but does not exist in the actual response semantic distribution table, the final confidence weight of the semantic database is multiplied by a preset attenuation coefficient.

[0021] By using the above technical solution, the comparison and analysis of the actual response semantic distribution table and the reference response semantic distribution table quantifies the user preference changes; the confidence enhancement algorithm driven by the response frequency growth rate strengthens the decision weight of high-frequency selection labels; the confidence suppression mechanism triggered by the response frequency decay reduces the interference of inefficient labels; the environment factor weighted adjustment amplitude model ensures the environmental adaptability of the modification operation; the preset attenuation coefficient for non-response labels automatically optimizes the database performance, and finally realizes the continuous iteration of the semantic database based on the actual interaction data, greatly improving the personalized experience of long-term use.

[0022] The application also provides an intelligent hearing aid earphone semantic interaction system based on multi-modal perception, comprising the following modules: A multi-modal signal acquisition module configured to acquire user voice signals and environmental acoustic signals through a directional microphone array, a bone conduction sensor, and an environmental noise monitoring unit; A cooperative noise reduction processing module configured to perform cooperative noise reduction processing on the user voice signals and environmental acoustic signals to generate an enhanced voice stream, including generating a primary noise reduction signal using an adaptive beamforming algorithm, and generating a frequency band compensation weight matrix by calling a pre-trained convolutional recurrent neural network model; a semantic understanding module configured to generate a semantic understanding result based on the enhanced speech stream, the semantic understanding result comprising at least one candidate semantic label and a corresponding final confidence weight thereof; a response generation module configured to generate a first response set based on the semantic understanding result and a second response set based on a user interaction signal and an interaction state; an interaction processing module configured to analyze the interaction signal parameter set generated by the user operation interaction module, calculate a key behavior indicator, and determine an interaction state type; a model updating module configured to modify the semantic database based on the first response set and the second response set, including adjusting the confidence weight of the semantic label or adding a new semantic label; a signal output module configured to convert the first response set into visual text data and output the visual text data to a user interface, and convert the second response set into an audio stream and output the audio stream to an audio output unit; a data storage module configured to store the semantic database, a preset response generation rule library, and a response adjustment strategy table.

[0023] In summary, the present application has the following advantages: a closed-loop system capable of deeply integrating acoustic noise reduction, context-aware semantic analysis, and real-time interaction optimization and knowledge base dynamic evolution based on user feedback is provided, thereby achieving precise, smooth, and continuously evolving semantic interaction experience in complex and variable real environments, effectively improving user communication efficiency and quality of life. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 a main flowchart of an embodiment; Figure 2 a system block diagram of an embodiment. DETAILED DESCRIPTION

[0025] The present application will be further described in detail below with reference to the accompanying drawings.

[0026] As shown in the accompanying Figure 1 and Figure 2 ; The embodiment discloses a semantic interaction method of an intelligent hearing aid earphone based on multi-modal perception, comprising the following steps: S100, acquiring user speech signals and environmental acoustic signals through a multi-modal signal acquisition unit integrated in the body of the intelligent hearing aid earphone; wherein the multi-modal signal acquisition unit comprises a directional microphone array, a bone conduction sensor, and an environmental noise monitoring module; the specific steps comprise: This embodiment achieves high-precision synchronous acquisition of user voice signals and environmental acoustic signals through a multimodal signal acquisition unit integrated into the smart hearing aid earphone body. The smart hearing aid earphone body adopts an ergonomic ear-fitting design and embeds a directional microphone array at the ear canal entrance. This array consists of four ultra-miniature omnidirectional microphone units with a unit spacing of [missing information]. They are spatially distributed in an equilateral triangle pattern, and their operating frequency bands cover [areas]. The sampling rate is set to Hardware-level synchronization clocks ensure phase consistency of multi-channel signals. A bone conduction sensor (model BCS-4030) is integrated into the inner wall of the earphone, close to the temporal bone region. The angle between its vibration-sensitive axis and the normal direction of the skull is ≤5°. Piezoelectric ceramic material is used to achieve wide-band capture of vocal cord vibration signals (frequency response range). The environmental noise monitoring module consists of an independently configured wideband sound pressure meter (frequency response range). It consists of a microphone and a digital MEMS microphone, located in the non-blocking area on the outside of the earphone, at a distance from the central axis of the ear canal. It is installed with an opening to collect 360° omnidirectional ambient sound wave signals in real time.

[0027] During signal acquisition, the directional microphone array separates the target sound source using a space-time filtering algorithm: first, it generates a beam to form a control vector using generalized cross-correlation time delay estimation; then, it combines a minimum variance distortionless response algorithm to suppress interference noise from non-target directions, outputting a clean time-domain waveform of the user's speech signal. The bone conduction sensor employs triaxial accelerometer technology and utilizes an adaptive threshold triggering mechanism (trigger threshold). The system captures the skull vibration characteristics when the user speaks, converts them into a voltage signal via a charge amplifier, and outputs the fundamental frequency envelope characteristics. The environmental noise monitoring module executes this synchronously. Octave analysis, in Calculation within a time window between The equivalent continuous sound pressure level of each sub-band is used to generate a frequency domain energy distribution matrix.

[0028] S200. Perform collaborative noise reduction processing on the user's speech signal and the ambient acoustic signal to generate an enhanced speech stream; wherein, the collaborative noise reduction processing includes using an adaptive beamforming algorithm to filter and generate a primary noise-reduced signal, and using a pre-trained convolutional recurrent neural network model to generate a frequency band compensation weight matrix; the specific steps include: First, the frequency domain energy distribution of the environmental acoustic signal is extracted using an environmental noise monitoring module. This distribution covers a preset frequency band. within Each sub-band quantifies environmental noise characteristics through real-time sound pressure level measurements; specifically, the broadband sound pressure meter of the environmental noise monitoring module uses... The time window serves as the analysis unit, calculating the equivalent continuous sound pressure level for each sub-band and generating a frequency domain energy distribution matrix. Each element represents the sound pressure level value (in dB) of the corresponding sub-band, and the environmental noise energy of each sub-band is obtained based on this transformation. The formula is expressed as: ,in Sub-band Environmental noise energy, It is a time-frequency domain sound pressure signal. The length of the time window ( Real-time computation is achieved through Discrete Fourier Transform (DFT). Simultaneously, frequency-band energy calculation is performed on the raw speech signal acquired by the directional microphone array: the clean speech time-domain waveform output by the directional microphone array after space-time filtering is used, and the system employs a Short-Time Fourier Transform with a Hanning window, under the same... Decompose the signal within each sub-band and calculate the speech signal energy of each sub-band. This constitutes the user's voice signal energy distribution matrix, as shown in the formula. ,in For frequency point The spectral amplitude value.

[0029] Based on the above environmental noise energy and user voice signal energy The system calculates the signal-to-noise ratio attenuation factor for each sub-band. The formula is defined as: ,in It is a minimal constant (take) To avoid division by zero errors, the signal-to-noise ratio attenuation factor reflects the degree of noise attenuation of the speech signal and is used for subsequent compensation processing. Next, an adaptive beamforming algorithm is used to spatially filter the original speech signal: the algorithm dynamically adjusts the coefficients of the finite impulse response filter based on the phase difference between each unit of the microphone array (generated by generalized cross-correlation delay estimation). In the specific implementation, the filter order is set to... Order, sampling rate The beam pattern is optimized using the minimum variance distortionless response algorithm, and a primary noise-reduced signal is generated by solving the constraint optimization problem.

[0030] Subsequently, the signal-to-noise ratio attenuation factor Input a pre-trained convolutional recurrent neural network model to generate a bandgap compensation weight matrix: This model structure contains three convolutional layers (kernel size...). Step length (Activation function ReLU) and two layers of long short-term memory units (hidden number of units) The training data is based on a multi-scene noise database (including street, restaurant, wind noise, and other environmental noise), and the input is... dimension The vector outputs the gain compensation value for the corresponding sub-band. This forms the frequency band compensation weight matrix; the training process uses the mean squared error loss function, the optimizer is Adam, and the learning rate is... ,through The model validation set accuracy reached [percentage] after epoch training. Finally, the primary noise-reduced signal and the speech feature vector output by the bone conduction sensor are weighted and fused according to the frequency band compensation weight matrix: the fundamental frequency envelope feature vector output by the bone conduction sensor... (dimension) The primary noise-reduced signal is converted into frequency domain characteristics via STFT. The fusion formula is = The weighting coefficient and Depend on Dynamic allocation; the specific weight allocation rule is: if the current sub-band signal-to-noise ratio attenuation factor Greater than the preset signal-to-noise ratio threshold (set to) ),but , Prioritize enhancing bone conduction characteristics to combat high noise; if the signal-to-noise ratio is less than or equal to a preset threshold, then... , The primary noise reduction signal is prioritized to preserve speech quality; the fused signal is then reconstructed into an enhanced speech stream output via inverse STFT to ensure improved speech clarity in complex environments. That's all. The entire processing chain is executed in real time through a hardware acceleration module, with latency lower than [previous value]. This meets the low latency requirements of hearing aid applications.

[0031] S300. Obtain a semantic understanding result based on the enhanced speech stream; wherein, the semantic understanding result includes at least one candidate semantic label and its corresponding final confidence weight, the candidate semantic label corresponds to a context keyword, and the context keyword is stored in a preset semantic database; the specific steps include: In the S300 semantic understanding result generation stage, the system first processes the enhanced speech stream output from the S200 into the end-to-end automatic speech recognition model. This model employs an encoder-decoder structure based on the Conformer architecture, where the encoder contains... The layered convolutional layer enhances the self-attention module, and the decoder employs an attention-based beam search algorithm (beam width set to...). Enhance the speech stream to Frame length, Frame shifting is used for frame division processing, and the data is extracted using the Mel filter bank. The log-mel spectrogram features are input into the model, and a text sequence is output. The text sequence contains discrete units of words or phrases, such as when the user inputs the voice "Book a cardiologist for tomorrow morning at 10 o'clock", the model outputs the corresponding token sequence ["tomorrow", "morning", "10 o'clock", "book", "cardiologist"].

[0032] Subsequently, the text sequence is input into a semantic analysis model constructed based on natural language processing technology. The model is composed of a bidirectional long short-term memory network and a conditional random field layer. The bidirectional long short-term memory network layer is set up with hidden units, receives word embedding vectors (dimension ) of the text sequence as input, and the conditional random field layer is responsible for label decoding. The model performs dual-task learning of named entity recognition and intent classification: named entity recognition labels key entities in the text (such as time entity "tomorrow morning at 10 o'clock" and medical entity "cardiologist"), and intent classification outputs user behavior intent (such as "book an appointment"). Finally, an initial set of semantic labels is generated, including multiple candidate semantic labels and corresponding initial confidence scores. For example, the above text may output the candidate label set {"medical appointment": , "time specification": , "department selection": }, where the initial confidence score of the "medical appointment" label is (the highest value indicates absolute certainty).

[0033] Then, association optimization based on a preset context database is performed. The context database stores a set of context keywords from historical interaction records, and uses a graph database to organize data nodes and associated edges (nodes represent keywords, and edge weights reflect co-occurrence frequency). The optimization process calculates the average semantic similarity score of each candidate semantic label with the set of context keywords: first, the candidate semantic label and the context keyword are mapped to a dimension semantic vector through a pre-trained word vector model (such as Word2Vec); then the cosine similarity algorithm is used to calculate the similarity value of each candidate label with all context keywords; finally, the arithmetic mean of all similarity values is taken to obtain the average semantic similarity score. For example, if the current context contains the recent high-frequency words "appointment" and "cardiovascular", the semantic similarity of the "cardiology" label with these keywords can reach , which is much higher than the association degree of ordinary words. The initial confidence score and the average semantic similarity score are weighted and fused according to a dynamic weight ratio, and the weight adjustment mechanism is realized according to the frequency energy distribution and semantic focus degree extracted by the environmental noise monitoring module: the environmental stability coefficient is obtained by calculating the energy variance of the environmental acoustic signal in sub-bands ( ), and the weight ratio is adjusted according to the environmental stability coefficient and the semantic focus degree. (Identified as a high-perturbation environment); semantic focus is calculated using the information entropy of the initial semantic label set (entropy value < (This indicates semantic concentration). When the environmental stability coefficient exceeds a preset threshold (take... When this happens, the system will increase the weight of the average semantic similarity score to [a higher value]. (Basic weight) Strengthen contextual relevance to resist noise interference; when semantic focus is below a preset threshold, increase the initial confidence score weight to [a higher threshold]. To prevent misjudgment due to over-reliance on context, the weighted fusion formula is as follows: ;in Indicates normalization to The final confidence weight of the interval; Indicates the initial confidence score; This represents the average semantic similarity score; and These are the corresponding weights; for example, in a noisy restaurant environment (environmental stability coefficient is...). ), assigning the label "10:00 AM" , This causes its final confidence weight to change from Upgraded to The final system selection is based on a confidence weight greater than a preset threshold (take...). The candidate semantic labels of ("medical appointment") constitute the semantic understanding result. This result is stored as a vector, where each element contains the label name and its normalized final confidence weight. For example, the output vector [("medical appointment", ...] is... ("Time specification") (Department Selection) This vector will serve as the direct input for generating the S400 response. The entire S300 process is accelerated by a dedicated digital signal processor. It completes the conversion from speech to semantics, meeting the real-time interaction needs of hearing aids.

[0034] S400. Generate a first response set based on the semantic understanding result; wherein, the first response set includes at least one candidate response content, the candidate response content is generated based on candidate semantic tags, and the first response set is used to output and display on the user interface of the smart hearing aid headphone body; the specific steps include: In step S400, the system performs the following detailed process to generate the first response set: First, the final confidence weights corresponding to each candidate semantic label in the semantic understanding result are normalized to generate normalized weight allocation coefficients. Specifically, the normalized weight allocation coefficients are calculated using the formula: Let the first... The final confidence weights of the candidate semantic labels are: The sum of the final confidence weights of all candidate semantic labels is ,in The total number of candidate semantic tags; Indicates the first One candidate semantic label; normalized weight allocation coefficient This calculation ensures that the sum of the weight coefficients is 1, facilitating subsequent confidence allocation. For example, if the semantic understanding result contains three candidate semantic labels, their final confidence weights are respectively... , and The normalized weight coefficients are respectively , , .

[0035] Next, a matching retrieval is performed based on a pre-set response generation rule base and candidate semantic tags to obtain an initial set of response content. The pre-set response generation rule base is stored in a graph-structured database, where nodes represent candidate semantic tags (such as "medical appointment" and "time specification"), and edges represent the relationships between tags (weights reflect co-occurrence probabilities). The matching retrieval process is executed using a semantic similarity algorithm: a pre-trained word embedding model (such as Word2Vec) is used to map candidate semantic tags to... A dimensional vector is used to calculate its cosine similarity with each preset response template in the rule base; if the similarity exceeds a preset threshold... If the template is not matched, the candidate response content corresponding to that template is added to the initial response content set. For example, for the candidate semantic tag "medical appointment", the rule base may match the response template "What department do you want to book?", generating candidate response content. For each candidate response content in the initial response content set, the system calculates the content confidence score: if the candidate response content maps to only a single candidate semantic tag (e.g., the response "Appointment time is?" is only associated with the tag "Time specified"), then the normalized weight allocation coefficient of that tag is directly used as the content confidence score; if the candidate response content maps to multiple candidate semantic tags (e.g., the response "Confirm appointment with cardiologist?" is associated with both "medical appointment" and "department selection" tags), then the maximum value among the normalized weight allocation coefficients of all associated candidate semantic tags is taken to generate the content confidence score; for example, a response is associated with two tags (normalized weights are respectively... and ), then take The system assigns a content confidence score. Then, it sorts all candidate response content in descending order of content confidence score using a quicksort algorithm to ensure low latency (execution time less than [time value missing]). ). Perform a filtering operation based on the preset baseline response count: Preset baseline response count (Value) ), if the total number of candidate response contents is greater than , then the first response set is composed of the top candidate response contents in the sorted order; if the total number is less than or equal to , then the first response set is directly arranged in descending order. The screening process ensures that high-confidence content is prioritized, enhancing user experience. Finally, the candidate response contents in the first response set are converted into a voice stream in a pre-set audio format, and visual text data is synchronously generated. The conversion process uses a text-to-speech engine (such as the Tacotron2 model), inputs the candidate response content text, and outputs a PCM format voice stream with a sampling rate of and a bit depth of ; and the voice stream is displayed as an interactive list on the user interface of the smart hearing aid earphone body (such as an embedded LCD touch screen). The entire S400 process is supported by a hardware acceleration module, with an end-to-end delay controlled within , meeting real-time interaction requirements.

[0036] S500, obtaining a user interaction signal and obtaining an interaction state based on the user interaction signal; wherein the user interaction signal is a signal emitted when the user operates the interaction module of the smart hearing aid earphone body, and the interaction state reflects the feedback type of the user to the candidate response content; the specific steps include: In the S500 step, the system achieves accurate intent recognition by accurately collecting user interaction behavior data. When the user operates the interaction module of the smart hearing aid earphone body, the system synchronously collects the interaction signal parameter set through the physical button unit and the touch screen unit. The physical button unit uses a thin film pressure sensor array (range , resolution ) to record the physical button pressure value and its change curve over time in real time at a sampling rate of ; the touch screen unit (capacitive multi-point touch screen) continuously captures the contact point coordinate sequence, accurately recording the displacement trajectory of the fingertip in the XY plane. The interaction start timestamp and the interaction end timestamp are generated by the hardware interrupt triggering mechanism, with a time resolution of , completely recording the time span of a single interaction. The system calculates four key behavior indicators based on the original interaction signal parameter set: first, the difference between the interaction end timestamp and the interaction start timestamp is confirmed as the interaction duration (unit: milliseconds); second, the trajectory reconstruction is performed on the touch screen contact point coordinate sequence, the least squares method is used to fit the motion trajectory and calculate the trajectory curvature radius (unit: millimeters) and the trajectory displacement (unit: pixels); finally, the peak value (unit: Newton) of the physical button pressure value and the pressure change gradient (unit: Newton / millisecond) are extracted, where the pressure change gradient is calculated by the first derivative of the pressure curve.

[0037] The key behavior indicator is matched with a preset interaction rule library to determine an interaction state type; wherein: if the interaction duration is less than a first threshold and the physical button pressure value peak is greater than a preset pressure threshold, or the trajectory displacement is less than a displacement threshold and the trajectory curvature radius is greater than a curvature threshold, it is determined as a confirmation selection operation; if the interaction duration is greater than a second threshold and the pressure change gradient is a negative value, or the trajectory displacement is greater than the displacement threshold and the trajectory curvature radius is less than the curvature threshold, it is determined as a rejection response operation; if the interaction duration is between the first threshold and the second threshold and the pressure change gradient is a positive value, or the trajectory displacement is less than the displacement threshold and there is periodic coordinate oscillation, it is determined as a request detail operation. Specifically, when the interaction duration is less than and the physical button pressure value peak is greater than , or the trajectory displacement is less than pixels and the trajectory curvature radius is greater than , it is determined as a confirmation selection operation; when the interaction duration exceeds and the pressure change gradient is a negative value (the pressure continues to weaken), or the trajectory displacement is greater than pixels and the trajectory curvature radius is less than (sharp turning trajectory), it is determined as a rejection response operation; when the interaction duration is between and the pressure change gradient is a positive value (the pressure continues to increase), or the trajectory displacement is less than pixels but there is periodic coordinate oscillation (user hesitation light touch) with a frequency , it is determined as a request detail operation. In the determination process, sliding window variance analysis is used to eliminate sensor noise interference. Finally, the system binds the identified interaction state type to the target candidate response content currently displayed on the user interface. The binding mechanism is based on the clustering results of the touch screen contact point coordinate sequence, and the spatial positions of the user operation area and the interface display item are associated through a coordinate mapping algorithm. The output data uses a structured coding format, including an enumeration value of the interaction state type and a unique identifier of the target candidate response content, which is transmitted to the main processor. The entire S500 process is implemented on an embedded coprocessor to realize end-to-end delay, ensuring the real-time nature of the interaction response.

[0038] S600, generating a second response set based on the interaction state and the first response set; wherein the second response set includes at least one updated candidate response content, and the second response set is used for output and played on the audio output unit of the intelligent hearing aid earphone; the specific steps include: When the system obtains the interaction state data output by S500, S600 first matches the preset response adjustment strategy table according to the interaction state type. The response adjustment strategy table is stored in the storage unit of the intelligent hearing aid earphone body, and the data is organized in a key-value pair structure; wherein the key is the enumeration value of the interaction state type (such as "confirmation selection operation", "refusal response operation", "request details operation"), and the value is the corresponding adjustment rule parameter set. The matching process is realized by a hash lookup algorithm to ensure efficient response. For example, if the interaction state type is "confirmation selection operation", the system extracts the preset enhancement coefficient (typical value is ) in the response adjustment strategy table, and performs weight promotion calculation on all candidate semantic labels associated with the target candidate response content: the formula for calculating the label weight promotion value is ; wherein represents the final confidence weight of the corresponding candidate semantic label; represents the label weight promotion value; for example, when the of a candidate semantic label is ; if the interaction state type is "refusal response operation", the system extracts the preset suppression coefficient (typical value is ), and calculates the label weight suppression value: the formula is ; wherein, represents the label weight suppression value; for example, when the of a candidate semantic label is . For "request details operation", the system retrieves the associated expansion semantic label set from the preset semantic database based on the candidate semantic labels of the target candidate response content; the retrieval process uses a semantic similarity algorithm (the cosine similarity threshold is set to ), obtains the expansion label through graph traversal technology, and merges the expansion semantic label set with the original candidate semantic label to generate a new response content identifier, which is used for subsequent expansion candidate response content operation; this step takes an average of , and relies on real-time indexing optimization of the semantic database.

[0039] Based on the generated dynamic adjustment parameters, the system reconstructs the first response set output by S400 to generate a second response set. The reconstruction process involves three core sub-logics: first, for each candidate response content in the first response set, if the associated candidate semantic label has a label weight promotion value , the content confidence score of the candidate response content is increased by a certain proportion; the increase amplitude is calculated by the formula ; wherein represents the first scaling factor (value ); the updated score content confidence score is obtained by subtracting the original content confidence score from the increase amplitude . Secondly, if the candidate semantic label has a label weight inhibition value , the content confidence score of the candidate response content is proportionally reduced; the reduction amplitude ; wherein represents a second scaling factor (the value ); the updated score content confidence score is obtained by subtracting the original content confidence score from the decrease amplitude . Finally, if there is a new response content identifier, the system generates a new candidate response content based on the extended semantic label set; the generation rule uses a preset response generation rule library to match the extended label and assigns an initial content confidence score (the base value ). After the above modification, the system combines all candidate response contents (including the modified original content and the new content), and reorders them in descending order of content confidence score. Finally, the system selects the top candidate response contents (K is a preset value, and the value ) to form a second response set, which is played through an audio output unit (such as a speaker driven by a digital signal processor) in the format of a PCM voice stream with a sampling rate of , and the user interface is updated synchronously to a scrollable list.

[0040] S700、based on the first response set and the second response set modify the semantic database; wherein the modification process includes adjusting the confidence weight corresponding to the semantic label or adding a new semantic label; the specific steps include: First, the system performs semantic label statistics on the audio output record of the second response set to generate an actual response semantic distribution table. Extract the candidate semantic labels (such as "medical appointment" and "time specification") corresponding to all played candidate response contents in the second response set, and accumulate the occurrence frequency of the same candidate semantic labels to generate the actual response frequency value of each candidate semantic label. This process uses a hash table data structure to achieve efficient counting: traverse each candidate response content in the second response set, parse its bound candidate semantic label list; for each label, if it does not exist in the hash table, initialize it to , if it exists, increment the count value; finally, an actual response semantic distribution table containing candidate semantic labels and their corresponding actual response frequency values is established, for example, the actual response frequency value of the label "medical appointment" may be . This process is executed in real time by an embedded processor, with a time delay controlled within Within this system, data timeliness is ensured. Simultaneously, the system performs similar statistical analysis on the initial candidate content of the first response set to generate a baseline response semantic distribution table. Candidate semantic tags corresponding to all initial candidate response content in the first response set are extracted, and the frequency of occurrence of the same tag is accumulated to generate a baseline response frequency value. In the implementation, the system caches tag frequency data when generating the first response set; here, the cache is read directly. If the cache is not read, the content of the first response set is re-parsed, and the same hash algorithm as above is used for counting. The baseline response semantic distribution table is output; for example, the baseline response frequency value for the tag "time specified" is... (Indicates that it appears in the first response set) (times). Both distribution tables are stored in JSON format for easy subsequent traversal operations.

[0041] Next, the system traverses the actual response semantic distribution table, performing semantic database modification operations item by item. The traversal process uses a depth-first search algorithm to access each candidate semantic tag item: If the current candidate semantic label does not appear in the baseline response semantic distribution table (i.e., the label is a new item), the system adds the label to the preset semantic database and assigns it a preset initial confidence weight (typical value is...). For example, when the "emergency registration" label appears in the actual distribution table but is missing in the baseline table, the system creates a new entry in the semantic database with an initial weight of [value missing]. If the current candidate semantic label exists in both distribution tables, and its actual response frequency is greater than the baseline response frequency, the system performs a confidence enhancement operation: calculating the response frequency growth ratio, using the formula: response frequency growth ratio. ;in, This represents the actual response frequency value; This represents the reference response frequency value. The enhancement amplitude is adjusted based on the current acoustic environmental factors (calculated in real-time by the environmental noise monitoring module). The acoustic environmental factors are calculated from the frequency domain energy distribution of the environmental acoustic signal: the system acquires... within the frequency band Energy data for each sub-frequency band is used to calculate its variance, which is then used as an environmental stability coefficient (e.g., variance value). (Representing a high-noise environment); the formula for calculating the enhancement amplitude is the enhancement amplitude. ;in The environmental stability coefficient (calculated from the frequency domain energy variance of the environmental acoustic signal, unit: ...) ); This represents the preset third scaling factor (value). For example, the actual frequency of the tag "department selection" is... The reference frequency is The growth rate is If the environmental stability coefficient is If the actual response frequency value is greater than the baseline response frequency value, the system performs a confidence boost operation: it calculates a response frequency boost ratio, which is given by ; the system increases the final confidence weight of the tag in the semantic database by a boost amplitude , e.g., from to . If the actual response frequency value is less than the baseline response frequency value, the system performs a confidence suppression operation: it calculates a response frequency decay ratio, which is given by ; the suppression amplitude is . For example, the tag "appointment confirmation" has an actual frequency of , a baseline frequency of , and a decay ratio of ; the suppression amplitude is ; the system decreases the final confidence weight of the tag in the semantic database from to . Finally, the system performs a decay operation on candidate semantic tags in the baseline response semantic distribution table that are not covered by the actual response semantic distribution table. It iterates through the baseline distribution table: if a candidate semantic tag exists in the baseline table but is absent from the actual table (i.e., the tag was not selected by the user), the system multiplies its final confidence weight in the semantic database by a preset decay coefficient (a typical value is ). For example, the tag "doctor query" has a frequency of in the baseline table but is absent from the actual table, and its weight is decayed from to . The entire S700 process is executed in parallel by a dedicated hardware module, with an end-to-end delay of no more than , ensuring real-time updating of the semantic database and thus optimizing the performance of subsequent semantic understanding.

[0042] The embodiment also discloses an intelligent hearing aid earphone semantic interaction system based on multi-modal perception, comprising the following modules: a multi-modal signal acquisition module configured to acquire user voice signals and environmental acoustic signals through a directional microphone array, a bone conduction sensor, and an environmental noise monitoring unit; a collaborative noise reduction processing module configured to perform collaborative noise reduction processing on the user voice signals and the environmental acoustic signals to generate an enhanced voice stream, including generating a primary noise reduction signal using an adaptive beamforming algorithm and generating a frequency band compensation weight matrix by calling a pre-trained convolutional recurrent neural network model; a semantic understanding module configured to generate a semantic understanding result based on the enhanced voice stream, the semantic understanding result comprising at least one candidate semantic tag and a corresponding final confidence weight thereof; a response generation module configured to generate a first response set based on the semantic understanding result and a second response set based on user interaction signals and an interaction state; An interaction processing module is configured to analyze the interaction signal parameter set generated by the user operation interaction module, calculate the key behavior index, and determine the interaction state type; A model updating module is configured to modify the semantic database based on the first response set and the second response set, including adjusting the confidence weight of the semantic label or adding a new semantic label; A signal output module is configured to convert the first response set into visual text data and output to the user interface, and convert the second response set into an audio stream and output to the audio output unit. A data storage module is configured to store the semantic database, the preset response generation rule library, and the response adjustment strategy table.

[0043] The specific embodiments are only an explanation of the present application, which is not a limitation of the present application. Those skilled in the art can make modifications to the embodiments without creative contribution after reading the specification, as long as the modifications are within the scope of the claims of the present application.

Claims

1. A semantic interaction method for intelligent hearing aid headphones based on multimodal perception, characterized in that, Includes the following steps: S100: Acquire user voice signals and environmental acoustic signals through a multimodal signal acquisition unit integrated into the smart hearing aid headphone body; wherein, the multimodal signal acquisition unit includes a directional microphone array, a bone conduction sensor, and an environmental noise monitoring module; S200. Perform collaborative noise reduction processing on the user's voice signal and the ambient acoustic signal to generate an enhanced voice stream; wherein, the collaborative noise reduction processing includes using an adaptive beamforming algorithm to filter and generate a primary noise reduction signal, and using a pre-trained convolutional recurrent neural network model to generate a frequency band compensation weight matrix. S300. Obtain a semantic understanding result based on the enhanced speech stream; wherein, the semantic understanding result includes at least one candidate semantic label and its corresponding final confidence weight, the candidate semantic label corresponds to a context keyword, and the context keyword is stored in a preset semantic database; S400. Generate a first response set based on the semantic understanding result; wherein, the first response set includes at least one candidate response content, the candidate response content is generated based on candidate semantic tags, and the first response set is used to output and display on the user interface of the smart hearing aid headphone body; S500. Acquire user interaction signals and obtain interaction status based on the user interaction signals; wherein, the user interaction signals are signals emitted when the user operates the interaction module of the smart hearing aid headphone body, and the interaction status reflects the type of feedback from the user to the candidate response content. S600. Generate a second response set based on the interaction state and the first response set; wherein, the second response set includes at least one updated candidate response content, and the second response set is used to output and play in the audio output unit of the smart hearing aid headphones; S700. Modify the semantic database based on the first response set and the second response set; wherein, the modification process includes adjusting the confidence weights corresponding to the semantic tags or adding new semantic tags.

2. The semantic interaction method for intelligent hearing aid headphones based on multimodal perception according to claim 1, characterized in that: S200 specifically includes the following steps: The frequency domain energy distribution of the environmental acoustic signal is extracted by the environmental noise monitoring module; wherein, the frequency domain energy distribution includes sound pressure level measurements of multiple sub-bands within a preset frequency range, and the environmental noise energy of each sub-band is obtained based on the frequency domain energy distribution. The raw speech signal acquired by the directional microphone array is divided into frequency bands for energy calculation to generate the user speech signal energy distribution; based on the environmental noise energy and the user speech signal energy distribution, the signal-to-noise ratio attenuation factor of each sub-band is calculated. An adaptive beamforming algorithm is used to spatially filter the raw speech signal acquired by the directional microphone array to generate a primary noise reduction signal; wherein, the adaptive beamforming algorithm dynamically adjusts the filter coefficients based on the phase difference between each unit of the microphone array. The signal-to-noise ratio attenuation factor is input into a pre-trained convolutional recurrent neural network model to generate a frequency band compensation weight matrix; wherein, the convolutional recurrent neural network model includes convolutional layers and recurrent neural network layers, and the frequency band compensation weight matrix includes gain compensation values ​​corresponding to multiple sub-frequency bands. Based on the frequency band compensation weight matrix, the primary noise reduction signal and the speech feature vector output by the bone conduction sensor are weighted and fused to generate the enhanced speech stream.

3. The semantic interaction method for intelligent hearing aid headphones based on multimodal perception according to claim 2, characterized in that: In the step of generating the enhanced speech stream, the primary noise reduction signal and the speech feature vector output by the bone conduction sensor are weighted and fused according to the frequency band compensation weight matrix. If the signal-to-noise ratio attenuation factor of the current sub-band is greater than the preset signal-to-noise ratio threshold, then a higher fusion weight is assigned to the speech feature vector output by the bone conduction sensor; if it is less than or equal to the preset signal-to-noise ratio threshold, then a higher fusion weight is assigned to the primary noise reduction signal.

4. The semantic interaction method for intelligent hearing aid headphones based on multimodal perception according to claim 1, characterized in that: S300 specifically includes the following steps: The enhanced speech stream is input into a speech recognition model for processing to obtain a text sequence; wherein, the speech recognition model is an end-to-end automatic speech recognition model, and the text sequence includes multiple words or phrases; The text sequence is input into a semantic analysis model for processing to obtain an initial semantic label set; wherein, the initial semantic label set includes multiple candidate semantic labels and an initial confidence score corresponding to each candidate semantic label; The initial semantic tag set is subjected to association optimization based on a preset context database; wherein, the context database stores a set of context keywords of historical interaction records, and the association optimization includes calculating the average semantic similarity score between each candidate semantic tag and all context keywords in the context keyword set; The initial confidence score and the average semantic similarity score are weighted and fused according to a preset weight ratio to generate the final confidence weight; wherein, the preset weight ratio is dynamically adjusted according to the current acoustic environment; Candidate semantic labels with a final confidence weight greater than a preset threshold are selected and combined with their corresponding final confidence weights to confirm the semantic understanding result.

5. The semantic interaction method for intelligent hearing aid headphones based on multimodal perception according to claim 4, characterized in that: The preset weight ratio is dynamically adjusted according to the current acoustic environment, specifically including the following steps: The environmental stability coefficient is determined based on the frequency domain energy distribution extracted from the environmental acoustic signal by the environmental noise monitoring module; the semantic focus is determined by the discreteness of the initial semantic tag set. The preset weight ratio is adjusted based on the environmental stability coefficient and semantic focus.

6. The semantic interaction method for intelligent hearing aid headphones based on multimodal perception according to claim 1, characterized in that: The S400 specifically includes the following steps: The final confidence weights corresponding to each candidate semantic label in the semantic understanding result are normalized to generate normalized weight allocation coefficients; wherein, the normalized weight allocation coefficients are calculated by dividing the final confidence weight of a single candidate semantic label by the sum of the final confidence weights of all candidate semantic labels. Based on a preset response generation rule base and the candidate semantic tags, a matching retrieval is performed to obtain an initial response content set; for each candidate response content in the initial response content set: If it maps to a single candidate semantic label, then the normalized weight allocation coefficient of that label is used as the content confidence score. If it maps to multiple candidate semantic tags, then the maximum value among the normalized weight allocation coefficients of all associated candidate semantic tags is taken to generate the content confidence score. All candidate response contents are sorted in descending order of content confidence score; Perform a filtering operation based on a preset baseline number of responses: If the total number of candidate response contents is greater than the preset baseline number of responses, then the first N candidate response contents after sorting are selected to form the first response set. If the total number of candidate response contents is less than or equal to the preset baseline response quantity, then all candidate response contents are sorted in descending order and confirmed as the first response set; The candidate response content in the first response set is converted into a speech stream in a preset audio format, and visualized text data is generated simultaneously and displayed through the user interface of the smart hearing aid headphone.

7. The semantic interaction method for intelligent hearing aid headphones based on multimodal perception according to claim 1, characterized in that: The S500 specifically includes the following steps: When the user operates the interaction module of the smart hearing aid headphone, the interaction signal parameter set is collected in real time; wherein, the interaction signal parameter set includes the interaction start timestamp, physical button pressure value, touch screen contact point coordinate sequence and interaction end timestamp, and the interaction module includes a physical button unit and a touch screen unit. Key behavioral indicators are calculated based on the set of interaction signal parameters; wherein, the key behavioral indicators include: determining the time difference between the interaction start time stamp and the interaction end time stamp as the interaction duration; performing trajectory fitting on the touch screen contact point coordinate sequence to generate the trajectory curvature radius and trajectory displacement; and extracting the peak value and pressure change gradient of the physical button pressure value. The interaction state type is determined by matching the key behavioral indicators with a preset interaction rule base. The interaction state type is bound to the target candidate response content currently displayed on the user interface; wherein, the target candidate response content is the candidate response content in the first response set corresponding to the user operation area; Output interaction state data that includes the interaction state type and the target candidate response content.

8. The semantic interaction method for intelligent hearing aid headphones based on multimodal perception according to claim 7, characterized in that: The S600 specifically includes the following steps: Based on the interaction state type, a preset response adjustment strategy table is matched to generate dynamic adjustment parameters; wherein, the response adjustment strategy table is stored in the storage unit of the smart hearing aid headphone body; the step of generating dynamic adjustment parameters includes: If the interaction state type is a confirmation selection operation, then all candidate semantic tags associated with the target candidate response content are extracted, and the final confidence weight of each candidate semantic tag is multiplied by a preset enhancement coefficient to generate a tag weight enhancement value; if the interaction state type is a rejection response operation, then all candidate semantic tags associated with the target candidate response content are extracted, and the final confidence weight of each candidate semantic tag is multiplied by a preset suppression coefficient to generate a tag weight suppression value; if the interaction state type is a request details operation, then based on the candidate semantic tags of the target candidate response content, an associated extended semantic tag set is retrieved from a preset semantic database, the extended semantic tag set is merged with the candidate semantic tags, and then the extended candidate response content operation is executed. Based on the dynamically adjusted parameters, the first response set is reconstructed to generate a second response set, including: If the candidate semantic tags associated with the candidate response content have tag weight boosting values, the content confidence score of the candidate response content is increased based on the tag weight boosting values; if the candidate semantic tags associated with the candidate response content have tag weight suppression values, the content confidence score of the candidate response content is decreased based on the tag weight suppression values; if there is a new response content identifier, a new candidate response content is generated based on the extended semantic tag set, and an initial content confidence score is assigned to the new content. The revised candidate response content is merged with the newly added candidate response content, and then rearranged according to the content confidence score from high to low to generate an updated sequence. The top results of the sorted sequence are then selected. The candidate response content forms a second response set, which is used to output and play the audio output unit of the smart hearing aid headphones.

9. A semantic interaction method for intelligent hearing aids based on multimodal perception according to claim 8, characterized in that: The S700 specifically includes the following steps: Semantic tag statistics are performed on the audio output records of the second response set to generate an actual response semantic distribution table containing the actual response frequency values ​​of each candidate semantic tag; semantic tag statistics are performed on the initial candidate content of the first response set to generate a baseline response semantic distribution table containing the baseline response frequency values ​​of each candidate semantic tag. Iterate through the actual response semantic distribution table and perform semantic database modification operations: If the current candidate semantic label does not appear in the baseline response semantic distribution table, then the current candidate semantic label is added to the semantic database and assigned a preset initial confidence weight. If the current candidate semantic label exists in both the baseline response semantic distribution table and the actual response semantic distribution table, and its actual response frequency value is greater than the baseline response frequency value, then a confidence enhancement operation is performed to obtain the enhancement magnitude, and the final confidence weight of the current candidate semantic label in the semantic database is increased by the enhancement magnitude; wherein, the enhancement magnitude is calculated based on the response frequency growth rate and the current acoustic environment factors; If the current candidate semantic label exists in both the baseline response semantic distribution table and the actual response semantic distribution table, and its actual response frequency value is less than the baseline response frequency value, then a confidence suppression operation is performed to obtain the suppression magnitude, and the final confidence weight of the current candidate semantic label in the semantic database is reduced by the suppression magnitude; wherein, the enhancement magnitude is calculated based on the response frequency attenuation ratio and the current acoustic environment factor; wherein, the acoustic environment factor is calculated based on the frequency domain energy distribution of the ambient acoustic signal; Perform a decay operation on candidate semantic labels in the baseline response semantic distribution table that are not covered by the actual response semantic distribution table: if a candidate semantic label exists in the baseline response semantic distribution table but does not appear in the actual response semantic distribution table, multiply its final confidence weight in the semantic database by a preset decay coefficient.

10. A semantic interaction system for intelligent hearing aids based on multimodal perception, applied to the semantic interaction method for intelligent hearing aids based on multimodal perception as described in any one of claims 1-9, characterized in that, It includes the following modules: The multimodal signal acquisition module is configured to acquire user voice signals and environmental acoustic signals through a directional microphone array, bone conduction sensor and environmental noise monitoring unit; The collaborative noise reduction processing module is configured to perform collaborative noise reduction processing on the user's speech signal and the ambient acoustic signal to generate an enhanced speech stream, including using an adaptive beamforming algorithm to generate a primary noise reduction signal and calling a pre-trained convolutional recurrent neural network model to generate a frequency band compensation weight matrix. The semantic understanding module is configured to generate a semantic understanding result based on the enhanced speech stream, wherein the semantic understanding result includes at least one candidate semantic label and its corresponding final confidence weight. The response generation module is configured to generate a first response set based on the semantic understanding results, and to generate a second response set based on user interaction signals and interaction states; The interaction processing module is configured to parse the set of interaction signal parameters generated by the user's operation interaction module, calculate key behavioral indicators, and determine the interaction state type. The model update module is configured to modify the semantic database based on the first response set and the second response set, including adjusting the confidence weight of semantic labels or adding new semantic labels; The signal output module is configured to convert the first response set into visual text data and output it to the user interface, and to convert the second response set into an audio stream and output it to the audio output unit; The data storage module is configured to store a semantic database, a preset response generation rule base, and a response adjustment strategy table.

Images