Intelligent sound box voice processing method and system based on artificial intelligence

Abstract

The invention provides an intelligent sound box voice processing method and system based on artificial intelligence, and the method comprises the steps: obtaining audio data and mouth shape video data, carrying out the processing of the audio data and the mouth shape video data, and carrying out the multi-modal feature fusion, and obtaining a fusion feature; performing bimodal voice activity detection on the fusion features to obtain effective voice data; performing context sensing recognition of audio and video fusion on the effective voice data to obtain a first text; constructing a user feature model, and performing semantic understanding on the text based on the model to obtain an understanding result; performing intention recognition and slot filling based on the understanding result to obtain user intention and key information; generating a response strategy in combination with the user intention, the key information and the environment perception data; generating response voice according to the response strategy; and monitoring feedback information of the user to the response voice in real time, and updating the user feature model and the response strategy evaluation model based on feedback. According to the scheme, the voice can be recognized more accurately, and the safety and robustness of the system are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of voice processing technology, and in particular to an artificial intelligence-based smart speaker voice processing method and system. Background Art

[0002] Currently, smart speaker products have been widely deployed in the market. However, their core technologies still have room for improvement in key indicators such as recognition accuracy, security, and anti-interference capabilities. In particular, the voice recognition accuracy of existing solutions is low in noisy environments. Summary of the Invention

[0003] Based on the above problems, the present invention proposes an artificial intelligence-based smart speaker voice processing method and system. Through the solution of the present invention, the smart speaker can recognize voice more accurately in noisy environments and enhance the security and robustness of the system.

[0004] In view of this, one aspect of the present invention proposes an intelligent speaker voice processing method based on artificial intelligence, comprising: Acquire multiple channels of initial audio data and user lip movement video data, and pre-process the initial audio data and the lip movement video data to obtain first audio data and first lip movement data, respectively; Performing multimodal feature fusion on the first audio data and the first lip movement data to obtain a first multimodal fusion feature; Performing bimodal voice activity detection on the first multimodal fusion feature to obtain first valid voice data; Performing context-aware speech recognition with audio and video fusion on the first valid speech data to obtain a first text; Constructing a first user feature model, and performing semantic understanding on the first text based on the first user feature model to obtain a first understanding result; Performing intent recognition and slot filling based on the first understanding result to obtain a first user intent and first key information; generating a first response strategy based on the first user intention and the first key information in combination with environmental perception data; generating a first response voice according to the first response strategy; The user's feedback information on the first response voice is monitored in real time, and the first user feature model and the first response strategy evaluation model are updated based on the feedback information.

[0005] Optionally, the step of performing bimodal voice activity detection on the first multimodal fusion feature to obtain first valid voice data includes: Performing temporal alignment and calibration on the audio acoustic features and visual mouth-motion features in the first multimodal fusion features to eliminate the difference in audio and video acquisition delays; Extract high-level features at the semantic level, and semantically encode the fused features through a pre-trained language model to form a semantic feature vector; Construct an acoustic attention submodule to calculate the attention weight distribution of audio spectrum features in the time-frequency domain, focusing on the frequency bands and time periods related to speech activity; Construct a visual attention submodule to calculate the spatial-temporal attention weights of visual features based on lip movement intensity and mouth shape change amplitude; Construct a semantic attention submodule to calculate the semantic-level attention weights using the contextual relevance of semantic features; Design a cross-modal attention fusion mechanism that dynamically balances the contributions of the three attention sub-modules through learnable weight parameters to form a unified multimodal attention representation; The multimodal attention representation is input into a binary classification decision network, wherein the decision network adopts a multi-layer perceptron structure; The classification results are judged based on a preset confidence threshold and the speech activity probability score for each time frame is output; Applying a time series smoothing filter algorithm to smooth the voice activity probability score, eliminating short-term fluctuations and avoiding frequent voice activity state switching, thereby forming a continuous voice activity probability sequence; Identifying consecutive high-probability intervals in the voice activity probability sequence as candidate voice segments, wherein the high-probability intervals are time periods in which the probability value is continuously higher than a preset activation threshold; Performing boundary refinement processing on the candidate speech segment, and fine-tuning the start and end positions of the speech segment using the joint features of audio energy and visual motion; Short speech segments whose duration is lower than a preset minimum threshold are filtered, and a final valid speech segment sequence is output as the first valid speech data.

[0006] Optionally, the step of performing context-aware speech recognition with audio and video fusion on the first valid voice data to obtain the first text includes: Extracting a first audio acoustic feature and a first visual mouth-motor feature within a time window corresponding to the first valid speech data from the first multimodal fusion feature; Performing a multi-scale analysis on the first audio acoustic feature to extract a first audio acoustic feature vector including a Mel-frequency cepstral coefficient, a fundamental frequency contour, and a spectral centroid; Extracting a first visual mouth movement feature vector including lip shape geometric features, motion trajectory features, and mouth shape change dynamic features from the first visual mouth movement features; Mapping the first audio acoustic feature vector and the first visual mouth movement feature vector to a unified feature space through a linear transformation layer, and organizing them into a first audio acoustic feature sequence and a first visual mouth movement feature sequence in chronological order, respectively, to ensure dimensional consistency in subsequent multimodal fusion processing; Construct an audio encoder and use a multi-head self-attention mechanism to encode the first audio acoustic feature sequence to capture the temporal dependency of acoustic features; Construct a visual encoder that uses a combination of convolutional neural networks and recurrent neural networks to encode the first visual mouth-movement feature sequence and extract the spatiotemporal representation of visual speech; Design a cross-modal fusion layer that uses the audio encoder output as the audio modality input and the visual encoder output as the visual modality input. A cross-attention mechanism is used to achieve bidirectional information interaction between audio acoustic features and visual mouth-motion features, generating a fused multimodal representation. Constructing a decoder network to receive the multimodal representation and fuse phoneme-level, word-level, and sentence-level language knowledge to output character-level or word-level recognition probability distribution; Extract user interaction history, including recent conversation content, common vocabulary, and expression habits, to build a personalized language model; Calculating a context prior probability based on the personalized language model and performing Bayesian correction on the recognition probability distribution; Combined with the contextual information of the current dialogue turn, the conditional random field model is used to optimize the sequence annotation of the recognition results to improve semantic coherence; Based on the sequence annotation optimization results, the first recognition sequence is obtained by combining the knowledge graph to enhance semantic understanding; Convert the first recognition sequence into continuous text, perform punctuation prediction and case normalization processing, and obtain the initial text; Calculating the confidence score of the initial text and performing a comprehensive evaluation based on three dimensions: acoustic model confidence, language model confidence, and multimodal consistency; Output the initial text and its confidence score as the first text.

[0007] Optionally, the step of constructing a first user feature model and performing semantic understanding on the first text based on the first user feature model to obtain a first understanding result includes: Extract users' historical interaction records, voice habit parameters, preference data, and usage scenario information, and construct a multi-dimensional user feature set; Building an adaptive user feature model based on the multi-dimensional user feature set and the federated learning framework; Inputting the first text into the adaptive user feature model to generate personalized adjustment parameters for the current user; In combination with the personalized adjustment parameters, performing deep semantic analysis on the first text to extract a semantic representation vector of the text; Based on the user's time usage pattern and environmental context characteristics, the context weight of the semantic representation vector is adjusted to enhance the understanding of the user's intention in a specific scenario; The user's common vocabulary features are used to provide personalized interpretations of the unregistered words and professional terminology components in the adjusted semantic representation vector, and the possible meanings of new words are inferred based on the user's historical usage records. Apply user expression pattern features to perform consistency check on the semantic representation vector after personalized interpretation, generate the final personalized semantic understanding result, and ensure that the interpretation result conforms to the user's language habits and expression logic; Performing a weighted fusion of the personalized semantic understanding result and the standard semantic understanding result based on the original initial text to calculate a comprehensive confidence score of the personalized semantic understanding; Perform semantic integrity checks on the fused semantic understanding results to identify and repair semantic inconsistencies caused by personalized adjustments. Based on user feedback history, the semantic understanding results after the completeness check are finally optimized to adjust the interpretation tendency of uncertain semantic components; Output the final optimized semantic understanding result and its confidence evaluation as the first understanding result.

[0008] Optionally, the step of performing intent recognition and slot filling based on the first understanding result to obtain the first user intent and first key information includes: Preprocess and extract features from the first understanding result, and output a multi-dimensional semantic feature vector; Design a shared encoder network to map the multi-dimensional semantic feature vector into a unified shared latent layer representation, providing a common feature basis for intent recognition and slot filling tasks; Constructing an intent recognition branch network, using a multi-layer perceptron structure to classify the shared hidden layer representation, and outputting a probability distribution vector of the user's intent; Constructing a slot filling branch network, using a sequence labeling structure of a conditional random field combined with a bidirectional long short-term memory network to predict slot labels on the shared hidden layer representation to obtain a slot label sequence; Design a joint loss function that combines the cross-entropy loss of intent recognition and the sequence labeling loss of slot filling, improving the collaborative performance of the two tasks through multi-task joint optimization. Conduct uncertainty modeling and confidence assessment to obtain comprehensive confidence assessment results; Based on the slot label sequence, a recurrent neural network is used to generate slot filling results; Based on the slot filling results, identify the complex command structure containing multiple sub-intents and perform hierarchical decomposition of the intent; For each sub-intent in the hierarchical intent structure, the intent recognition process is re-executed to obtain the hierarchical intent recognition result; Verify the logical consistency of each slot in complex commands based on slot dependencies, and identify and fix conflicting slot values; Build a command integrity check mechanism to ensure that the necessary slots of complex commands are fully filled, mark missing slots and generate a completion strategy to obtain a complete slot filling result; Integrate the hierarchical intent recognition results and the complete slot filling results to generate a structured intent representation containing the main intent, sub-intent, and corresponding slot values; Extract key information based on the structured intent representation, mark the main intent and sub-intent as a primary intent category and a secondary intent category, respectively, and extract entity values ​​and parameter values ​​in the slot filling result as key entity values ​​and operation parameters, respectively; Confidence annotation is performed on the structured intent representation and the key information, and based on the comprehensive confidence evaluation result, a corresponding credibility score is assigned to each intent category and key information element; Output the first user intention and the first key information that have been labeled with confidence.

[0009] Optionally, the step of generating a first response strategy based on the first user intention and the first key information in combination with environmental perception data includes: Collect environmental perception data and perform fusion processing to obtain comprehensive environmental perception data; Perform correlation analysis between the user intention and the environmental data based on the first user intention, the first key information, and the comprehensive environmental perception data to obtain an intention execution plan; Decision modeling based on knowledge graph enhancement to obtain the optimal decision strategy; In combination with the comprehensive environmental perception data, adjusting the response priority of the optimal decision strategy to obtain a response priority scheme; generating a collaborative control strategy based on the graph neural network and the response priority scheme; Integrating the optimal decision-making strategy, the response priority scheme, and the collaborative control strategy to construct a multi-level comprehensive response strategy framework; Verify the feasibility of the comprehensive response strategy framework, check the logical consistency, resource availability and time feasibility of the strategy, and identify potential execution risks; Based on the verification results, a complete response strategy set including the main execution strategy, alternative strategies and exception handling strategies is generated to ensure the robustness of strategy execution; Output the final first response strategy, including device control instruction sequence, execution timing arrangement, content importance weight, urgency identification, expected effect evaluation and exception handling mechanism.

[0010] Optionally, the step of generating a first response voice according to the first response strategy includes: Based on the emotional tags and semantic content in the first response strategy, a variational autoencoder is used to encode the response text into acoustic feature vectors and emotional feature vectors, thereby achieving decoupled representation of acoustic information and emotional information; According to the personalized parameters output by the first user feature model and the current interaction scenario, the emotion feature vector is modulated by a generative adversarial network to generate a target emotion feature that meets the user's preference; Inputting the target emotional feature and the acoustic feature vector into a preset prosody model, dynamically calculating the fundamental frequency, duration, and energy parameters of the speech, and adjusting the rhythm, stress, and intonation of the speech; Based on the content importance weight and urgency indicator in the first response strategy, an adaptive speech rate adjustment algorithm is used to calculate the target duration ratio of each voice segment, and the duration parameters output by the prosody model are dynamically adjusted according to the target duration ratio to achieve differentiated speech rate control, where important information is expressed slowly and secondary information is conveyed quickly. The adjusted acoustic features, emotional features and prosodic parameters are input into the neural vocoder to synthesize and output a natural speech signal with personalized emotional expression.

[0011] Optionally, the multimodal feature fusion processing process in performing multimodal feature fusion on the first audio data and the first lip movement data includes: Construct an adaptive time-frequency joint optimization beamforming algorithm:

[0012]

[0013]

[0014]

[0015] in: represents the optimal beamforming weight matrix at frequency f and time point t; argmin W represents the minimization operation on W, that is, finding the value of W that minimizes the objective function; represents the error function based on the audio signal, which is used to evaluate the enhancement effect of beamforming on the audio signal; represents the error function based on visual lip movement information, which characterizes the consistency between the beam direction and the visually predicted sound source position; Represents the constraint term of beamforming, ensuring the physical feasibility of beamforming; and Represent the adaptive weight coefficients of audio signal, visual information and constraint terms respectively; represents the estimated value of the audio signal-to-noise ratio at the current time point t; It represents the visual capture quality and quantifies the reliability of lip movement information under the current visual conditions; 、 and It is a positive adjustment parameter that controls the dynamic adjustment rate of each modal weight.

[0016] Optionally, the step of monitoring user feedback information on the first response voice in real time and updating the first user feature model and the first response strategy evaluation model based on the feedback information further includes: Constructing a multi-objective reinforcement learning framework to optimize user satisfaction:

[0017]

[0018]

[0019]

[0020]

[0021]

[0022]

[0023]

[0024] in: represents the optimization goal, i.e., maximizing the expectation of user cumulative satisfaction; represents the mathematical expectation; Indicates that the parameter is The policy function of Represents the discount factor, which ranges from (0,1) and is used to balance short-term and long-term satisfaction; Represents the state of step h in the interaction history, including speech signals, lip movement data, and environmental information; represents the response action taken by the system at step h; H represents the length of the interaction history; represents a multidimensional reward function; represents the mth reward component; M represents the number of reward components; Represents the weight of the mth reward component and satisfies ; represents the reward function based on the user's explicit feedback; represents the reward function based on the user's implicit feedback; Indicates the rate at which users accept responses; Indicates the number of times the user repeats the request; Indicates how often users correct system responses; represents the state sequence from time h to h+1; 、 and is the weight parameter of the explicit feedback reward function; represents the emotional state scoring function; Indicates the dialogue coherence indicator; represents the interaction delay penalty function; 、 and is the weight parameter of the implicit feedback reward function; and denote the probabilities of positive and negative emotions detected, respectively; 、 and is the adjustment parameter of the affective state scoring function; represents the semantic similarity function, which is used to evaluate the coherence of the dialogue; and are the slope and threshold parameters of the conversation coherence index, respectively; Indicates the system response time; and are the slope and threshold parameters of the interaction delay penalty function, respectively; represents the reward weight mapping matrix; Represents the feature vector extracted from the user profile.

[0025] Another aspect of the present invention provides an artificial intelligence-based smart speaker voice processing system for executing an artificial intelligence-based smart speaker voice processing method, comprising: a smart speaker and a server; The server is configured to: Acquire multiple channels of initial audio data and user lip movement video data, and pre-process the initial audio data and the lip movement video data to obtain first audio data and first lip movement data, respectively; Performing multimodal feature fusion on the first audio data and the first lip movement data to obtain a first multimodal fusion feature; Performing bimodal voice activity detection on the first multimodal fusion feature to obtain first valid voice data; Performing context-aware speech recognition with audio and video fusion on the first valid speech data to obtain a first text; Constructing a first user feature model, and performing semantic understanding on the first text based on the first user feature model to obtain a first understanding result; Performing intent recognition and slot filling based on the first understanding result to obtain a first user intent and first key information; generating a first response strategy based on the first user intention and the first key information in combination with environmental perception data; generating a first response voice according to the first response strategy; The user's feedback information on the first response voice is monitored in real time, and the first user feature model and the first response strategy evaluation model are updated based on the feedback information.

[0026] The technical solution of the present invention is adopted, and the artificial intelligence-based smart speaker voice processing method includes: obtaining initial audio data and user lip movement video data of multiple channels, and preprocessing the initial audio data and the lip movement video data to obtain first audio data and first lip movement data, respectively; performing multimodal feature fusion on the first audio data and the first lip movement data to obtain first multimodal fusion features; performing bimodal voice activity detection on the first multimodal fusion features to obtain first valid voice data; performing context-aware voice recognition of audio and video fusion on the first valid voice data to obtain a first text; constructing a first user feature model, and performing semantic understanding on the first text based on the first user feature model to obtain a first understanding result; performing intent recognition and slot filling based on the first understanding result to obtain a first user intent and first key information; generating a first response strategy based on the first user intent and the first key information in combination with environmental perception data; generating a first response voice according to the first response strategy; monitoring user feedback information on the first response voice in real time, and updating the first user feature model and the first response strategy evaluation model based on the feedback information. The solution of the present invention can enable the smart speaker to recognize speech more accurately in a noisy environment and enhance the security and robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flowchart of an artificial intelligence-based smart speaker voice processing method provided by one embodiment of the present invention; Figure 2This is a schematic block diagram of an artificial intelligence-based smart speaker voice processing system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0030] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0031] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0032] Refer to the following Figures 1 to 2 To describe an artificial intelligence-based smart speaker voice processing method and system provided according to some embodiments of the present invention.

[0033] like Figure 1 As shown, one embodiment of the present invention provides an intelligent speaker voice processing method based on artificial intelligence, comprising: Acquire multiple channels of initial audio data and user lip movement video data, and pre-process the initial audio data and the lip movement video data to obtain first audio data and first lip movement data, respectively; The preprocessing step includes: performing noise removal, echo removal, and signal normalization on the initial audio signal to obtain first audio data; performing face detection, mouth region extraction, and dynamic feature enhancement on the lip-syncing video data to obtain first lip-syncing data; thereby enhancing the accuracy of subsequent multimodal processing; Performing multimodal feature fusion on the first audio data and the first lip movement data to obtain a first multimodal fusion feature; It is understood that the specific implementation of this step includes: implementing beamforming technology on the first audio data, calculating the sound source direction through a deep neural network, and applying a dynamic adaptive beamforming algorithm to adjust the beam pointing to accurately track and enhance the target sound source; extracting visual speech features from the first lip movement data, covering mouth dynamic changes, mouth opening and closing degree, and mouth movement path characteristics; constructing a time-synchronized audio and video feature fusion model, and dynamically assigning weights to audio and visual features through a cross-modal attention mechanism, thereby significantly improving the quality of the speech signal in a noisy environment; Performing bimodal voice activity detection on the first multimodal fusion feature to obtain first valid voice data; The dual-modal voice activity detection includes: constructing a triple attention mechanism model that integrates acoustic features, visual features, and semantic features, wherein the triple attention mechanism model focuses on and detects the spectral features, mouth movement features, and semantic content of the signal; utilizing the triple attention mechanism model to implement high-precision voice activity detection on the first multimodal fusion features, thereby reducing missed detection rates and false detection rates and improving detection performance in low signal-to-noise ratio environments; Performing context-aware speech recognition with audio and video fusion on the first valid speech data to obtain a first text; The context-aware speech recognition with audio and video fusion includes: using a multimodal Transformer model to simultaneously process audio features and visual oral features, and integrate language knowledge at the phoneme level, word level, and sentence level; building an audio and video complementary enhancement mechanism to increase the weight of visual features when the ambient noise exceeds a preset noise value, and to increase the weight of audio features when the light does not meet the preset light intensity, thereby achieving adaptive modal weight adjustment; introducing user interaction history records as context information to dynamically adjust the language model and improve the recognition accuracy of specific user habitual expressions; using a knowledge graph-enhanced semantic understanding module to achieve the integration of domain knowledge and improve the recognition ability of professional vocabulary and terminology; Constructing a first user feature model, and performing semantic understanding on the first text based on the first user feature model to obtain a first understanding result; The first user feature model includes: extracting multi-dimensional features such as the user's historical interaction data, voice feature data, preference settings, and usage scenarios; building and continuously updating the user feature model based on the federated learning framework to ensure user privacy and security; and adjusting the attention weight of the semantic understanding module based on the personalized parameters provided by the user feature model to achieve semantic parsing that is more in line with the user's expression habits; Performing intent recognition and slot filling based on the first understanding result to obtain a first user intent and first key information; The intent recognition and slot filling process involves: building a multi-task learning framework to simultaneously optimize both intent recognition and slot filling; introducing an uncertainty modeling mechanism to identify recognition results with confidence levels below a preset confidence value and triggering corresponding interpretation dialogue strategies; and using a recursive neural network structure to capture the dependencies between slots and improve the accuracy of parsing complex commands. generating a first response strategy based on the first user intention and the first key information in combination with environmental perception data; The environmental perception data includes smart home device status data, indoor environment data, and user activity status data. The process of generating the first response strategy includes: constructing a first response strategy evaluation model that integrates knowledge graph and reinforcement learning to evaluate the value of different response strategies; dynamically adjusting response priorities based on multimodal context information to achieve intelligent decision-making in complex scenarios; and using graph neural networks to create associations between devices and generate collaborative control strategies to improve the overall coordination between smart speakers and other smart home devices. generating a first response voice according to the first response strategy; This step specifically includes: constructing a decoupled representation of acoustic and emotional features based on a variational autoencoder; utilizing a generative adversarial network to achieve fine-grained control of emotional transfer and speech style; introducing a prosodic model to adjust the rhythm and stress of speech to enhance its expressiveness and naturalness; and using adaptive sampling rate adjustment technology to dynamically adjust the speaking speed of the synthesized speech based on the importance of the response content to improve the user experience. The user's feedback information on the first response voice is monitored in real time, and the first user feature model and the first response strategy evaluation model are updated based on the feedback information.

[0034] The feedback information includes: direct interaction behaviors such as user confirmation, repeated requests, and correction instructions; user satisfaction indirectly inferred through voice sentiment analysis, interaction interruption patterns, and the correlation between subsequent instructions; based on the reinforcement learning framework, user satisfaction is used as a reward signal to continuously optimize various model parameters in the voice processing process.

[0035] By adopting the technical solution of this embodiment, ...

[0036] In some possible implementations of the present invention, the step of performing bimodal voice activity detection on the first multimodal fusion feature to obtain first valid voice data includes: Performing temporal alignment and calibration on the audio acoustic features and visual mouth-motion features in the first multimodal fusion features to eliminate the difference in audio and video acquisition delays; Extract high-level features at the semantic level, and semantically encode the fused features through a pre-trained language model to form a semantic feature vector; Construct an acoustic attention submodule to calculate the attention weight distribution of audio spectrum features in the time-frequency domain, focusing on the frequency bands and time periods related to speech activity; Construct a visual attention submodule to calculate the spatial-temporal attention weights of visual features based on lip movement intensity and mouth shape change amplitude; Construct a semantic attention submodule to calculate the semantic-level attention weights using the contextual relevance of semantic features; Design a cross-modal attention fusion mechanism that dynamically balances the contributions of the three attention sub-modules through learnable weight parameters to form a unified multimodal attention representation; The multimodal attention representation is input into a binary classification decision network, wherein the decision network adopts a multi-layer perceptron structure; The classification results are judged based on a preset confidence threshold and the speech activity probability score for each time frame is output; Applying a time series smoothing filter algorithm to smooth the voice activity probability score, eliminating short-term fluctuations and avoiding frequent voice activity state switching, thereby forming a continuous voice activity probability sequence; Identifying consecutive high-probability intervals in the voice activity probability sequence as candidate voice segments, wherein the high-probability intervals are time periods in which the probability value is continuously higher than a preset activation threshold; Performing boundary refinement processing on the candidate speech segment, and fine-tuning the start and end positions of the speech segment using the joint features of audio energy and visual motion; Short speech segments whose duration is lower than a preset minimum threshold are filtered, and a final valid speech segment sequence is output as the first valid speech data.

[0037] In this embodiment, when the audio performance degrades in a noisy environment, visual lip movement information can provide effective compensation and improve detection accuracy; through the introduction of semantic features, it can better distinguish meaningful speech from irrelevant sounds and reduce the false detection rate; in a strong noise environment, it can still maintain a high detection performance with a low missed detection rate; through the dynamic weight adjustment of the cross-modal attention mechanism, the audio feature weight is automatically increased when the quality of visual features degrades due to insufficient light; through the parallel calculation of the triple attention mechanism, the calculation delay is reduced; the subsequent recognition performance loss caused by inaccurate speech segment boundaries in traditional methods is avoided; more accurate speech segment input is provided for subsequent speech recognition, improving the overall speech recognition accuracy; the system error activation caused by false detection is reduced, and the naturalness of the interaction of the smart speaker is improved; this dual-modal voice activity detection method achieves high-precision and high-robustness voice activity detection in complex environments through multi-level and multi-dimensional feature fusion and intelligent decision-making, laying a solid foundation for improving the performance of the entire intelligent voice processing system.

[0038] In some possible implementations of the present invention, the step of performing context-aware speech recognition with audio and video fusion on the first valid voice data to obtain the first text includes: Extracting a first audio acoustic feature and a first visual mouth-motor feature within a time window corresponding to the first valid speech data from the first multimodal fusion feature; Performing a multi-scale analysis on the first audio acoustic feature to extract a first audio acoustic feature vector including a Mel-frequency cepstral coefficient, a fundamental frequency contour, and a spectral centroid; Extracting a first visual mouth movement feature vector including lip shape geometric features, motion trajectory features, and mouth shape change dynamic features from the first visual mouth movement features; Mapping the first audio acoustic feature vector and the first visual mouth movement feature vector to a unified feature space through a linear transformation layer, and organizing them into a first audio acoustic feature sequence and a first visual mouth movement feature sequence in chronological order, respectively, to ensure dimensional consistency in subsequent multimodal fusion processing; Construct an audio encoder and use a multi-head self-attention mechanism to encode the first audio acoustic feature sequence to capture the temporal dependency of acoustic features; Construct a visual encoder that uses a combination of convolutional neural networks and recurrent neural networks to encode the first visual mouth-movement feature sequence and extract the spatiotemporal representation of visual speech; Design a cross-modal fusion layer that uses the audio encoder output as the audio modality input and the visual encoder output as the visual modality input. A cross-attention mechanism is used to achieve bidirectional information interaction between audio acoustic features and visual mouth-motion features, generating a fused multimodal representation. Constructing a decoder network to receive the multimodal representation and fuse phoneme-level, word-level, and sentence-level language knowledge to output character-level or word-level recognition probability distribution; Extract user interaction history, including recent conversation content, common vocabulary, and expression habits, to build a personalized language model; Calculating a context prior probability based on the personalized language model and performing Bayesian correction on the recognition probability distribution; Combined with the contextual information of the current dialogue turn, the conditional random field model is used to optimize the sequence annotation of the recognition results to improve semantic coherence; Based on the sequence annotation optimization results, combined with the knowledge graph to enhance semantic understanding, a first recognition sequence is obtained, including: receiving the sequence annotation optimization results, identifying professional vocabulary and domain terms through a named entity recognition module; querying a pre-built knowledge graph to obtain semantic association information and context constraints of the professional vocabulary; using a graph attention network to fuse knowledge graph information to perform semantic disambiguation and error correction on the professional vocabulary in the recognition results; and performing global optimization on the entire recognition sequence based on semantic consistency constraints to ensure the semantic rationality of the output text. Convert the first recognition sequence into continuous text, perform punctuation prediction and case normalization processing, and obtain the initial text; Calculating the confidence score of the initial text and performing a comprehensive evaluation based on three dimensions: acoustic model confidence, language model confidence, and multimodal consistency; Output the initial text and its confidence score as the first text.

[0039] The solution of this embodiment can improve recognition accuracy, improve environmental robustness, optimize personalized experience, and enhance computing efficiency and real-time performance. This audio and video fusion context-aware speech recognition method achieves high-precision, personalized speech recognition in complex environments through multi-level feature fusion, intelligent environmental adaptation, and rich semantic enhancement, providing strong technical support for improving the user experience of intelligent voice interaction systems.

[0040] In some possible implementations of the present invention, the step of constructing a first user feature model and performing semantic understanding on the first text based on the first user feature model to obtain a first understanding result includes: Extracting the user's historical interaction records, voice habit parameters, preference setting data, and usage scenario information, and constructing a multidimensional user feature set, including: performing text preprocessing on the historical interaction records, including word segmentation, part-of-speech tagging, and named entity recognition, to extract language style features including the user's commonly used vocabulary and expression patterns; performing statistical analysis on the voice habit parameters to extract personalized acoustic features including the user's pronunciation characteristics, speaking speed preferences, pause patterns, and intonation habits; integrating the language style features, the personalized acoustic features, the preference setting data, and the usage scenario information to construct a multidimensional user feature set including a language feature vector, an acoustic feature vector, a function preference vector, a time usage pattern, and an environmental context feature; Based on the multidimensional user feature set and the federated learning framework, an adaptive user feature model is constructed, including: constructing a user personalized feature vector on a local device based on the multidimensional user feature set to ensure that the original user data does not leave the local environment; designing a federated learning aggregation algorithm to securely aggregate the local user feature vector with the global feature model to learn the common features of the user group while protecting privacy; adding noise perturbations to the personalized feature vector through a differential privacy mechanism to prevent leakage of user privacy information; and constructing an adaptive user feature model, which includes personalized weight parameters and global shared parameters and can dynamically adjust the preference tendency of semantic understanding according to user characteristics. Inputting the first text into the adaptive user feature model to generate personalized adjustment parameters for the current user; In some possible implementations of the present invention, a semantic understanding adjustment operation is also included, including: calculating the weight coefficients of each attention layer in the semantic understanding module based on the personalized adjustment parameters, focusing on the semantic dimensions and expression habits of user preferences; adjusting the priority of word sense disambiguation according to the user's language style characteristics, giving priority to vocabulary meanings that are consistent with the user's expression habits; using the user's function preference vector to adjust the probability distribution of intent classification, and improving the recognition confidence of the user's commonly used function-related intentions.

[0041] In combination with the personalized adjustment parameters, performing deep semantic analysis on the first text to extract a semantic representation vector of the text; Based on the user's time usage pattern and environmental context characteristics, the context weight of the semantic representation vector is adjusted to enhance the understanding of the user's intention in a specific scenario; The user's common vocabulary features are used to provide personalized interpretations of the unregistered words and professional terminology components in the adjusted semantic representation vector, and the possible meanings of new words are inferred based on the user's historical usage records. Apply user expression pattern features to perform consistency check on the semantic representation vector after personalized interpretation, generate the final personalized semantic understanding result, and ensure that the interpretation result conforms to the user's language habits and expression logic; Performing a weighted fusion of the personalized semantic understanding result and the standard semantic understanding result based on the original initial text to calculate a comprehensive confidence score of the personalized semantic understanding; Perform semantic integrity checks on the fused semantic understanding results to identify and repair semantic inconsistencies caused by personalized adjustments. Based on user feedback history, the semantic understanding results after the completeness check are finally optimized to adjust the interpretation tendency of uncertain semantic components; Output the final optimized semantic understanding result and its confidence evaluation as the first understanding result.

[0042] The personalized semantic understanding method based on the adaptive user feature model in this embodiment achieves both safe and accurate personalized speech understanding through privacy-preserving personalized modeling, dynamic semantic adjustment, and continuous model optimization, laying a solid technical foundation for improving user experience and industrial application of intelligent voice interaction systems.

[0043] In some possible implementations of the present invention, an incremental update step of the user feature model is further included, specifically including: Record the match between semantic understanding results and subsequent user feedback to evaluate the effectiveness of personalized adjustments; Based on the matching situation, the weight parameters in the user feature vector are updated to strengthen effective personalized features and weaken ineffective features; The adaptive user feature model is incrementally updated through online learning algorithms to maintain the model's adaptability to changes in user habits.

[0044] In some possible implementations of the present invention, the step of performing intent recognition and slot filling based on the first understanding result to obtain the first user intent and the first key information includes: Preprocessing and feature extraction are performed on the first understanding result to output a multi-dimensional semantic feature vector, including: performing structured analysis on the first understanding result to extract syntactic features, semantic role annotations, and entity boundary information; identifying uncertainty areas in the first understanding result based on confidence assessment, and marking text segments with confidence levels below a preset threshold as key focus areas; constructing a context feature vector, fusing the first understanding result with semantic association information from historical user conversations to form a contextual representation of multiple conversations; extracting domain-related semantic features, performing domain classification and professional terminology annotation on the first understanding result based on a pre-built domain knowledge base, and outputting a multi-dimensional semantic feature vector; Design a shared encoder network to map the multi-dimensional semantic feature vector into a unified shared latent layer representation, providing a common feature basis for intent recognition and slot filling tasks; Constructing an intent recognition branch network, using a multi-layer perceptron structure to classify the shared hidden layer representation, and outputting a probability distribution vector of the user's intent; Constructing a slot filling branch network, using a sequence labeling structure of a conditional random field combined with a bidirectional long short-term memory network to predict slot labels on the shared hidden layer representation to obtain a slot label sequence; Design a joint loss function that combines the cross-entropy loss of intent recognition and the sequence labeling loss of slot filling, improving the collaborative performance of the two tasks through multi-task joint optimization. Uncertainty modeling and confidence assessment are performed to obtain comprehensive confidence assessment results, including: quantifying the uncertainty of the intent probability distribution based on the uncertainty areas identified from the first understanding results, and calculating the entropy and variance of intent recognition as uncertainty indicators; conducting confidence assessment on the slot label sequence, calculating the predicted probability of each slot label and the consistency score between labels; constructing a comprehensive confidence model, integrating the intent recognition uncertainty indicator and the slot filling confidence score, and generating a confidence assessment of the overall task execution; based on the evaluation results of the comprehensive confidence model, identifying low-confidence recognition results that need to trigger clarification dialogues and generating targeted clarification strategies; Based on the slot label sequence, a recursive neural network is used to generate a slot filling result, including: constructing a slot dependency graph based on the slot label sequence to identify the semantic associations and constraint relationships between different slots; designing a recursive neural network structure to perform graph convolution processing on the slot dependency graph, and capturing high-order dependency features between slots through multi-layer recursive calculations; using an attention mechanism to dynamically adjust the dependency weights between slots to generate a recursive processing result that integrates high-order dependency features and attention weights; and modifying and optimizing the slot label sequence based on the recursive processing result to generate a slot filling result that takes slot dependencies into account. Based on the slot filling results, identify the complex command structure containing multiple sub-intents and perform hierarchical decomposition of the intent; For each sub-intent in the hierarchical intent structure, the intent recognition process is re-executed to obtain the hierarchical intent recognition result; Verify the logical consistency of each slot in complex commands based on slot dependencies, and identify and fix conflicting slot values; Build a command integrity check mechanism to ensure that the necessary slots of complex commands are fully filled, mark missing slots and generate a completion strategy to obtain a complete slot filling result; Integrate the hierarchical intent recognition results and the complete slot filling results to generate a structured intent representation containing the main intent, sub-intent, and corresponding slot values; Extract key information based on the structured intent representation, mark the main intent and sub-intent as a primary intent category and a secondary intent category, respectively, and extract entity values ​​and parameter values ​​in the slot filling result as key entity values ​​and operation parameters, respectively; Confidence annotation is performed on the structured intent representation and the key information, and based on the comprehensive confidence evaluation result, a corresponding credibility score is assigned to each intent category and key information element; Output the first user intention and first key information marked with confidence; the first user intention includes the main intention category and its confidence, the secondary intention category and its confidence; the first key information includes the key entity value and its confidence, the target device information, the operation parameters and their confidence.

[0045] In an embodiment of the present invention, the intent recognition and slot filling method based on multi-task learning achieves high-precision and high-robustness natural language understanding through collaborative optimization between tasks, intelligent uncertainty processing and hierarchical parsing of complex commands, providing core technical support for the practical application and industrialization of intelligent voice interaction systems.

[0046] In some possible implementations of the present invention, the step of generating a first response strategy based on the first user intention and the first key information in combination with environmental perception data includes: Collecting environmental perception data and fusing it to obtain comprehensive environmental perception data, including: real-time collection of smart home device status data, including the operating status, connection status, function availability, and current parameter settings of each device, and constructing a device state vector representation; obtaining indoor environmental parameter data, including temperature, humidity, light intensity, noise level, and air quality indicators, and generating an environmental parameter feature vector; detecting user activity status data, identifying the user's location, activity category, physiological state, and attention concentration through a sensor network, and forming a user state feature representation; and multi-dimensionally fusing the device state vector, environmental parameter feature vector, and user state feature representation to generate comprehensive environmental perception data; Based on the first user intention, the first key information, and the comprehensive environmental perception data, an association analysis is performed between the user intention and the environmental data to obtain an intention execution plan, including: extracting the user intention (primary intention category, secondary intention category), user intention confidence, operation parameters, and target device information from the first user intention and the first key information; matching device availability based on the target device information and the comprehensive environmental perception data to identify the set of devices that can execute the user intention in the current environment; constructing an intention-environment compatibility evaluation model to calculate the execution feasibility score and expected effect evaluation of the extracted user intention under the current environmental conditions; and generating an intention execution plan based on the compatibility evaluation results, including device selection priority, execution timing arrangement, and environmental constraints. Optimal decision-making strategies are derived through decision modeling enhanced by knowledge graphs, including: constructing a multi-level knowledge graph that integrates device relationships, scenario knowledge, and user preferences, encompassing device entities, functional attributes, associations, and scenario rules; performing a path search within the multi-level knowledge graph based on the intended execution plan to identify multiple possible execution paths that achieve user intent; designing a reinforcement learning decision model that uses user satisfaction as a reward signal to evaluate the long-term value and risk assessment of each identified possible execution path; and optimizing the strategy of the reinforcement learning decision model using a Monte Carlo tree search algorithm to generate an optimal decision strategy that considers long-term effects. In combination with the comprehensive environmental perception data, the optimal decision-making strategy is adjusted for response priority to obtain a response priority scheme, including: analyzing the urgency of the current scenario based on the comprehensive environmental perception data, and identifying safety-related, comfort-related, and efficiency-related response requirements that need to be prioritized; combining the user intent confidence extracted from the first user intent and the optimal decision-making strategy to construct a multi-objective optimization model to balance response speed, execution effect, and resource consumption; introducing a time decay mechanism to dynamically adjust the response priority weight according to the user waiting time and the urgency of the intent to generate a time-sensitive priority sorting result; based on the priority sorting result, the optimal decision-making strategy is reordered and time-series adjusted to generate a response priority scheme adapted to the current context; Based on the graph neural network and the response priority scheme, a collaborative control strategy is generated, including: constructing a smart home device collaborative relationship graph, using each device as a graph node and the functional dependencies, physical connections and collaborative effects between devices as graph edges to form a device collaborative network topology; based on the response priority scheme, using a graph attention network to calculate the importance weight and collaborative influence factor of each device node in the current task execution; propagating status information and control instructions between devices through a graph convolutional neural network to achieve collaborative decision-making and conflict resolution between devices, and generate a conflict-free device control sequence; based on the device control sequence, combined with device response time and execution effect prediction, generating a collaborative control strategy that includes execution timing, parameter settings and exception handling mechanism; Integrating the optimal decision-making strategy, the response priority scheme, and the collaborative control strategy to construct a multi-level comprehensive response strategy framework; Verify the feasibility of the comprehensive response strategy framework, check the logical consistency, resource availability and time feasibility of the strategy, and identify potential execution risks; Based on the verification results, a complete response strategy set including the main execution strategy, alternative strategies and exception handling strategies is generated to ensure the robustness of strategy execution; Output the final first response strategy, including device control instruction sequence, execution timing arrangement, content importance weight, urgency identification, expected effect evaluation and exception handling mechanism.

[0047] This embodiment scheme achieves a leap from single-dimensional decision-making to multi-dimensional comprehensive decision-making by integrating user intention, environmental perception and knowledge graph, thereby improving decision-making accuracy; the introduction of reinforcement learning model enables the system to consider the long-term impact of decisions, avoid the negative effects of short-sighted decisions, and improve user satisfaction; real-time environmental perception and intention matching mechanism enables the system to automatically adjust execution strategies according to environmental changes, improving environmental adaptability; priority adjustment based on multimodal context realizes true scene-aware service, and the relevance and timeliness of response are significantly improved; the device collaboration mechanism of graph neural network modeling effectively solves the conflict problem in concurrent control of multiple devices and improves device collaboration efficiency; through intelligent collaboration between devices, it realizes the transition from local optimization to global optimization and improves overall system performance; strategy generation combining user characteristics and environmental context realizes highly personalized intelligent services and improves user experience satisfaction.

[0048] In some possible implementations of the present invention, the step of generating a first response voice according to the first response strategy includes: Based on the emotional tags and semantic content in the first response strategy, a variational autoencoder is used to encode the response text into acoustic feature vectors and emotional feature vectors, thereby achieving decoupled representation of acoustic information and emotional information; According to the personalized parameters output by the first user feature model and the current interaction scenario, the emotion feature vector is modulated by a generative adversarial network to generate a target emotion feature that meets the user's preference; Inputting the target emotional feature and the acoustic feature vector into a preset prosody model, dynamically calculating the fundamental frequency, duration, and energy parameters of the speech, and adjusting the rhythm, stress, and intonation of the speech; Based on the content importance weight and urgency indicator in the first response strategy, an adaptive speech rate adjustment algorithm is used to calculate the target duration ratio of each voice segment, and the duration parameters output by the prosody model are dynamically adjusted according to the target duration ratio to achieve differentiated speech rate control, where important information is expressed slowly and secondary information is conveyed quickly. The adjusted acoustic features, emotional features and prosodic parameters are input into the neural vocoder to synthesize and output a natural speech signal with personalized emotional expression.

[0049] The solution of this embodiment, through the decoupled representation of the variational autoencoder, can accurately control the emotional components in the speech, avoid mutual interference between emotional information and acoustic information, and improve the accuracy and consistency of emotional expression; combined with the user feature model and the adversarial generative network, it can generate personalized voice styles that meet the preferences of different users, significantly improving the satisfaction of the user interaction experience; through the fine adjustment of the rhythm model and adaptive speech rate control, the synthesized speech is more natural and fluent in rhythm, stress and intonation, close to the effect of real-life speech expression; the speech rate is dynamically adjusted according to the importance of the response content, important information is slowed down for easy understanding, and secondary information is appropriately accelerated to improve the efficiency of information transmission; the entire speech synthesis process can be dynamically adjusted according to real-time interaction scenarios and user feedback, and has good environmental adaptability and user adaptability. The solution of this embodiment achieves end-to-end optimization from text to emotional speech through multi-level feature decoupling and fusion, providing smart speakers with more intelligent and humanized voice interaction capabilities.

[0050] In some possible implementations of the present invention, the multimodal feature fusion processing process in performing multimodal feature fusion on the first audio data and the first lip movement data includes: Construct an adaptive time-frequency joint optimization beamforming algorithm:

[0051]

[0052]

[0053]

[0054] in: represents the optimal beamforming weight matrix at frequency f and time point t; argmin W represents the minimization operation on W, that is, finding the value of W that minimizes the objective function; represents the error function based on the audio signal, which is used to evaluate the enhancement effect of beamforming on the audio signal; represents the error function based on visual lip movement information, which characterizes the consistency between the beam direction and the visually predicted sound source position; Represents the constraint term of beamforming, ensuring the physical feasibility of beamforming; and Represent the adaptive weight coefficients of audio signal, visual information and constraint terms respectively; represents the estimated value of the audio signal-to-noise ratio at the current time point t; It represents the visual capture quality and quantifies the reliability of lip movement information under the current visual conditions; 、 and It is a positive adjustment parameter that controls the dynamic adjustment rate of each modal weight.

[0055] In this embodiment, by adaptively adjusting the weights of audio and visual modalities, optimal beamforming is achieved in different noise environments and lighting conditions. Compared with the traditional fixed weight method, the speech enhancement performance in complex environments is improved, and smooth transitions can be achieved when the quality of modal information changes dramatically, avoiding sudden changes in system performance.

[0056] In some possible implementations of the present invention, the step of monitoring the user's feedback information on the first response voice in real time and updating the first user feature model and the first response strategy evaluation model based on the feedback information further includes: Constructing a multi-objective reinforcement learning framework to optimize user satisfaction:

[0057]

[0058]

[0059]

[0060]

[0061]

[0062]

[0063]

[0064] in: represents the optimization goal, i.e., maximizing the expectation of user cumulative satisfaction; represents the mathematical expectation; Indicates that the parameter is The policy function of Represents the discount factor, which ranges from (0,1) and is used to balance short-term and long-term satisfaction; Represents the state of step h in the interaction history, including speech signals, lip movement data, and environmental information; represents the response action taken by the system at step h; H represents the length of the interaction history; represents a multidimensional reward function; represents the mth reward component; M represents the number of reward components; Represents the weight of the mth reward component and satisfies ; represents the reward function based on the user's explicit feedback; represents the reward function based on the user's implicit feedback; Indicates the rate at which users accept responses; Indicates the number of times the user repeats the request; Indicates how often users correct system responses; represents the state sequence from time h to h+1; 、 and is the weight parameter of the explicit feedback reward function; represents the emotional state scoring function; Indicates the dialogue coherence indicator; represents the interaction delay penalty function; 、 and is the weight parameter of the implicit feedback reward function; and denote the probabilities of positive and negative emotions detected, respectively; 、 and is the adjustment parameter of the affective state scoring function; represents the semantic similarity function, which is used to evaluate the coherence of the dialogue; and are the slope and threshold parameters of the conversation coherence index, respectively; Indicates the system response time; and are the slope and threshold parameters of the interaction delay penalty function, respectively; represents the reward weight mapping matrix; Represents the feature vector extracted from the user profile.

[0065] The solution of this embodiment achieves accurate modeling and optimization of user satisfaction by integrating explicit and implicit user feedback to establish a multi-objective reinforcement learning framework. This method can adaptively adjust system behavior based on multi-dimensional factors such as the user's emotional state, interaction coherence, and response timeliness. Compared with traditional single-objective optimization methods, this method improves user satisfaction and significantly enhances the personalization of system responses, especially in long-term conversations and complex task execution, demonstrating continuous adaptation and improvement capabilities.

[0066] See Figure 2 , another embodiment of the present invention provides an artificial intelligence-based smart speaker voice processing system for executing an artificial intelligence-based smart speaker voice processing method, comprising: a smart speaker and a server; The server is configured to: Acquire multiple channels of initial audio data and user lip movement video data, and pre-process the initial audio data and the lip movement video data to obtain first audio data and first lip movement data, respectively; Performing multimodal feature fusion on the first audio data and the first lip movement data to obtain a first multimodal fusion feature; Performing bimodal voice activity detection on the first multimodal fusion feature to obtain first valid voice data; Performing context-aware speech recognition with audio and video fusion on the first valid speech data to obtain a first text; Constructing a first user feature model, and performing semantic understanding on the first text based on the first user feature model to obtain a first understanding result; Performing intent recognition and slot filling based on the first understanding result to obtain a first user intent and first key information; generating a first response strategy based on the first user intention and the first key information in combination with environmental perception data; generating a first response voice according to the first response strategy; The user's feedback information on the first response voice is monitored in real time, and the first user feature model and the first response strategy evaluation model are updated based on the feedback information.

[0067] It should be known that Figure 2 The block diagram of the AI-based smart speaker voice processing system is for illustration only, and the number of modules shown does not limit the scope of protection of the present invention. The AI-based smart speaker voice processing system provided in this embodiment can be used to implement the various embodiments of the corresponding AI-based smart speaker voice processing method. For the specific implementation process, please refer to the description of the various method embodiments and will not be repeated here.

[0068] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0069] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0070] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0071] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0072] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0073] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned memory includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0074] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0075] The above is a detailed introduction to the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0076] Although the present invention is disclosed above, it is not limited thereto. Any person skilled in the art may readily conceive of variations or substitutions, and may make various modifications and alterations without departing from the spirit and scope of the present invention. Combinations of the above-described functions and implementation steps, including software and hardware implementations, are all within the scope of protection of the present invention.

Claims

1. A method for processing voice of an intelligent speaker based on artificial intelligence, characterized in that: include: Acquire multiple channels of initial audio data and user lip movement video data, and pre-process the initial audio data and the lip movement video data to obtain first audio data and first lip movement data, respectively; Performing multimodal feature fusion on the first audio data and the first lip movement data to obtain a first multimodal fusion feature; Performing bimodal voice activity detection on the first multimodal fusion feature to obtain first valid voice data; Performing context-aware speech recognition with audio and video fusion on the first valid speech data to obtain a first text; Constructing a first user feature model, and performing semantic understanding on the first text based on the first user feature model to obtain a first understanding result; Performing intent recognition and slot filling based on the first understanding result to obtain a first user intent and first key information; generating a first response strategy based on the first user intention and the first key information in combination with environmental perception data; generating a first response voice according to the first response strategy; The user's feedback information on the first response voice is monitored in real time, and the first user feature model and the first response strategy evaluation model are updated based on the feedback information.

2. The method for processing speech of an intelligent speaker based on artificial intelligence according to claim 1, characterized in that: The step of performing bimodal voice activity detection on the first multimodal fusion feature to obtain first valid voice data includes: Performing temporal alignment and calibration on the audio acoustic features and visual mouth-motion features in the first multimodal fusion features to eliminate the difference in audio and video acquisition delays; Extract high-level features at the semantic level, and semantically encode the fused features through a pre-trained language model to form a semantic feature vector; Construct an acoustic attention submodule to calculate the attention weight distribution of audio spectrum features in the time-frequency domain, focusing on the frequency bands and time periods related to speech activity; Construct a visual attention submodule to calculate the spatial-temporal attention weights of visual features based on lip movement intensity and mouth shape change amplitude; Construct a semantic attention submodule to calculate the semantic-level attention weights using the contextual relevance of semantic features; Design a cross-modal attention fusion mechanism that dynamically balances the contributions of the three attention sub-modules through learnable weight parameters to form a unified multimodal attention representation; The multimodal attention representation is input into a binary classification decision network, wherein the decision network adopts a multi-layer perceptron structure; The classification results are judged based on a preset confidence threshold and the speech activity probability score for each time frame is output; Applying a time series smoothing filter algorithm to smooth the voice activity probability score, eliminating short-term fluctuations and avoiding frequent voice activity state switching, thereby forming a continuous voice activity probability sequence; Identifying consecutive high-probability intervals in the voice activity probability sequence as candidate voice segments, wherein the high-probability intervals are time periods in which the probability value is continuously higher than a preset activation threshold; Performing boundary refinement processing on the candidate speech segment, and fine-tuning the start and end positions of the speech segment using the joint features of audio energy and visual motion; Short speech segments whose duration is lower than a preset minimum threshold are filtered, and a final valid speech segment sequence is output as the first valid speech data.

3. The method for processing speech of an intelligent speaker based on artificial intelligence according to claim 2, characterized in that: The step of performing context-aware speech recognition with audio and video fusion on the first valid speech data to obtain a first text includes: Extracting a first audio acoustic feature and a first visual mouth-motor feature within a time window corresponding to the first valid speech data from the first multimodal fusion feature; Performing a multi-scale analysis on the first audio acoustic feature to extract a first audio acoustic feature vector including a Mel-frequency cepstral coefficient, a fundamental frequency contour, and a spectral centroid; Extracting a first visual mouth movement feature vector including lip shape geometric features, motion trajectory features, and mouth shape change dynamic features from the first visual mouth movement features; Mapping the first audio acoustic feature vector and the first visual mouth movement feature vector to a unified feature space through a linear transformation layer, and organizing them into a first audio acoustic feature sequence and a first visual mouth movement feature sequence in chronological order, respectively, to ensure dimensional consistency in subsequent multimodal fusion processing; Construct an audio encoder and use a multi-head self-attention mechanism to encode the first audio acoustic feature sequence to capture the temporal dependency of acoustic features; Construct a visual encoder that uses a combination of convolutional neural networks and recurrent neural networks to encode the first visual mouth-movement feature sequence and extract the spatiotemporal representation of visual speech; Design a cross-modal fusion layer that uses the audio encoder output as the audio modality input and the visual encoder output as the visual modality input. A cross-attention mechanism is used to achieve bidirectional information interaction between audio acoustic features and visual mouth-motion features, generating a fused multimodal representation. Constructing a decoder network to receive the multimodal representation and fuse phoneme-level, word-level, and sentence-level language knowledge to output character-level or word-level recognition probability distribution; Extract user interaction history, including recent conversation content, common vocabulary, and expression habits, to build a personalized language model; Calculating a context prior probability based on the personalized language model and performing Bayesian correction on the recognition probability distribution; Combined with the contextual information of the current dialogue turn, the conditional random field model is used to optimize the sequence annotation of the recognition results to improve semantic coherence; Based on the sequence annotation optimization results, the first recognition sequence is obtained by combining the knowledge graph to enhance semantic understanding; Convert the first recognition sequence into continuous text, perform punctuation prediction and case normalization processing, and obtain the initial text; Calculating the confidence score of the initial text and performing a comprehensive evaluation based on three dimensions: acoustic model confidence, language model confidence, and multimodal consistency; Output the initial text and its confidence score as the first text.

4. The method for processing speech of an intelligent speaker based on artificial intelligence according to claim 3, characterized in that: The step of constructing a first user feature model and performing semantic understanding on the first text based on the first user feature model to obtain a first understanding result includes: Extract users' historical interaction records, voice habit parameters, preference data, and usage scenario information, and construct a multi-dimensional user feature set; Building an adaptive user feature model based on the multi-dimensional user feature set and the federated learning framework; Inputting the first text into the adaptive user feature model to generate personalized adjustment parameters for the current user; In combination with the personalized adjustment parameters, performing deep semantic analysis on the first text to extract a semantic representation vector of the text; Based on the user's time usage pattern and environmental context characteristics, the context weight of the semantic representation vector is adjusted to enhance the understanding of the user's intention in a specific scenario; The user's common vocabulary features are used to provide personalized interpretations of the unregistered words and professional terminology components in the adjusted semantic representation vector, and the possible meanings of new words are inferred based on the user's historical usage records. Apply user expression pattern features to perform consistency check on the semantic representation vector after personalized interpretation, generate the final personalized semantic understanding result, and ensure that the interpretation result conforms to the user's language habits and expression logic; Performing a weighted fusion of the personalized semantic understanding result and the standard semantic understanding result based on the original initial text to calculate a comprehensive confidence score of the personalized semantic understanding; Perform semantic integrity checks on the fused semantic understanding results to identify and repair semantic inconsistencies caused by personalized adjustments. Based on user feedback history, the semantic understanding results after the completeness check are finally optimized to adjust the interpretation tendency of uncertain semantic components; Output the final optimized semantic understanding result and its confidence evaluation as the first understanding result.

5. The method for processing speech of an intelligent speaker based on artificial intelligence according to claim 4, characterized in that: The step of performing intent recognition and slot filling based on the first understanding result to obtain the first user intent and first key information includes: Preprocess and extract features from the first understanding result, and output a multi-dimensional semantic feature vector; Design a shared encoder network to map the multi-dimensional semantic feature vector into a unified shared latent layer representation, providing a common feature basis for intent recognition and slot filling tasks; Constructing an intent recognition branch network, using a multi-layer perceptron structure to classify the shared hidden layer representation, and outputting a probability distribution vector of the user's intent; Constructing a slot filling branch network, using a sequence labeling structure of a conditional random field combined with a bidirectional long short-term memory network to predict slot labels on the shared hidden layer representation to obtain a slot label sequence; Design a joint loss function that combines the cross-entropy loss of intent recognition and the sequence labeling loss of slot filling, improving the collaborative performance of the two tasks through multi-task joint optimization. Conduct uncertainty modeling and confidence assessment to obtain comprehensive confidence assessment results; Based on the slot label sequence, a recurrent neural network is used to generate slot filling results; Based on the slot filling results, identify the complex command structure containing multiple sub-intents and perform hierarchical decomposition of the intent; For each sub-intent in the hierarchical intent structure, the intent recognition process is re-executed to obtain the hierarchical intent recognition result; Verify the logical consistency of each slot in complex commands based on slot dependencies, and identify and fix conflicting slot values; Build a command integrity check mechanism to ensure that the necessary slots of complex commands are fully filled, mark missing slots and generate a completion strategy to obtain a complete slot filling result; Integrate the hierarchical intent recognition results and the complete slot filling results to generate a structured intent representation containing the main intent, sub-intent, and corresponding slot values; Extract key information based on the structured intent representation, mark the main intent and sub-intent as a primary intent category and a secondary intent category, respectively, and extract entity values ​​and parameter values ​​in the slot filling results as key entity values ​​and operation parameters, respectively; Confidence annotation is performed on the structured intent representation and the key information, and based on the comprehensive confidence evaluation result, a corresponding credibility score is assigned to each intent category and key information element; Output the first user intention and the first key information that have been labeled with confidence.

6. The method for processing speech of an intelligent speaker based on artificial intelligence according to claim 5, characterized in that: The step of generating a first response strategy based on the first user intention and the first key information in combination with environmental perception data includes: Collect environmental perception data and perform fusion processing to obtain comprehensive environmental perception data; Perform correlation analysis between the user intention and the environmental data based on the first user intention, the first key information, and the comprehensive environmental perception data to obtain an intention execution plan; Decision modeling based on knowledge graph enhancement to obtain the optimal decision strategy; In combination with the comprehensive environmental perception data, adjusting the response priority of the optimal decision strategy to obtain a response priority scheme; generating a collaborative control strategy based on the graph neural network and the response priority scheme; Integrating the optimal decision-making strategy, the response priority scheme, and the collaborative control strategy to construct a multi-level comprehensive response strategy framework; Verify the feasibility of the comprehensive response strategy framework, check the logical consistency, resource availability and time feasibility of the strategy, and identify potential execution risks; Based on the verification results, a complete response strategy set including the main execution strategy, alternative strategies and exception handling strategies is generated to ensure the robustness of strategy execution; Output the final first response strategy, including device control instruction sequence, execution timing arrangement, content importance weight, urgency identification, expected effect evaluation and exception handling mechanism.

7. The method for processing speech of an intelligent speaker based on artificial intelligence according to claim 6, characterized in that: The step of generating a first response voice according to the first response strategy includes: Based on the emotional tags and semantic content in the first response strategy, a variational autoencoder is used to encode the response text into acoustic feature vectors and emotional feature vectors, thereby achieving decoupled representation of acoustic information and emotional information; According to the personalized parameters output by the first user feature model and the current interaction scenario, the emotion feature vector is modulated by a generative adversarial network to generate a target emotion feature that meets the user's preference; Inputting the target emotional feature and the acoustic feature vector into a preset prosody model, dynamically calculating the fundamental frequency, duration, and energy parameters of the speech, and adjusting the rhythm, stress, and intonation of the speech; Based on the content importance weight and urgency indicator in the first response strategy, an adaptive speech rate adjustment algorithm is used to calculate the target duration ratio of each voice segment, and the duration parameters output by the prosody model are dynamically adjusted according to the target duration ratio to achieve differentiated speech rate control, where important information is expressed slowly and secondary information is conveyed quickly. The adjusted acoustic features, emotional features and prosodic parameters are input into the neural vocoder to synthesize and output a natural speech signal with personalized emotional expression.

8. The method for processing speech of an intelligent speaker based on artificial intelligence according to claim 7, characterized in that: The multimodal feature fusion processing process in performing multimodal feature fusion on the first audio data and the first lip movement data includes: Construct an adaptive time-frequency joint optimization beamforming algorithm: in: represents the optimal beamforming weight matrix at frequency f and time point t; argmin W represents the minimization operation on W, that is, finding the value of W that minimizes the objective function; represents the error function based on the audio signal, which is used to evaluate the enhancement effect of beamforming on the audio signal; represents the error function based on visual lip movement information, which characterizes the consistency between the beam direction and the visually predicted sound source position; Represents the constraint term of beamforming, ensuring the physical feasibility of beamforming; and Represent the adaptive weight coefficients of audio signal, visual information and constraint terms respectively; represents the estimated value of the audio signal-to-noise ratio at the current time point t; It represents the visual capture quality and quantifies the reliability of lip movement information under the current visual conditions; 、 and It is a positive adjustment parameter that controls the dynamic adjustment rate of each modal weight.

9. The method for processing speech of an intelligent speaker based on artificial intelligence according to claim 8, characterized in that: The step of monitoring the user's feedback information on the first response voice in real time and updating the first user feature model and the first response strategy evaluation model based on the feedback information further includes: Constructing a multi-objective reinforcement learning framework to optimize user satisfaction: in: represents the optimization goal, i.e., maximizing the expectation of user cumulative satisfaction; represents the mathematical expectation; Indicates that the parameter is The policy function of Represents the discount factor, which ranges from (0,1) and is used to balance short-term and long-term satisfaction; Represents the state of step h in the interaction history, including speech signals, lip movement data, and environmental information; represents the response action taken by the system at step h; H represents the length of the interaction history; represents a multidimensional reward function; represents the mth reward component; M represents the number of reward components; Represents the weight of the mth reward component and satisfies ; represents the reward function based on the user's explicit feedback; represents the reward function based on the user's implicit feedback; Indicates the rate at which users accept responses; Indicates the number of times the user repeats the request; Indicates how often users correct system responses; represents the state sequence from time h to h+1; 、 and is the weight parameter of the explicit feedback reward function; represents the emotional state scoring function; represents the dialogue coherence indicator; represents the interaction delay penalty function; 、 and is the weight parameter of the implicit feedback reward function; and denote the probabilities of positive and negative emotions detected, respectively; 、 and is the adjustment parameter of the affective state scoring function; represents the semantic similarity function, which is used to evaluate the coherence of the dialogue; and are the slope and threshold parameters of the conversation coherence index, respectively; Indicates the system response time; and are the slope and threshold parameters of the interaction delay penalty function, respectively; represents the reward weight mapping matrix; Represents the feature vector extracted from the user profile.

10. An artificial intelligence-based smart speaker voice processing system, used to execute the artificial intelligence-based smart speaker voice processing method according to any one of claims 1 to 9, characterized in that: include: smart speakers and servers; The server is configured to: Acquire multiple channels of initial audio data and user lip movement video data, and pre-process the initial audio data and the lip movement video data to obtain first audio data and first lip movement data, respectively; Performing multimodal feature fusion on the first audio data and the first lip movement data to obtain a first multimodal fusion feature; Performing bimodal voice activity detection on the first multimodal fusion feature to obtain first valid voice data; Performing context-aware speech recognition with audio and video fusion on the first valid speech data to obtain a first text; Constructing a first user feature model, and performing semantic understanding on the first text based on the first user feature model to obtain a first understanding result; Performing intent recognition and slot filling based on the first understanding result to obtain a first user intent and first key information; generating a first response strategy based on the first user intention and the first key information in combination with environmental perception data; generating a first response voice according to the first response strategy; The user's feedback information on the first response voice is monitored in real time, and the first user feature model and the first response strategy evaluation model are updated based on the feedback information.

Images