Voice recognition method and system of intelligent voice robot

Abstract

The invention relates to the field of voice recognition, in particular to a voice recognition method and system of an intelligent voice robot. The method comprises the following steps: collecting user real-time dialogue audio and a user dialogue monitoring image; performing noise reduction separation processing on the real-time dialogue audio of the user, and performing audio distortion optimization to obtain a distortion optimized dialogue audio; performing micro-expression recognition on the user dialogue monitoring image, and performing emotional state feature mining so as to generate real-time emotional state features of the user; performing user semantic pause recognition on the distortion optimization dialogue audio, and performing key vocabulary feature analysis to obtain a key field of each frequency domain unit; and carrying out multi-scale convolutional learning on the key field of each frequency domain unit, and then carrying out audio context association mining so as to generate a user audio context association map. According to the invention, efficient and accurate speech recognition is realized.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition, and in particular to a speech recognition method and system for an intelligent speech robot. Background Art

[0002] With the continuous advancement of artificial intelligence technology, intelligent voice robots are becoming one of the main ways of human-computer interaction. This kind of intelligent robot based on voice interaction can not only realize voice recognition and voice synthesis, but also has intelligent functions such as natural language understanding and dialogue management, providing users with a more intimate and convenient interactive experience. However, to achieve high-performance voice recognition of intelligent voice robots, there are many challenges. In addition, with the widespread popularization of smart devices, users' demand for voice interaction shows a diversified trend. In different scenarios, users expect to obtain personalized, natural and smooth interactive experience. However, the current voice recognition technology is difficult to dynamically optimize and customize according to user habits and interaction needs. There are often problems with inaccurate recognition and slow recognition speed due to the influence of noise and scenes. Therefore, there is an urgent need for a more intelligent voice recognition method for voice robots. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a speech recognition method and system for an intelligent speech robot to solve at least one of the above technical problems.

[0004] To achieve the above object, the present invention provides a speech recognition method for an intelligent speech robot, comprising the following steps:

[0005] Step S1: collecting user real-time conversation audio and user conversation monitoring images; performing noise reduction and separation processing on the user real-time conversation audio, and performing audio distortion optimization to obtain distortion-optimized conversation audio;

[0006] Step S2: performing micro-expression recognition on the user dialogue monitoring image and performing emotional state feature mining to generate the user's real-time emotional state feature;

[0007] Step S3: performing user semantic pause recognition on the distortion-optimized conversation audio and performing key word feature analysis to obtain key fields of each frequency domain unit;

[0008] Step S4: multi-scale convolution learning is performed on the key fields of each frequency domain unit, and then audio context association mining is performed to generate a user audio context association map;

[0009] Step S5: performing dynamic audio semantic recognition on the distortion-optimized conversation audio based on the user audio context association graph and the user's real-time emotional state characteristics, and performing current scene semantic optimization, thereby generating scene semantic recognition optimization data;

[0010] Step S6: Predict the interaction tendency of the scene semantic recognition optimization data, perform deep iterative learning, and build a user audio prediction and recognition model to complete the speech recognition task.

[0011] The present invention can improve the quality of user conversation audio, reduce environmental interference, and improve speech recognition accuracy through noise reduction and separation processing. Audio distortion optimization helps to improve audio quality, so that the speech recognition system can better understand the user's speech content. Micro-expression recognition and emotional state feature mining can help robots better understand user emotions and achieve emotional intelligent interaction. Generating real-time emotional state features helps intelligent robots make responses that are closer to user needs. Recognizing semantic pauses and analyzing key words helps to more accurately understand the user's speech content and improve the accuracy of speech recognition. Obtaining key fields can help robots more accurately capture user intentions and provide more accurate answers and services. Multi-scale convolutional learning and audio context association mining help to better understand user voice information Contextual relationships, improve the accuracy and coherence of speech recognition, and generate user audio context association maps to help robots understand user intentions and needs more comprehensively and provide a smarter interactive experience. Dynamic semantic recognition based on audio context association maps and real-time emotional state features can enable robots to more accurately understand user voice content and make corresponding optimizations based on the situation. Scene semantic recognition optimization data helps improve the interactive experience and allows robots to respond more intelligently to voice commands and needs in different scenarios. Interaction tendency prediction and deep iterative learning help build smarter audio prediction recognition models and improve the accuracy and efficiency of speech recognition. Completing speech recognition tasks can enable intelligent voice robots to better understand user intentions and achieve a more natural and smooth voice interaction experience.

[0012] Preferably, step S1 comprises the following steps:

[0013] Step S11: collecting user real-time conversation audio and user conversation monitoring images;

[0014] Step S12: Perform environmental sound difference recognition on the user's real-time conversation audio, and extract the real-time conversation audio and the real-time environmental audio;

[0015] Step S13: performing multi-frequency decomposition on the real-time ambient audio, thereby obtaining an ambient sound frequency domain of multiple frequencies;

[0016] Step S14: performing frequency-by-frequency noise component analysis on the ambient sound frequency domain of multiple frequencies to generate a noise component for each frequency;

[0017] Step S15: performing noise reduction and separation processing on the real-time conversation audio according to the noise component of each frequency, thereby obtaining noise reduction and separation conversation audio;

[0018] Step S16: performing audio distortion optimization on the noise reduction and separation conversation audio, thereby obtaining distortion optimized conversation audio.

[0019] The present invention can provide multimodal data by collecting real-time user conversation audio and monitoring images, helping the intelligent voice robot to understand the user's needs and situations more comprehensively. The collection of multimodal data helps to improve the accuracy of voice recognition and the interactive experience. Ambient sound difference recognition can help identify the sound characteristics in different environments and provide a reference for subsequent audio processing. Extracting real-time ambient audio can help the intelligent robot better adapt to the current environment and improve the accuracy of voice recognition. Multi-frequency decomposition can decompose ambient sound audio into components of different frequencies, which helps to analyze the characteristics of ambient sound more finely. Obtaining multiple frequency ambient sound frequency domains is helpful for subsequent noise analysis and audio processing. Frequency-by-frequency noise component analysis can help distinguish between components at different frequencies. Noise components provide guidance for subsequent noise reduction processing. Generating noise components for each frequency helps to remove environmental noise in a targeted manner and improve the clarity of conversation audio. Noise reduction and separation processing of conversation audio based on noise components of each frequency can effectively reduce the interference of environmental noise on speech recognition and improve speech recognition accuracy. Obtaining noise-reduced and separated conversation audio can enable the intelligent voice robot to capture user voice content more clearly and provide a more accurate interactive experience. Audio distortion optimization for noise-reduced and separated conversation audio can further improve audio quality and make voice signals clearer and more natural. Generating distortion-optimized conversation audio can help the intelligent voice robot understand user voice content more accurately and provide a smoother and more efficient voice recognition and interactive experience.

[0020] Preferably, the specific steps of step S16 are:

[0021] Define the timing frame length;

[0022] Decomposing the noise reduction and separation dialogue audio frame by frame according to the time sequence frame length, thereby obtaining a plurality of frame segment dialogue audios;

[0023] Fit the conversation timeline to the real-time conversation audio to obtain the user's real-time conversation timeline;

[0024] Using the user's real-time conversation timeline, compare the delay differences of multiple frame segments of conversation audio frame by frame, and extract the audio delay frame segments;

[0025] Performing delay calculation on the audio delay frame segment to obtain a delay parameter of the segment;

[0026] Performing audio delay compensation calculation on the segment delay parameter to generate an audio delay compensation value;

[0027] Performing time domain interpolation compensation on each frame segment dialogue audio based on the audio delay compensation value, thereby obtaining time delay compensated audio;

[0028] Correct the time-frequency distortion of the delay-compensated audio to construct distortion-optimized dialogue audio.

[0029] The present invention helps to divide audio data into continuous time frames by defining the timing frame length, and provides a time axis reference for subsequent audio processing and analysis. The definition of the timing frame length can help to accurately time-cut and process audio data. Frame-by-frame decomposition of the noise reduction and separation conversation audio can split the audio data into multiple time segments, which is convenient for subsequent time domain operations and analysis. Obtaining multiple frame segment conversation audio helps to finely process and analyze each time segment. Conversation time axis fitting and real-time conversation time axis extraction help to accurately capture the time information of user conversations and achieve audio and time alignment. Extracting the user's real-time conversation time axis can help the robot accurately track the user's conversation progress and provide more timely responses and services. Extracting audio delay frame segments by comparing frame-by-frame delay differences can discover the time delay in the audio, which is beneficial for subsequent delay processing. Providing a basis, calculating delay parameters helps to quantify audio delay and provide accurate data support for compensation and correction. The audio delay compensation calculation can generate accurate delay compensation values ​​based on the delay parameters, which are used to adjust the audio timeline and eliminate delay effects. The generated delay compensation values ​​can help maintain the time continuity of the audio and improve the accuracy and stability of speech processing. Through time domain interpolation compensation, the time domain adjustment of each frame segment dialogue audio is performed to eliminate delay effects and align the audio timeline. Generating time-delay compensated audio helps to improve audio clarity and continuity, improve user experience and speech recognition effects. Time-frequency distortion correction can optimize the spectral characteristics of the audio, improve audio quality and clarity, and provide better input for subsequent speech recognition and analysis. Constructing distortion-optimized dialogue audio can make speech clearer and more natural, and improve speech recognition accuracy and user experience.

[0030] Preferably, the specific steps of step S2 are:

[0031] Step S21: performing micro-expression recognition on the user dialogue monitoring image to obtain the user's real-time micro-expression data;

[0032] Step S22: performing face and mouth recognition on the user conversation monitoring image to extract the user's mouth area image;

[0033] Step S23: Analyze the mouth shape change of the user's mouth area image to generate the user's mouth shape change data;

[0034] Step S24: performing time-series pitch evolution on the real-time conversation audio to obtain a user conversation pitch curve;

[0035] Step S25: mining the emotional state features of the user's real-time micro-expression data, the user's lip shape change data, and the user's conversation tone curve, thereby generating the user's real-time emotional state features.

[0036] The present invention can capture the subtle emotional fluctuations of users during the conversation process through micro-expression recognition, and provide richer information for conversation emotion analysis. Extracting real-time micro-expression data of users helps to more accurately understand the emotional state of users, and provides a basis for subsequent emotion recognition and response adjustment. Through face mouth recognition and differentiation, the user's mouth area can be accurately located, which is convenient for subsequent analysis and recognition of lip shape changes. Extracting the user's mouth area image helps to capture the lip shape details information, and provides more reliable data support for emotional state feature mining. Through lip shape change analysis, the dynamic changes of the user's mouth in the conversation can be detected, reflecting the user's expression characteristics and emotional state, and generating the user's lip shape change The digitized data helps to understand the user's lip shape change pattern in detail and provide a more comprehensive perspective for emotional state analysis. The time-series pitch evolution analysis can capture the changing trend of the user's tone in the conversation and reveal the relationship between voice features and emotions. Extracting the user's conversation pitch curve helps to understand the user's voice change pattern and provide data support for emotion recognition and emotional state feature mining. The emotional state feature mining based on the user's real-time micro-expression data, lip shape change data and conversation pitch curve can grasp the user's emotional expression more comprehensively. Generating the user's real-time emotional state features can help the intelligent voice robot understand the user's emotional needs more accurately and provide more personalized and emotional service responses.

[0037] Preferably, step S3 specifically comprises the following steps:

[0038] Step S31: performing short-time Fourier transform processing on the distortion-optimized dialogue audio to obtain audio frequency domain data;

[0039] Step S32: performing user semantic pause recognition on the audio frequency domain data to extract the user conversation pause points;

[0040] Step S33: performing time-frequency decomposition according to the pause points of the user's conversation, and extracting multiple audio frequency domain units;

[0041] Step S34: performing key word feature analysis on multiple audio frequency domain units to obtain a key field of each frequency domain unit.

[0042] The present invention can convert distortion-optimized conversation audio into frequency domain representation through short-time Fourier transform processing, better understand the spectral characteristics of audio signals, extract audio frequency domain data to help analyze the frequency domain information of audio, and provide basic support for subsequent semantic analysis and keyword extraction, identify user semantic pauses to help understand the semantic structure in the conversation, accurately capture the user's speech rhythm and logical relationship, extract user conversation pause points to help segment the conversation into semantic units, and provide time reference for subsequent semantic analysis and keyword extraction, perform time-frequency decomposition according to user conversation pause points to cut the audio into multiple audio frequency domain units, which is convenient for analyzing the frequency domain characteristics of different semantic units, extract multiple audio frequency domain units to help deeply explore the frequency domain characteristic changes of different semantic units, and provide a more detailed data basis for key vocabulary feature analysis, through key vocabulary feature analysis, key information in each frequency domain unit can be identified, which helps to identify the key content and intention in the conversation, and extract the key fields of each frequency domain unit to capture the important semantic features in the conversation, and provide more accurate support for the speech recognition and understanding of the intelligent voice robot.

[0043] Preferably, the specific steps of step S4 are:

[0044] Step S41: performing convolution feature coding on the key field of each frequency domain unit, thereby obtaining a plurality of key field convolution codes;

[0045] Step S42: performing multi-scale convolution learning on multiple key field convolution codes to extract multiple convolution feature vectors;

[0046] Step S43: performing time series fitting on multiple convolution feature vectors to obtain a convolution feature vector sequence;

[0047] Step S44: extracting the timestamp of each key field based on the key field of each frequency domain unit;

[0048] Step S45: performing inter-field local dependency analysis on the convolution feature vector sequence according to the timestamp of each key field to extract the local dependency relationship of the convolution features;

[0049] Step S46: Perform audio context association mining on the local dependency of the convolutional features to generate a user audio context association graph.

[0050] The present invention can extract and encode the key fields in each frequency domain unit through convolution feature coding, which is helpful to capture key information. Extracting multiple key field convolution coding helps to encode the key information in each frequency domain unit, providing richer features for subsequent steps. Through multi-scale convolution learning, features can be captured at different scales to improve the ability to understand speech data. Extracting multiple convolution feature vectors helps to capture feature representations at different levels and provide richer feature information for subsequent sequence analysis. Through time series fitting, multiple convolution feature vectors can be combined into a sequence, retaining time information, which is helpful to understand the time series relationship of speech. Extracting convolution feature vector sequence helps to perform time series analysis on speech data, providing serialized feature representation for subsequent association analysis, and extracting each key The timestamp of the field helps to determine the position of each field in the speech sequence and provide a time reference for subsequent local dependency analysis. The timestamp information can help accurately locate the temporal position of key fields and help understand the temporal structure in speech data. The local dependency analysis between fields can reveal the correlation between different fields and help understand the local dependencies in speech data. Extracting convolutional feature local dependencies helps capture the correlation information between fields in speech data and provides a basis for audio context association mining. Audio context association mining can deeply understand the contextual relationships in speech data and help better interpret the user's speech content. Generating a user audio context association map helps to present the correlation in speech data and provide richer information support for speech recognition and understanding of intelligent voice robots.

[0051] Preferably, the specific steps of step S5 are:

[0052] Step S51: performing user interaction scene clustering analysis on the user's real-time emotional state characteristics according to the real-time ambient audio to obtain the current user interaction scene type;

[0053] Step S52: performing dynamic audio semantic recognition on the distortion-optimized conversation audio based on the user audio context association graph, thereby obtaining dynamic audio recognition data;

[0054] Step S53: Perform current scene semantic optimization on the dynamic audio recognition data based on the user interaction scene type, thereby generating scene semantic recognition optimization data.

[0055] The present invention can infer the user's emotional state based on real-time environmental audio through cluster analysis of user emotional state features, which is helpful for understanding the user's emotional feedback; dynamic audio semantic recognition based on user audio context association map is helpful for more accurate semantic understanding based on the user's voice content and context information; extracting dynamic audio recognition data is helpful for capturing the semantic content in real-time conversations and providing data support for subsequent scene semantic optimization; current scene semantic optimization of dynamic audio recognition data based on user interaction scene type is helpful for adjusting semantic information according to specific scenes and improving the accuracy of speech recognition; generating scene semantic recognition optimization data is helpful for optimizing the semantic understanding of conversations according to different scenes and improving the effect and accuracy of intelligent voice robots in real-time interaction; extracting the current user interaction scene type is helpful for associating the user's real-time emotional state with the specific interaction scene and providing more specific background information for subsequent semantic understanding.

[0056] Preferably, the specific steps of step S6 are:

[0057] Step S61: performing interactive behavior evolution on the distortion-optimized conversation audio to generate real-time interactive behavior data of the user;

[0058] Step S62: Performing personalized behavior habit analysis on the user's real-time interaction behavior data, thereby obtaining the current user's interaction behavior habit characteristics;

[0059] Step S63: predicting the interaction tendency of the current user's interaction behavior habit characteristics to obtain user interaction tendency prediction data;

[0060] Step S64: Use the user interaction tendency prediction data to perform deep iterative learning on the scene semantic recognition optimization data to build a user audio prediction recognition model to complete the speech recognition task.

[0061] The present invention can simulate the real-time interaction process of users by performing interactive behavior evolution on distortion-optimized conversation audio, and generate authentic interactive behavior data. Generating real-time interactive behavior data of users helps to capture the behavioral patterns of users during the conversation process, and provides a data basis for subsequent personalized analysis. Performing personalized behavior habit analysis helps to identify the personalized characteristics and behavioral preferences of users in the interaction, and provides personalized services for intelligent systems. Extracting the current user's interactive behavior habit characteristics helps to deeply understand the user's interactive behavior patterns, and provides a basis for subsequent interactive tendency predictions. By performing interactive tendency predictions on the current user's interactive behavior habit characteristics, the user's interactive preferences in specific situations can be predicted, and the personalization level of the interaction can be improved. Generating user interactive tendency prediction data helps the system to better understand the user's needs and expectations, and provides guidance for the subsequent construction of a speech recognition model. Using the user interactive tendency prediction data to perform deep iterative learning on the scene semantic recognition optimization data can construct a more personalized user audio prediction and recognition model. Constructing a user audio prediction and recognition model helps to improve the accuracy and personalization level of speech recognition, thereby completing speech recognition tasks and providing a better user experience.

[0062] In this specification, a speech recognition system of an intelligent speech robot is provided, which is used to execute the speech recognition method of the intelligent speech robot as described above, including:

[0063] The audio optimization module collects the user's real-time conversation audio and the user's conversation monitoring image; performs noise reduction and separation processing on the user's real-time conversation audio, and performs audio distortion optimization to obtain distortion-optimized conversation audio;

[0064] The emotional state module recognizes micro-expressions in user conversation monitoring images and mines emotional state features to generate real-time emotional state features of users;

[0065] The key vocabulary module recognizes user semantic pauses in the distortion-optimized conversation audio and performs key vocabulary feature analysis to obtain the key fields of each frequency domain unit;

[0066] The context association module performs multi-scale convolution learning on the key fields of each frequency domain unit, and then performs audio context association mining to generate a user audio context association map;

[0067] The audio semantic recognition module performs dynamic audio semantic recognition on the distortion-optimized conversation audio based on the user audio context association graph and the user's real-time emotional state characteristics, and performs semantic optimization of the current scene, thereby generating scene semantic recognition optimization data;

[0068] The interaction tendency prediction module predicts the interaction tendency of the scene semantic recognition optimization data, performs deep iterative learning, and builds a user audio prediction and recognition model to complete the speech recognition task.

[0069] The present invention helps to improve the quality of user conversation audio, reduce interference and noise, and improve the accuracy of speech recognition through noise reduction separation processing and audio distortion optimization. Micro-expression recognition and emotional state feature mining can better understand the user's emotional state, which helps the intelligent system to respond to user emotions more intelligently. User semantic pause recognition and key vocabulary feature analysis help capture key information in the conversation and improve the accuracy and comprehension of speech recognition. Multi-scale convolution learning and audio context association mining help to build a user audio context association map and provide richer background information for speech understanding. Dynamic audio semantic recognition and current scene semantic optimization can improve the accuracy and personalization level of speech recognition according to the user's emotional state and context information. Interaction tendency prediction and user audio prediction recognition model construction can achieve more intelligent speech recognition according to the user's interaction behavior pattern and semantic tendency, and provide services that are closer to user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A schematic diagram of the steps of a speech recognition method of an intelligent speech robot according to the present invention;

[0071] Figure 2 Detailed implementation flow chart of step S1;

[0072] Figure 3 Detailed implementation flow chart of step S2;

[0073] Figure 4 Detailed implementation flow chart of step S3. DETAILED DESCRIPTION

[0074] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0075] This application example provides a method and system for speech recognition of an intelligent speech robot. The execution subject of the method and system for speech recognition of an intelligent speech robot includes but is not limited to: mechanical equipment, data processing platform, cloud server node, network upload device, etc. equipped with the system can be regarded as the general computing node of this application, and the data processing platform includes but is not limited to: at least one of an audio image management system, an information management system, and a cloud data management system.

[0076] See also Figures 1 to 4 The present invention provides a speech recognition method for an intelligent speech robot, and the speech recognition method for an intelligent speech robot comprises the following steps:

[0077] Step S1: collecting user real-time conversation audio and user conversation monitoring images; performing noise reduction and separation processing on the user real-time conversation audio, and performing audio distortion optimization to obtain distortion-optimized conversation audio;

[0078] Step S2: performing micro-expression recognition on the user dialogue monitoring image and performing emotional state feature mining to generate the user's real-time emotional state feature;

[0079] Step S3: performing user semantic pause recognition on the distortion-optimized conversation audio and performing key word feature analysis to obtain key fields of each frequency domain unit;

[0080] Step S4: multi-scale convolution learning is performed on the key fields of each frequency domain unit, and then audio context association mining is performed to generate a user audio context association map;

[0081] Step S5: performing dynamic audio semantic recognition on the distortion-optimized conversation audio based on the user audio context association graph and the user's real-time emotional state characteristics, and performing current scene semantic optimization, thereby generating scene semantic recognition optimization data;

[0082] Step S6: Predict the interaction tendency of the scene semantic recognition optimization data, perform deep iterative learning, and build a user audio prediction and recognition model to complete the speech recognition task.

[0083] The present invention can improve the quality of user conversation audio, reduce environmental interference, and improve speech recognition accuracy through noise reduction and separation processing. Audio distortion optimization helps to improve audio quality, so that the speech recognition system can better understand the user's speech content. Micro-expression recognition and emotional state feature mining can help robots better understand user emotions and achieve emotional intelligent interaction. Generating real-time emotional state features helps intelligent robots make responses that are closer to user needs. Recognizing semantic pauses and analyzing key words helps to more accurately understand the user's speech content and improve the accuracy of speech recognition. Obtaining key fields can help robots more accurately capture user intentions and provide more accurate answers and services. Multi-scale convolutional learning and audio context association mining help to better understand user voice information Contextual relationships, improve the accuracy and coherence of speech recognition, and generate user audio context association maps to help robots understand user intentions and needs more comprehensively and provide a smarter interactive experience. Dynamic semantic recognition based on audio context association maps and real-time emotional state features can enable robots to more accurately understand user voice content and make corresponding optimizations based on the situation. Scene semantic recognition optimization data helps improve the interactive experience and allows robots to respond more intelligently to voice commands and needs in different scenarios. Interaction tendency prediction and deep iterative learning help build smarter audio prediction recognition models and improve the accuracy and efficiency of speech recognition. Completing speech recognition tasks can enable intelligent voice robots to better understand user intentions and achieve a more natural and smooth voice interaction experience.

[0084] In the embodiment of the present invention, refer to Figure 1 , is a schematic flow chart of the steps of a speech recognition method of an intelligent speech robot of the present invention. In this example, the steps of the speech recognition method of the intelligent speech robot include:

[0085] Step S1: collecting user real-time conversation audio and user conversation monitoring images; performing noise reduction and separation processing on the user real-time conversation audio, and performing audio distortion optimization to obtain distortion-optimized conversation audio;

[0086] In this embodiment, a high-quality microphone (such as a USB microphone or a professional recording device) is selected to improve the clarity of audio acquisition. Use audio acquisition software (such as Audacity, PyAudio) to set an appropriate sampling rate (such as 44.1kHz) and bit depth (such as 16 bits). Configure recording parameters, including the number of channels (mono or stereo) and recording duration. Start audio recording and collect the user's conversation audio in real time. Make sure the background environment is as quiet as possible to reduce the impact of external noise on audio quality. Use a high-resolution camera (such as a high-definition webcam or a professional camera) to capture user images. Set a suitable resolution (such as 1920x1080) and frame rate (such as 30fps) to ensure the clarity and smoothness of the image. Start an image acquisition program (such as OpenCV) to capture the user's facial image in real time. Make sure the camera is aimed at the user and try to avoid light reflections and shadows that affect the image quality. Perform a short-time Fourier transform (STFT) on the recorded audio signal and convert it into a frequency domain representation. Select the silent segment of the audio signal and estimate the background noise spectrum. Subtract the estimated noise spectrum from the spectrum of the audio signal to obtain the denoised spectrum. Apply the Inverse Short Time Fourier Transform (ISTFT) to the denoised spectrum to convert it back to the time domain signal. Identify the type of distortion in the audio signal (such as clipping distortion, noise distortion, etc.) and determine the target that needs to be optimized. Use dynamic range compression, equalization, resampling and other techniques to optimize audio. Adjust the dynamic range of the audio signal, use a compressor to reduce the loud parts and boost the low-loud parts. Use an equalizer to adjust the frequency response and enhance or weaken specific frequencies (such as the mid-frequency band of human voice). If necessary, resample the audio signal to match the desired playback or processing format. Use audio processing libraries (such as PyDub, Librosa) to achieve distortion optimization of audio.

[0087] Step S2: performing micro-expression recognition on the user dialogue monitoring image and performing emotional state feature mining to generate the user's real-time emotional state feature;

[0088] In this embodiment, a high-resolution camera is used to capture the user's facial image in real time. The OpenCV library can be used to capture video streams, and frames are saved regularly for subsequent processing. The captured images are preprocessed, including grayscale, normalization, denoising, etc., to improve the accuracy of subsequent micro-expression recognition. Use a deep learning model (such as a convolutional neural network, CNN) for micro-expression recognition. It is possible to consider using open source models (such as FER, OpenFace, etc.) for facial expression recognition. Use a pre-trained model to extract facial features. For micro-expressions, changes in facial key points (such as eyebrows, mouth, eyes, etc.) are usually focused on. The extracted features are input into a micro-expression classifier (such as a support vector machine SVM, random forest, etc.) to identify the user's micro-expressions. According to the recognition results of the micro-expressions, the micro-expressions are mapped to emotional states (such as happiness, anger, surprise, sadness, etc.). Emotional tag libraries (such as Ekman's basic emotions) can be used for mapping. Emotion-related features are extracted, such as: micro-expression duration, the amplitude of facial feature changes, and the intensity of expression, and the extracted emotional features are organized into structured data for subsequent analysis and storage. The generated user's real-time emotional state features are stored in a database or file for subsequent analysis and use. Libraries such as Matplotlib can be used to visualize the emotional state for easy observation and analysis.

[0089] Step S3: performing user semantic pause recognition on the distortion-optimized conversation audio and performing key word feature analysis to obtain key fields of each frequency domain unit;

[0090] In this embodiment, the waveform data obtained from the voice input is subjected to distortion optimization processing. Here, a waveform distortion compensation algorithm based on an acoustic model can be used. The pre-trained acoustic model is used to perform spectrum compensation on the input voice waveform to eliminate distortion caused by hardware, environment and other factors. The optimized voice waveform is subjected to semantic pause recognition. Here, a semantic parsing technology based on deep learning is used. Specifically, a LSTM-based neural network model is constructed, and optimized waveform features (such as MFCC, spectrum, etc.) are input. The semantic pause probability of each time frame is output, and a certain probability threshold is set. Time frames above the threshold are recognized as semantic pause points, so that consecutive pauses can be recognized. The continuous speech waveform is divided into several semantic segments, and keyword analysis is performed on each semantic segment. First, the speech features of each segment are extracted, including acoustic features (such as MFCC, spectrum, etc.) and semantic features (such as word embedding, emotion, etc.). Then, the keyword extraction algorithm based on the attention mechanism is used to automatically identify the keywords in each semantic segment. The keywords here include not only high-frequency words, but also keywords with specific semantics. The keyword features of each semantic segment are mapped to the corresponding frequency domain units to obtain the keyword information of each frequency domain unit. In this way, the semantic key field information of each frequency domain unit is obtained, which provides an important basis for subsequent speech analysis and understanding.

[0091] Step S4: multi-scale convolution learning is performed on the key fields of each frequency domain unit, and then audio context association mining is performed to generate a user audio context association map;

[0092] In this embodiment, the key fields of each frequency domain unit are loaded, and these key fields will be used as the basis for subsequent analysis. The key fields are encoded, usually using methods such as one-hot encoding or word embedding, so that their format is suitable for input into a convolutional neural network. A multi-scale convolutional neural network is designed, and convolution kernels of different sizes are used to capture the multi-scale features of the key fields. The structure of the network usually includes multiple convolution layers and pooling layers to extract local features and reduce dimensions. The prepared key field data is used for model training, and the data is divided into a training set and a validation set. The network weights are optimized through multiple rounds of iterations to improve the accuracy and generalization ability of the model. The trained multi-scale convolutional model is used to infer the key field data and extract context features of the audio. These features can reflect the semantic information of the audio content and the relationship with other audio clips. According to the extracted context features, an audio context association map is constructed, which usually involves calculating the similarity or correlation between features. Features with high similarity will be regarded as semantically associated. The generated audio context association map is visualized to help understand the relationship between the audio content. At the same time, the map data is saved for subsequent analysis and use.

[0093] Step S5: performing dynamic audio semantic recognition on the distortion-optimized conversation audio based on the user audio context association graph and the user's real-time emotional state characteristics, and performing current scene semantic optimization, thereby generating scene semantic recognition optimization data;

[0094] In this embodiment, relevant information is extracted from the previously generated user audio context association map. This information will help understand the context and emotion of the audio, obtain the user's real-time emotional state features, and these features will be used to adjust the context of semantic recognition. A suitable audio semantic recognition model is selected, such as a sequence-to-sequence model (Seq2Seq), a long short-term memory network (LSTM), or a Transformer model based on deep learning. These models can process time series data and generate semantic outputs. Features are extracted from audio signals. Commonly used features include Mel-frequency cepstral coefficients (MFCC), pitch, energy, etc. These features will be used as input to the model. The extracted features are input into the semantic recognition model to dynamically identify the semantic content in the audio. The model will output the recognition The semantic label or intention of the audio context association graph and the characteristics of the user's emotional state are used to adjust the recognition results. For example, when the user's emotions fluctuate greatly, it may be necessary to prioritize the recognition of semantic content related to emotions to improve the recognition accuracy. According to the recognition results and the user's emotional state, the characteristics of the current scene are extracted, which may include the user's interaction type (such as inquiry, feedback) and emotions (such as anxiety, excitement). An optimization strategy is formulated to tune the semantic recognition results based on the extracted scene features. For example, if the user appears anxious, the threshold of the recognition algorithm can be adjusted to make it more inclined to recognize positive or soothing semantics. According to the optimized semantic recognition results and scene characteristics, scene semantic recognition optimization data is constructed, which will be used for further analysis and decision support.

[0095] Step S6: Predict the interaction tendency of the scene semantic recognition optimization data, perform deep iterative learning, and build a user audio prediction and recognition model to complete the speech recognition task.

[0096] In this embodiment, relevant information is extracted from the optimized data according to scene semantic recognition, including the user's interaction features, emotional state and semantic recognition results, and the data is preprocessed, including removing missing values, standardizing or normalizing features to ensure the effectiveness of model training, selecting features related to interaction tendencies, such as user historical behaviors, emotional states and current scene features, and selecting a suitable prediction model, such as logistic regression, decision trees, random forests or deep learning models (such as LSTM, GRU). Deep learning models are suitable for processing time series data and can capture the dynamic characteristics of user interactions. The optimized data is divided into a training set and a test set. Usually 80% of the data is used for training and 20% of the data is used for testing to verify the accuracy and generalization ability of the model. The selected model is trained using the training set, and the model parameters are optimized to improve the prediction accuracy. The cross-validation technique can be used to evaluate the model performance to ensure its stability on different data sets. The test set is input into the trained model to generate the user's interaction tendency prediction results, which can be used by the user in Behavior prediction in specific situations, such as active questioning, feedback or silence, etc., designs a deep iterative learning framework that allows the model to update itself after receiving new interaction data. Incremental learning methods can be used to enable the model to continuously absorb new data and adjust the prediction strategy. The user's actual interaction behavior is compared with the predicted results, the prediction error is calculated, and this information is fed back to the model to optimize its learning process. In each round of iteration, the model is retrained using new interaction data, and hyperparameters are adjusted to improve the accuracy and robustness of the model. According to the results of the interaction tendency prediction, an audio prediction recognition model is designed. The model can be a sequence-to-sequence deep learning network that can process the input audio features and output the corresponding recognition results. Audio features (such as MFCC, spectrogram, etc.) are extracted from the distortion-optimized conversation audio and used as the input of the model. The scene semantic recognition is used to optimize the data and its corresponding interaction tendency, and the audio prediction recognition model is trained. The performance of the model on the test set is evaluated to ensure its effectiveness in practical applications.

[0097] In this embodiment, refer to Figure 2 , is a flowchart of detailed implementation steps of step S1. In this embodiment, the detailed implementation steps of step S1 include:

[0098] Step S11: collecting user real-time conversation audio and user conversation monitoring images;

[0099] Step S12: Perform environmental sound difference recognition on the user's real-time conversation audio, and extract the real-time conversation audio and the real-time environmental audio;

[0100] Step S13: performing multi-frequency decomposition on the real-time ambient audio, thereby obtaining an ambient sound frequency domain of multiple frequencies;

[0101] Step S14: performing frequency-by-frequency noise component analysis on the ambient sound frequency domain of multiple frequencies to generate a noise component for each frequency;

[0102] Step S15: performing noise reduction and separation processing on the real-time conversation audio according to the noise component of each frequency, thereby obtaining noise reduction and separation conversation audio;

[0103] Step S16: performing audio distortion optimization on the noise reduction and separation conversation audio, thereby obtaining distortion optimized conversation audio.

[0104] In this embodiment, a high-quality microphone is used to ensure clear speech, a high-definition camera is used for image acquisition, and the acquisition environment is ensured to be relatively quiet to reduce interference to improve the quality of audio and image. An audio acquisition software (such as Audacity, OBSStudio) is used to record the user conversation audio in real time, and the recording format (such as WAV, MP3) is ensured to meet the subsequent processing requirements. A video acquisition software or device is used to record the user conversation monitoring image, and the image quality is ensured to be high enough to include the user's expression and action. A signal processing algorithm (such as short-time Fourier transform STFT) is used to analyze the time-frequency characteristics of the audio signal, identify the difference between the conversation audio and the ambient audio, extract the characteristics of the audio signal (such as MFCC, spectrogram), and use a classification algorithm (such as support vector machine, convolutional neural network) to identify the ambient sound and the conversation sound, realize real-time separation of the conversation audio and the ambient audio, generate two independent audio streams, perform fast Fourier transform (FFT) on the real-time ambient audio signal, convert the time domain signal into a frequency domain signal, extract the spectrum information of the ambient audio, analyze the frequency distribution, and use a filter group (such as bandpass A filter is used to decompose the frequency domain signal to obtain multiple frequency components for subsequent analysis. Signal processing techniques (such as wavelet transform and spectral subtraction) are used to analyze the noise components of each frequency component and identify the noise characteristics. Each frequency component is analyzed to generate the corresponding noise component, and its amplitude and frequency characteristics are recorded. An appropriate noise reduction algorithm (such as spectral subtraction, Wiener filtering, and deep learning noise reduction network) is selected for noise reduction of real-time conversation audio. The identified noise components are used to perform noise reduction processing on the real-time conversation audio to reduce the impact of environmental noise on the conversation audio. After the processing is completed, noise-reduced and separated conversation audio is generated to ensure that the voice is clear and discernible. Audio analysis software (such as Adobe Audition) is used to detect the distortion of the audio after noise reduction and evaluate the audio quality. Audio processing techniques (such as dynamic range compression and equalizer adjustment) are used to optimize the distortion of the noise-reduced audio to improve the sound quality. Audio post-processing is performed to ensure that the optimized audio is clear and natural, eliminate possible distortion and artifacts, and export the optimized audio to a high-quality format (such as WAV or FLAC) for subsequent use and analysis.

[0105] In this embodiment, the specific steps of step S16 are:

[0106] Define the timing frame length;

[0107] Decomposing the noise reduction and separation dialogue audio frame by frame according to the time sequence frame length, thereby obtaining a plurality of frame segment dialogue audios;

[0108] Fit the conversation timeline to the real-time conversation audio to obtain the user's real-time conversation timeline;

[0109] Using the user's real-time conversation timeline, compare the delay differences of multiple frame segments of conversation audio frame by frame, and extract the audio delay frame segments;

[0110] Performing delay calculation on the audio delay frame segment to obtain a delay parameter of the segment;

[0111] Performing audio delay compensation calculation on the segment delay parameter to generate an audio delay compensation value;

[0112] Performing time domain interpolation compensation on each frame segment dialogue audio based on the audio delay compensation value, thereby obtaining time delay compensated audio;

[0113] Correct the time-frequency distortion of the delay-compensated audio to construct distortion-optimized dialogue audio.

[0114] In this embodiment, a suitable timing frame length is determined according to the application scenario (such as real-time communication, speech recognition), usually in milliseconds (such as 20ms, 50ms), and the overlap between frames (such as 50% overlap) is considered to ensure the continuity and integrity of the audio information. According to the defined timing frame length, the noise-reduced and separated conversation audio is decomposed frame by frame using a sliding window method, and each frame segment is saved as an independent audio file or data structure for subsequent processing. The energy and pitch changes of the real-time conversation audio are analyzed, and the start and end time of the conversation are identified by audio feature extraction (such as MFCC). The timeline of the user's real-time conversation is generated by interpolation or smoothing to ensure that the timeline reflects the actual rhythm of the conversation. The start time of each frame segment is compared with the user's real-time conversation timeline, and the delay difference is calculated. According to the calculated delay difference, the frame segments with significant delays are extracted and recorded. Record its timestamp and delay amount, measure the duration of the extracted audio delay frame fragments, calculate the delay parameters of each fragment, usually expressed in milliseconds, calculate the compensation value of each delayed fragment based on the delay parameters, usually the negative value of the delay time (for example, if the delay is 50ms, the compensation value is -50ms), select a suitable interpolation method (such as linear interpolation, spline interpolation) to perform time domain interpolation compensation on each frame fragment, adjust the time of each frame fragment based on the calculated audio delay compensation value, ensure that the audio timeline is consistent with the real dialogue timeline, use audio analysis tools (such as spectrum analysis) to detect the audio distortion after delay compensation, evaluate the sound quality, use dynamic range compression, equalizer adjustment or deep learning algorithms (such as GAN) to correct the distortion, optimize the audio quality, generate distortion-optimized dialogue audio, and ensure that its clarity and naturalness meet the requirements.

[0115] In this embodiment, refer to Figure 3 , is a flowchart of detailed implementation steps of step S2. In this embodiment, the detailed implementation steps of step S2 include:

[0116] Step S21: performing micro-expression recognition on the user dialogue monitoring image to obtain the user's real-time micro-expression data;

[0117] Step S22: performing face and mouth recognition on the user conversation monitoring image to extract the user's mouth area image;

[0118] Step S23: Analyze the mouth shape change of the user's mouth area image to generate the user's mouth shape change data;

[0119] Step S24: performing time-series pitch evolution on the real-time conversation audio to obtain a user conversation pitch curve;

[0120] Step S25: mining the emotional state features of the user's real-time micro-expression data, the user's lip shape change data, and the user's conversation tone curve, thereby generating the user's real-time emotional state features.

[0121] In this embodiment, the user dialogue monitoring image is subjected to denoising, color correction and lighting balance to improve the accuracy of micro-expression recognition, a deep learning model (such as convolutional neural network CNN, OpenFace or EmotionNet) is used for micro-expression recognition, if a custom model is used, it is necessary to use labeled micro-expression data for training to improve the recognition accuracy, the pre-processed image is input into the micro-expression recognition model, the user's real-time micro-expression data is obtained, the user's micro-expression category (such as happiness, sadness, surprise, etc.) and its duration are extracted and saved, a Haar feature classifier or a deep learning method (such as MTCNN or Dlib) is used to detect the face area in the image, the face key point detection technology (such as FacialLandmarkDetection) is used to identify the key point position of the mouth, the mouth area image is cropped according to the key point coordinates, and saved as an independent image file, the mouth area image is marked with key points, the mouth opening and closing degree, lip shape changes and other features are identified, and morphological processing is used The lip shape change data is analyzed by using techniques (such as expansion and corrosion), and the mouth shape change data of each frame is recorded, such as the degree of opening and closing, lip shape features, etc., to generate the user's lip shape change data, and the real-time conversation audio is denoised and normalized to ensure the clarity of the audio signal. The pitch curve of the audio is extracted using a pitch tracking algorithm (such as the YIN algorithm, HPS, or a pitch detection method based on deep learning), and the curve of the pitch change over time is recorded to generate the user's conversation pitch curve. The user's real-time micro-expression data, lip shape change data, and pitch curve are integrated into a data structure to prepare for subsequent analysis. According to psychological theories and previous studies, relevant features (such as micro-expression types, degree of mouth shape change, and pitch change amplitude) are selected as indicators of emotional state. The emotion recognition model is trained using machine learning algorithms (such as support vector machines, random forests, and deep learning). The user's emotional state features are generated based on the integrated features. The user's real-time emotional state features, such as happiness, anxiety, or neutrality, are output through model prediction, and their confidence is recorded.

[0122] In this embodiment, refer to Figure 4 , is a flowchart of detailed implementation steps of step S3. In this embodiment, the detailed implementation steps of step S3 include:

[0123] Step S31: performing short-time Fourier transform processing on the distortion-optimized dialogue audio to obtain audio frequency domain data;

[0124] Step S32: performing user semantic pause recognition on the audio frequency domain data to extract the user conversation pause points;

[0125] Step S33: performing time-frequency decomposition according to the pause points of the user's conversation, and extracting multiple audio frequency domain units;

[0126] Step S34: performing key word feature analysis on multiple audio frequency domain units to obtain a key field of each frequency domain unit.

[0127] In this embodiment, an audio processing library (such as Librosa, SciPy) is used to load the distortion-optimized conversation audio, ensure that the sampling rate and format of the audio signal are suitable for subsequent processing, select a suitable window function and window length (such as 1024 points) to balance the frequency resolution and time resolution, use short-time Fourier transform to convert the time domain audio signal into a frequency domain representation, obtain complex spectrum data, calculate the amplitude spectrum and phase spectrum of STFT for subsequent analysis, calculate the energy of each frame (such as through amplitude spectrum calculation), and set an energy threshold, identify frames with energy below the threshold as potential pause points, smooth the energy signal (such as using a moving average) to reduce the interference of noise on pause detection, extract the timestamp of the pause point based on the frames with energy below the threshold, record the start and end time of the pause, save the identified pause point as a list or data structure for subsequent processing, and extract the energy of each frame based on the energy of the frame. The pause timestamp is used to divide the audio signal into multiple sections, each section includes one or more frequency domain units, and the original audio signal is cut into multiple audio segments using the identified pause points for subsequent analysis. Short-time Fourier transform is applied again to each extracted audio segment to obtain the frequency domain representation of each time period, and the frequency domain data of each audio segment is saved as a frequency domain unit for subsequent analysis. An audio feature extraction algorithm (such as MFCC, Chroma feature, pitch, etc.) is used to analyze each frequency domain unit to extract key vocabulary features, and corresponding features are calculated for each frequency domain unit to generate a feature matrix. For example, MFCC is used to extract key vocabulary features, and a model (such as HMM, neural network) is used to analyze the extracted features to identify key vocabulary in the frequency domain unit, and the key vocabulary features of each frequency domain unit are saved as a data structure for further analysis and processing.

[0128] In this embodiment, step S4 includes the following steps:

[0129] Step S41: performing convolution feature coding on the key field of each frequency domain unit, thereby obtaining a plurality of key field convolution codes;

[0130] Step S42: performing multi-scale convolution learning on multiple key field convolution codes to extract multiple convolution feature vectors;

[0131] Step S43: performing time series fitting on multiple convolution feature vectors to obtain a convolution feature vector sequence;

[0132] Step S44: extracting the timestamp of each key field based on the key field of each frequency domain unit;

[0133] Step S45: performing inter-field local dependency analysis on the convolution feature vector sequence according to the timestamp of each key field to extract the local dependency relationship of the convolution features;

[0134] Step S46: Perform audio context association mining on the local dependency of the convolutional features to generate a user audio context association graph.

[0135] In this embodiment, a small convolutional neural network is designed, including a convolutional layer, an activation function (such as ReLU) and a pooling layer to extract the features of key fields. The key fields of each frequency domain unit are taken as input and feature extraction is performed through the convolutional layer. After convolution and pooling operations, convolutional encodings of multiple key fields are obtained. A multi-scale convolutional neural network is designed, and convolution kernels of different sizes (such as 3x3, 5x5 and 7x7) are used to capture features of different scales. Multiple key field convolutional encodings are input into a multi-scale convolutional network. After multi-scale convolution and pooling, multiple convolution feature vectors are extracted. A long short-term memory network (LSTM) or a gated recurrent unit (GRU) is used to fit the time series of convolution feature vectors. An LSTM or GRU model is designed, and multiple convolution feature vectors are input for time series learning. Through model training and prediction, a time series of convolution feature vectors is obtained. The key fields of each frequency domain unit are analyzed and their respective start and end timestamps are recorded. Traverse the key fields of each frequency domain unit, extract their corresponding timestamps, and save them as data structures (such as dictionaries or lists). Select a suitable model (such as conditional random fields, graph neural networks) to analyze the local dependencies between fields in the convolution feature vector sequence. According to the extracted timestamps, build a dependency graph of key fields and analyze the mutual influence between fields. Through model calculation, extract the local dependencies of convolution features and record related features. Use graph mining or sequence mining technology to analyze the local dependencies of convolution features and mine audio context associations. Use historical data to train the model and evaluate the predictability and relevance of audio context. Generate an audio context association map based on the mining results to show the relationship between fields and the context information of the conversation audio.

[0136] In this embodiment, the specific steps of step S5 are:

[0137] Step S51: performing user interaction scene clustering analysis on the user's real-time emotional state characteristics according to the real-time ambient audio to obtain the current user interaction scene type;

[0138] Step S52: performing dynamic audio semantic recognition on the distortion-optimized conversation audio based on the user audio context association graph, thereby obtaining dynamic audio recognition data;

[0139] Step S53: Perform current scene semantic optimization on the dynamic audio recognition data based on the user interaction scene type, thereby generating scene semantic recognition optimization data.

[0140] In this embodiment, environmental audio data including background noise, audio features, etc. are obtained from the real-time conversation of the user, and the features of the environmental audio are extracted using methods (such as MFCC, Chroma features, etc.), and converted into numerical representations for clustering analysis. According to the characteristics of the data, a suitable clustering algorithm (such as K-Means, DBSCAN or hierarchical clustering) is selected to analyze the user's interaction scene, the extracted audio features are input into the clustering algorithm, and the clustering operation is performed to obtain different types of user interaction scenes, and the effectiveness of the clustering results is evaluated to ensure that the clustering results of different scenes are clear and distinguishable, and the type of the current user interaction scene (such as work, leisure, social, etc.) is recorded and output to provide a basis for subsequent processing, and a suitable audio semantic recognition model (such as deep learning model, HMM, C++) is selected according to application requirements. RF, etc.), input the distortion-optimized conversation audio into the semantic recognition model for dynamic audio recognition. The model outputs dynamic semantic recognition data of the audio, including recognized keywords, intents, and context information. A semantic optimization algorithm based on the scene type (such as rule-based optimization, deep learning optimization, etc.) is used to make semantic adjustments according to the user's interaction scene type. For the current user interaction scene type, the dynamic audio recognition data is optimized and processed, and keywords, intents, and contexts are adjusted to ensure that they adapt to the current scene. The optimized scene semantic recognition data is output to ensure its accuracy and relevance. The effectiveness of the optimization results is evaluated through user feedback and test data, and feedback information is collected to further improve the optimization algorithm. Based on the feedback information, the semantic recognition model and optimization algorithm are regularly updated to improve the overall performance of the system.

[0141] In this embodiment, the specific steps of step S6 are:

[0142] Step S61: performing interactive behavior evolution on the distortion-optimized conversation audio to generate real-time interactive behavior data of the user;

[0143] Step S62: Performing personalized behavior habit analysis on the user's real-time interaction behavior data, thereby obtaining the current user's interaction behavior habit characteristics;

[0144] Step S63: predicting the interaction tendency of the current user's interaction behavior habit characteristics to obtain user interaction tendency prediction data;

[0145] Step S64: Use the user interaction tendency prediction data to perform deep iterative learning on the scene semantic recognition optimization data to build a user audio prediction recognition model to complete the speech recognition task.

[0146] In this embodiment, the time domain and frequency domain features (such as energy, spectrum, MFCC, etc.) of the audio signal are extracted to provide basic data for the evolution of the interactive behavior. A state transition model (such as a Markov model) or a deep learning model (such as LSTM) is used to model the evolution of the user's interactive behavior. The model is trained through historical interactive data so that it can capture the pattern and change of the user's interactive behavior. The trained model is used to infer the real-time audio data to generate the user's real-time interactive behavior data. The generated user's real-time interactive behavior data is cleaned and sorted, and missing values ​​and noise are processed to ensure data quality. Cluster analysis (such as K-Means, DBSCAN) or classification algorithms (such as decision trees, random forests) are used to identify the user's behavioral habits. Features (such as frequency, duration, interactive content, etc.) are extracted from the interactive behavior data, and the features that best represent the user's habits are selected. The selected algorithm is applied to analyze the user's behavioral habits to generate the interactive behavior habit features of the current user, and a suitable prediction model (such as linear regression) is selected according to the data characteristics. , support vector machines, deep learning models, etc.), use historical interaction behavior habit features and corresponding interaction result data to train the prediction model, evaluate the accuracy of the model through cross-validation or holdout method, adjust parameters to improve prediction performance, input the current user's interaction behavior habit features into the trained model, generate user interaction tendency prediction data, indicating the user's preference and possible response in future interactions, build a deep learning framework (such as using TensorFlow or PyTorch), realize the iterative update capability of the model, combine the user interaction tendency prediction data with the scene semantic recognition optimization data to form a new training set, input the integrated data into the deep learning model, train through back propagation and optimization algorithms (such as Adam, SGD), optimize the recognition model, and after multiple rounds of iterative training, generate an audio prediction recognition model that can accurately recognize user voice, evaluate the performance of the audio prediction recognition model on the test set to ensure its effectiveness in actual scenarios, deploy the trained model to actual applications, and complete the speech recognition task.

[0147] In this embodiment, a speech recognition system of an intelligent speech robot is provided, which is used to execute the speech recognition method of the intelligent speech robot as described above, including:

[0148] The audio optimization module collects the user's real-time conversation audio and the user's conversation monitoring image; performs noise reduction and separation processing on the user's real-time conversation audio, and performs audio distortion optimization to obtain distortion-optimized conversation audio;

[0149] The emotional state module recognizes micro-expressions in user conversation monitoring images and mines emotional state features to generate real-time emotional state features of users;

[0150] The key vocabulary module recognizes user semantic pauses in the distortion-optimized conversation audio and performs key vocabulary feature analysis to obtain the key fields of each frequency domain unit;

[0151] The context association module performs multi-scale convolution learning on the key fields of each frequency domain unit, and then performs audio context association mining to generate a user audio context association map;

[0152] The audio semantic recognition module performs dynamic audio semantic recognition on the distortion-optimized conversation audio based on the user audio context association graph and the user's real-time emotional state characteristics, and performs semantic optimization of the current scene, thereby generating scene semantic recognition optimization data;

[0153] The interaction tendency prediction module predicts the interaction tendency of the scene semantic recognition optimization data, performs deep iterative learning, and builds a user audio prediction and recognition model to complete the speech recognition task.

[0154] The present invention helps to improve the quality of user conversation audio, reduce interference and noise, and improve the accuracy of speech recognition through noise reduction separation processing and audio distortion optimization. Micro-expression recognition and emotional state feature mining can better understand the user's emotional state, which helps the intelligent system to respond to user emotions more intelligently. User semantic pause recognition and key vocabulary feature analysis help capture key information in the conversation and improve the accuracy and comprehension of speech recognition. Multi-scale convolution learning and audio context association mining help to build a user audio context association map and provide richer background information for speech understanding. Dynamic audio semantic recognition and current scene semantic optimization can improve the accuracy and personalization level of speech recognition according to the user's emotional state and context information. Interaction tendency prediction and user audio prediction recognition model construction can achieve more intelligent speech recognition according to the user's interaction behavior pattern and semantic tendency, and provide services that are closer to user needs.

[0155] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.

[0156] The above is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A speech recognition method for an intelligent speech robot, characterized in that: The following steps are involved: Step S1: collecting user real-time conversation audio and user conversation monitoring images; performing noise reduction and separation processing on the user real-time conversation audio, and performing audio distortion optimization to obtain distortion-optimized conversation audio; Step S2: performing micro-expression recognition on the user dialogue monitoring image and performing emotional state feature mining to generate the user's real-time emotional state feature; Step S3: performing user semantic pause recognition on the distortion-optimized conversation audio and performing key word feature analysis to obtain key fields of each frequency domain unit; Step S4: multi-scale convolution learning is performed on the key fields of each frequency domain unit, and then audio context association mining is performed to generate a user audio context association map; Step S5: performing dynamic audio semantic recognition on the distortion-optimized conversation audio based on the user audio context association graph and the user's real-time emotional state characteristics, and performing current scene semantic optimization, thereby generating scene semantic recognition optimization data; Step S6: Predict the interaction tendency of the scene semantic recognition optimization data, perform deep iterative learning, and build a user audio prediction and recognition model to complete the speech recognition task.

2. The speech recognition method of the intelligent speech robot according to claim 1, characterized in that: The specific steps of step S1 are: Step S11: collecting user real-time conversation audio and user conversation monitoring images; Step S12: Perform environmental sound difference recognition on the user's real-time conversation audio, and extract the real-time conversation audio and the real-time environmental audio; Step S13: performing multi-frequency decomposition on the real-time ambient audio, thereby obtaining an ambient sound frequency domain of multiple frequencies; Step S14: performing frequency-by-frequency noise component analysis on the ambient sound frequency domain of multiple frequencies to generate a noise component for each frequency; Step S15: performing noise reduction and separation processing on the real-time conversation audio according to the noise component of each frequency, thereby obtaining noise reduction and separation conversation audio; Step S16: performing audio distortion optimization on the noise reduction and separation conversation audio, thereby obtaining distortion optimized conversation audio.

3. The speech recognition method of the intelligent speech robot according to claim 2, characterized in that: The specific steps of step S16 are: Define the timing frame length; Decomposing the noise reduction and separation dialogue audio frame by frame according to the time sequence frame length, thereby obtaining a plurality of frame segment dialogue audios; Fit the conversation timeline to the real-time conversation audio to obtain the user's real-time conversation timeline; Using the user's real-time conversation timeline, compare the delay differences of multiple frame segments of conversation audio frame by frame, and extract the audio delay frame segments; Performing delay calculation on the audio delay frame segment to obtain a delay parameter of the segment; Performing audio delay compensation calculation on the segment delay parameter to generate an audio delay compensation value; Performing time domain interpolation compensation on each frame segment dialogue audio based on the audio delay compensation value, thereby obtaining time delay compensated audio; Correct the time-frequency distortion of the delay-compensated audio to construct distortion-optimized dialogue audio.

4. The speech recognition method of the intelligent speech robot according to claim 1, characterized in that: The specific steps of step S2 are: Step S21: performing micro-expression recognition on the user dialogue monitoring image to obtain the user's real-time micro-expression data; Step S22: performing face and mouth recognition on the user conversation monitoring image to extract the user's mouth area image; Step S23: Analyze the mouth shape change of the user's mouth area image to generate the user's mouth shape change data; Step S24: performing time-series pitch evolution on the real-time conversation audio to obtain a user conversation pitch curve; Step S25: mining the emotional state features of the user's real-time micro-expression data, the user's lip shape change data, and the user's conversation tone curve, thereby generating the user's real-time emotional state features.

5. The speech recognition method of the intelligent speech robot according to claim 1, characterized in that: The specific steps of step S3 are: Step S31: performing short-time Fourier transform processing on the distortion-optimized dialogue audio to obtain audio frequency domain data; Step S32: performing user semantic pause recognition on the audio frequency domain data to extract the user conversation pause points; Step S33: performing time-frequency decomposition according to the pause points of the user's conversation, and extracting multiple audio frequency domain units; Step S34: performing key word feature analysis on multiple audio frequency domain units to obtain a key field of each frequency domain unit.

6. The speech recognition method of the intelligent speech robot according to claim 1, characterized in that: The specific steps of step S4 are: Step S41: performing convolution feature coding on the key field of each frequency domain unit, thereby obtaining a plurality of key field convolution codes; Step S42: performing multi-scale convolution learning on multiple key field convolution codes to extract multiple convolution feature vectors; Step S43: performing time series fitting on multiple convolution feature vectors to obtain a convolution feature vector sequence; Step S44: extracting the timestamp of each key field based on the key field of each frequency domain unit; Step S45: performing inter-field local dependency analysis on the convolution feature vector sequence according to the timestamp of each key field to extract the local dependency relationship of the convolution features; Step S46: Perform audio context association mining on the local dependency of the convolutional features to generate a user audio context association graph.

7. The speech recognition method of the intelligent speech robot according to claim 1, characterized in that: The specific steps of step S5 are: Step S51: performing user interaction scene clustering analysis on the user's real-time emotional state characteristics according to the real-time ambient audio to obtain the current user interaction scene type; Step S52: performing dynamic audio semantic recognition on the distortion-optimized conversation audio based on the user audio context association graph, thereby obtaining dynamic audio recognition data; Step S53: Perform current scene semantic optimization on the dynamic audio recognition data based on the user interaction scene type, thereby generating scene semantic recognition optimization data.

8. The speech recognition method of the intelligent speech robot according to claim 1, characterized in that: The specific steps of step S6 are: Step S61: performing interactive behavior evolution on the distortion-optimized conversation audio to generate real-time interactive behavior data of the user; Step S62: Performing personalized behavior habit analysis on the user's real-time interaction behavior data, thereby obtaining the current user's interaction behavior habit characteristics; Step S63: predicting the interaction tendency of the current user's interaction behavior habit characteristics to obtain user interaction tendency prediction data; Step S64: Use the user interaction tendency prediction data to perform deep iterative learning on the scene semantic recognition optimization data to build a user audio prediction recognition model to complete the speech recognition task.

9. A speech recognition system for an intelligent speech robot, characterized in that: The method for performing speech recognition of the intelligent speech robot according to claim 1 comprises: The audio optimization module collects the user's real-time conversation audio and the user's conversation monitoring image; performs noise reduction and separation processing on the user's real-time conversation audio, and performs audio distortion optimization to obtain distortion-optimized conversation audio; The emotional state module recognizes micro-expressions in user conversation monitoring images and mines emotional state features to generate real-time emotional state features of users; The key vocabulary module recognizes user semantic pauses in the distortion-optimized conversation audio and performs key vocabulary feature analysis to obtain the key fields of each frequency domain unit; The context association module performs multi-scale convolution learning on the key fields of each frequency domain unit, and then performs audio context association mining to generate a user audio context association map; The audio semantic recognition module performs dynamic audio semantic recognition on the distortion-optimized conversation audio based on the user audio context association graph and the user's real-time emotional state characteristics, and performs semantic optimization of the current scene, thereby generating scene semantic recognition optimization data; The interaction tendency prediction module predicts the interaction tendency of the scene semantic recognition optimization data, performs deep iterative learning, and builds a user audio prediction and recognition model to complete the speech recognition task.

Images