Digital interaction enhancement system based on multi-mode voice

Abstract

The invention relates to the technical field of digital interaction, in particular to a digital interaction enhancement system based on multi-modal voice, which comprises a multi-modal sensing module comprising a voice sensing unit, a visual sensing unit and an environment sensing unit and used for capturing multi-modal data input by a user; the multi-modal fusion module comprises a data synchronization unit, a feature extraction unit and a multi-modal fusion unit and is used for performing time synchronization, feature extraction and fusion processing on the multi-modal data; and the interactive understanding module comprises a natural language processing unit, a context understanding unit and an emotion analysis unit and is used for understanding the intention and the emotion state of the user. The deep learning technology is adopted to fuse voice, vision and environment data, and efficient and natural user interaction experience is achieved through the steps of feature extraction, feature fusion, intention recognition and the like. The system has excellent performance and a wide application prospect in a complex environment.

Description

Technical Field

[0001] The present invention relates to the field of digital interaction technology, and in particular to a digital interaction enhancement system based on multimodal speech. Background Art

[0002] Existing voice interaction systems mainly rely on single-mode speech recognition, which has problems such as low recognition accuracy, poor context understanding, and unnatural interaction. These systems are often unable to cope with complex tasks and changing environments. Multimodal interaction technology can provide a more natural and efficient interaction experience by combining multiple perception methods such as voice, vision, and gestures. However, there is currently a lack of a system on the market that can integrate and effectively process information from multiple modalities.

[0003] Therefore, a digital interaction enhancement system based on multimodal speech is proposed. Summary of the invention

[0004] The purpose of the present invention is to provide a digital interaction enhancement system based on multimodal voice, which is different from the prior art and can combine multimodal information such as voice, vision and gesture to provide a more accurate and natural interaction experience to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a digital interactive enhancement system based on multimodal speech, comprising:

[0006] A multimodal perception module, including a speech perception unit, a visual perception unit, and an environmental perception unit, for capturing multimodal data input by a user;

[0007] The multimodal fusion module includes a data synchronization unit, a feature extraction unit and a multimodal fusion unit, and is used to perform time synchronization, feature extraction and fusion processing on multimodal data;

[0008] The interaction understanding module includes a natural language processing unit, a context understanding unit, and a sentiment analysis unit, which is used to understand user intentions and emotional states;

[0009] A response generation module, including a speech synthesis unit and an action generation unit, for generating natural speech responses and corresponding actions;

[0010] The feedback adjustment module includes a user feedback collection unit and an adaptive learning unit, which are used to collect user feedback and adjust system parameters.

[0011] Preferably, the multimodal perception module:

[0012] Speech perception unit: captures user's voice input;

[0013] Visual perception unit: captures the user's facial expressions and gestures through the camera;

[0014] Environmental perception unit: monitors external conditions of ambient noise and light;

[0015] The preprocessing unit performs noise reduction and normalization operations on the data.

[0016] Preferably, the multimodal fusion module:

[0017] Data synchronization unit: time synchronization of data of different modes;

[0018] Feature extraction unit: extracts features from speech, vision and environmental data;

[0019] Multimodal fusion unit: uses deep learning algorithms to fuse multimodal features to form unified interactive instructions;

[0020] The feature extraction unit extracts features such as speech features, facial expression features, and gesture features from speech, vision, and environmental data.

[0021] Preferably, the interactive understanding module:

[0022] Natural language processing unit: performs semantic analysis on the fused voice commands;

[0023] Context understanding unit: understands user intent based on context information;

[0024] Sentiment Analysis Unit: Analyzes the user's emotional state to adjust the system's response.

[0025] Preferably, the response generation module:

[0026] Speech synthesis unit: generates natural, fluent speech responses;

[0027] Action generation unit: Generates corresponding visual or physical actions according to user needs.

[0028] Preferably, the feedback adjustment module:

[0029] User feedback collection unit: collects user feedback;

[0030] Adaptive learning unit: adjusts system parameters based on user feedback to improve system performance.

[0031] Preferably, the multimodal fusion module uses a deep learning algorithm to perform fusion processing on multimodal features.

[0032] Preferably, the feedback adjustment module collects user usage feedback through a user feedback collection unit, and adjusts system parameters through an adaptive learning unit.

[0033] Preferably, in multimodal fusion, time synchronization is the key to ensure effective integration of data from different modalities by using sliding window technology to perform data synchronization processing.

[0034] Preferably, timestamp recording: adding a timestamp to the data collected by each perception unit (speech, vision, environment);

[0035] Sliding window: Set a time window and treat the data within the window as synchronous data;

[0036] Window adjustment: Adjust the window size according to the sampling rate of different modal data to ensure that all modal data are aligned in the same time range.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] This paper proposes a digital interaction enhancement system based on multimodal speech, which uses deep learning technology to integrate speech, vision and environmental data, and achieves efficient and natural user interaction experience through steps such as feature extraction, feature fusion and intention recognition. The system has excellent performance and broad application prospects in complex environments.

[0039] The present invention provides a digital interaction enhancement system based on multimodal speech, which achieves higher recognition accuracy and more natural interaction experience by combining multimodal information such as speech, vision and gesture, and solves many deficiencies in the prior art. The system can work stably in complex environments and is continuously optimized through user feedback, and has broad application prospects.

[0040] High recognition accuracy: Through multimodal information fusion, the accuracy of speech recognition and intent understanding is improved.

[0041] Natural interaction experience: Combine multiple modes such as voice, vision and gesture to provide a more natural interaction experience.

[0042] Strong environmental adaptability: The environmental perception unit monitors external conditions in real time, and the system can work stably in complex environments.

[0043] Intelligent response generation: Generate personalized responses based on the user's emotional state to improve user satisfaction.

[0044] Continuous optimization: The adaptive learning unit continuously adjusts the system based on user feedback to improve interaction efficiency and user experience BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A block diagram of a digital interactive enhancement system based on multimodal speech according to the present invention;

[0047] Figure 2 This is a block diagram of the multimodal perception module of the present invention;

[0048] Figure 3 It is a block diagram of the multimodal fusion module of the present invention;

[0049] Figure 4 It is a block diagram of the interactive understanding module of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] See also Figures 1 to 4 , the present invention provides a technical solution:

[0052] The digital interactive enhancement system based on multimodal speech includes:

[0053] A multimodal perception module, including a speech perception unit, a visual perception unit, and an environmental perception unit, for capturing multimodal data input by a user;

[0054] The multimodal fusion module includes a data synchronization unit, a feature extraction unit and a multimodal fusion unit, and is used to perform time synchronization, feature extraction and fusion processing on multimodal data;

[0055] The interaction understanding module includes a natural language processing unit, a context understanding unit, and a sentiment analysis unit, which is used to understand user intentions and emotional states;

[0056] A response generation module, including a speech synthesis unit and an action generation unit, for generating natural speech responses and corresponding actions;

[0057] The feedback adjustment module includes a user feedback collection unit and an adaptive learning unit, which are used to collect user feedback and adjust system parameters.

[0058] The multimodal perception module:

[0059] Speech perception unit: captures user's voice input;

[0060] Visual perception unit: captures the user's facial expressions and gestures through the camera;

[0061] Environmental perception unit: monitors external conditions of ambient noise and light;

[0062] The preprocessing unit performs noise reduction and normalization operations on the data.

[0063] Multimodal fusion module:

[0064] Data synchronization unit: time synchronization of data of different modes;

[0065] Feature extraction unit: extracts features from speech, vision and environmental data;

[0066] Multimodal fusion unit: uses deep learning algorithms to fuse multimodal features to form unified interactive instructions;

[0067] The feature extraction unit extracts features such as speech features, facial expression features, and gesture features from speech, vision, and environmental data.

[0068] The interactive understanding module:

[0069] Natural language processing unit: performs semantic analysis on the fused voice commands;

[0070] Context understanding unit: understands user intent based on context information;

[0071] Sentiment Analysis Unit: Analyzes the user's emotional state to adjust the system's response.

[0072] Preferably, the response generation module:

[0073] Speech synthesis unit: generates natural, fluent speech responses;

[0074] Action generation unit: Generates corresponding visual or physical actions according to user needs.

[0075] The feedback adjustment module:

[0076] User feedback collection unit: collects user feedback;

[0077] Adaptive learning unit: adjusts system parameters based on user feedback to improve system performance.

[0078] The multimodal fusion module uses a deep learning algorithm to perform fusion processing on multimodal features.

[0079] The feedback adjustment module collects user usage feedback through a user feedback collection unit, and adjusts system parameters through an adaptive learning unit.

[0080] In multimodal fusion, time synchronization is the key to ensure the effective integration of data from different modalities by using sliding window technology to perform data synchronization processing.

[0081] Timestamp recording: add timestamps to the data collected by each perception unit (speech, vision, environment);

[0082] Sliding window: Set a time window and treat the data within the window as synchronous data;

[0083] Window adjustment: adjust the window size according to the sampling rate of different modal data to ensure that all modal data are aligned within the same time range;

[0084] Feature extraction: Extract information that can represent the characteristics of the modality from data of different modalities.

[0085] Speech feature extraction

[0086] 1. Preprocessing: remove background noise and silence.

[0087] 2. Feature extraction: Use MFCC (Mel-frequency cepstral coefficient) to extract speech features.

[0088] 3. Enhanced features: adding emotional features such as tone, rhythm, etc.

[0089] Visual feature extraction

[0090] 1. Preprocessing: image denoising and lighting balance.

[0091] 2. Feature extraction: Use convolutional neural network (CNN) to extract facial expression and gesture features.

[0092] 3. Enhanced features: Detect and encode facial key points and gesture trajectories.

[0093] Environmental feature extraction

[0094] 1. Preprocessing: Filter out environmental sensor noise.

[0095] 2. Feature extraction: Extract features such as ambient noise level and light intensity.

[0096] 3. Enhanced features: Analyze the dynamic change trends of the environment.

[0097] Feature Fusion

[0098] The features extracted from each modality are fused to form a unified feature representation.

[0099] Fusion Steps

[0100] 1. Feature normalization: Normalize all features to ensure that the scales of features of different modalities are consistent.

[0101] 2. Feature connection: concatenate the normalized feature vectors to form a high-dimensional feature vector.

[0102] 3. Feature dimensionality reduction: Use PCA (principal component analysis) or deep autoencoders to reduce the dimensionality of high-dimensional feature vectors to reduce redundant information and improve computational efficiency.

[0103] Converged Network

[0104] The fused features are processed using a deep neural network (DNN) to learn the relationship between multimodal features.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A digital interactive enhancement system based on multimodal speech, characterized by: include: A multimodal perception module, including a speech perception unit, a visual perception unit, and an environmental perception unit, for capturing multimodal data input by a user; The multimodal fusion module includes a data synchronization unit, a feature extraction unit and a multimodal fusion unit, and is used to perform time synchronization, feature extraction and fusion processing on multimodal data; The interaction understanding module includes a natural language processing unit, a context understanding unit, and a sentiment analysis unit, which is used to understand user intentions and emotional states; A response generation module, including a speech synthesis unit and an action generation unit, for generating natural speech responses and corresponding actions; The feedback adjustment module includes a user feedback collection unit and an adaptive learning unit, which are used to collect user feedback and adjust system parameters.

2. The digital interactive enhancement system based on multimodal speech according to claim 1, characterized in that: The multimodal perception module: Speech perception unit: captures user's voice input; Visual perception unit: captures the user's facial expressions and gestures through the camera; Environmental perception unit: monitors external conditions of ambient noise and light; The preprocessing unit performs noise reduction and normalization operations on the data.

3. The digital interactive enhancement system based on multimodal speech according to claim 1, characterized in that: Multimodal fusion module: Data synchronization unit: time synchronization of data of different modes; Feature extraction unit: extracts features from speech, vision and environmental data; Multimodal fusion unit: uses deep learning algorithms to fuse multimodal features to form unified interactive instructions; The feature extraction unit extracts features such as speech features, facial expression features, and gesture features from speech, vision, and environmental data.

4. The digital interactive enhancement system based on multimodal speech according to claim 1, characterized in that: The interactive understanding module: Natural language processing unit: performs semantic analysis on the fused voice commands; Context understanding unit: understands user intent based on context information; Sentiment Analysis Unit: Analyzes the user's emotional state to adjust the system's response.

5. The multimodal speech-based digital interaction enhancement system according to claim 1, characterized in that: The response generation module: Speech synthesis unit: generates natural, fluent speech responses; Action generation unit: Generates corresponding visual or physical actions according to user needs.

6. The digital interactive enhancement system based on multimodal speech according to claim 1, characterized in that: The feedback adjustment module: User feedback collection unit: collects user feedback; Adaptive learning unit: adjusts system parameters based on user feedback to improve system performance.

7. The digital interactive enhancement system based on multimodal speech according to claim 1, characterized in that: The multimodal fusion module uses a deep learning algorithm to perform fusion processing on multimodal features.

8. The multimodal speech-based digital interaction enhancement system according to claim 7, characterized in that: The feedback adjustment module collects user usage feedback through a user feedback collection unit, and adjusts system parameters through an adaptive learning unit.

9. The multimodal speech-based digital interaction enhancement system according to claim 8, characterized in that: In multimodal fusion, time synchronization is the key to ensure the effective integration of data from different modalities by using sliding window technology to perform data synchronization processing.

10. The multimodal speech-based digital interaction enhancement system according to claim 9, characterized in that: Timestamp recording: add timestamps to the data collected by each perception unit (speech, vision, environment); Sliding window: Set a time window and treat the data within the window as synchronous data; Window adjustment: Adjust the window size according to the sampling rate of different modal data to ensure that all modal data are aligned in the same time range.

Images