Intelligent voice interaction system and method based on multi-mode perception

Abstract

The invention discloses an intelligent voice interaction system and method based on multi-mode perception, and belongs to the technical field of man-machine interaction. The system comprises a voice acquisition and preprocessing module, a multi-mode perception module, a voice recognition, voice understanding and decision module and a multi-mode feedback module. Wherein the multi-mode sensing module fuses voice, vision and sensor data through a space-time alignment algorithm to realize comprehensive sensing of user intentions. According to the invention, through integration of information of a plurality of perception channels, the recognition accuracy and the understanding ability of the voice interaction system in a complex environment are improved, and the problem that traditional single voice interaction is difficult to recognize in a noise environment, a multi-person scene and a complex context is solved; and more natural and accurate man-machine interaction experience can be provided for the fields of intelligent home, intelligent vehicle-mounted, intelligent customer service and the like.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent voice interaction systems, and specifically relates to an intelligent voice interaction system and method based on multimodal perception. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, intelligent voice interaction systems have been widely used in a variety of fields, including smart homes, smart cars, and smart customer service. Traditional intelligent voice interaction systems mainly use technologies such as speech recognition, semantic understanding, and speech synthesis to achieve human-computer dialogue, providing users with a convenient interactive experience.

[0003] Currently, intelligent voice interaction systems typically include core components such as a sound collection module, a preprocessing module, and an intelligent interaction center. For example, CN107919126A discloses an intelligent voice interaction system comprising a sound collection module, a PFGA preprocessing module, and an intelligent interaction center. The preprocessing module includes an endpoint detection unit, while the intelligent interaction center includes a control unit, a speech recognition module, a semantic understanding module, and an interaction processing module. The system provides feedback on recognition information through a feedback module and displays it to the user through the control unit. The system also controls and modifies the sound matching model and language model library to improve the recognition accuracy of the interactive system.

[0004] With technological advancements, multimodal interaction technology has gradually become a development trend in intelligent interactive systems. CN119673154A discloses a digital interaction enhancement system based on multimodal speech. The system includes a multimodal perception module, a multimodal fusion module, and an interaction understanding module. The multimodal perception module includes a speech perception unit, a visual perception unit, and an environmental perception unit for capturing multimodal data input by the user. CN118016063A discloses a multimodal interaction system for service robots, including a data acquisition module, a scene configuration module, an algorithm configuration module, and a feedback coordination module. The system integrates the various capabilities of the robot through a fusion interaction engine system to achieve coordination between the various modules.

[0005] Furthermore, CN118426593B discloses a method and system for virtual digital human interaction, including a text recognition module, a speech recognition module, a decision-making module, an execution module, a human-computer interaction module, an emotion analysis module, a multimodal perception unit, a visual recognition module, and a visual assistance module. By collecting the user's facial expression features and feeding the extracted facial expression features back to the decision-making layer, the authenticity of the emotional output of the human-computer interaction module is improved. CN118226967B discloses a multimodal interactive intelligent control system, including a speech recognition module, a touch sensing module, a gesture recognition module, an environmental perception module, an interaction feedback module, and an intelligent modal coordination module. This intelligently selects and switches interaction modes based on user interaction history and real-time user behavior data.

[0006] However, existing intelligent voice interaction systems still have the following technical problems: First, environmental adaptability is poor. In complex environments, such as noisy ones or in far-field speech recognition scenarios, existing systems experience a significant drop in recognition accuracy, making it difficult to meet user needs in these complex environments. Traditional voice interaction systems primarily rely on single-source voice signal processing and lack the ability to effectively perceive and adapt to environmental factors.

[0007] Secondly, the problem of single-mode interaction is prominent. Most existing systems rely solely on voice input and lack the ability to integrate and process multimodal information such as user expressions, gestures, and the environment. Although some systems have begun to experiment with multimodal interaction, technical bottlenecks remain in the spatiotemporal alignment and fusion of multimodal data, making it difficult to achieve a truly natural and smooth human-computer interaction experience.

[0008] Finally, contextual understanding is insufficient. Existing systems have significant deficiencies in multi-round dialogue and contextual understanding. In particular, they struggle to accurately grasp user intent in complex contexts, leading to disjointed interactions and even irrelevant answers, severely impacting user experience.

[0009] Therefore, there is an urgent need for an intelligent voice interaction system that can effectively integrate multimodal perception information, improve environmental adaptability, and enhance context understanding capabilities to solve the problems existing in existing technologies. Summary of the Invention

[0010] The technical problem to be solved by the present invention is to provide an intelligent voice interaction system and method based on multimodal perception to achieve a more natural and accurate human-computer interaction experience, in response to the defects of existing intelligent voice interaction systems such as low recognition accuracy in complex environments, single interaction mode and insufficient context understanding ability.

[0011] The present invention provides an intelligent voice interaction system based on multimodal perception, which includes: a voice acquisition and preprocessing module for acquiring user voice signals and performing noise reduction, echo cancellation, and signal enhancement processing; a multimodal perception module for acquiring and fusing voice, visual, and environmental perception data, the multimodal perception module including a visual perception unit, an environmental perception unit, and a user state perception unit; a voice recognition module for converting the preprocessed voice signals into text information; a voice understanding and decision-making module for performing semantic analysis and intent recognition on text information in combination with multimodal perception information to generate a response strategy; and a multimodal feedback module for providing feedback of interaction results to the user via voice, vision, or touch.

[0012] The voice collection and preprocessing module includes: 1) Microphone array, used for collecting sound signals within 360 degrees or directional range; 2) Digital signal processor, used to pre-process the speech signal using adaptive filtering algorithms, beamforming technology and sound source localization algorithms.

[0013] The multimodal perception module fuses voice, vision and sensor data through a spatiotemporal alignment algorithm based on timestamp synchronization or deep learning feature matching mechanism, with time accuracy reaching millisecond or microsecond level.

[0014] The speech recognition module adopts a deep learning model based on the Transformer or Conformer structure, supports real-time speech transcription, and has a recognition accuracy rate of no less than 92% in a noisy environment.

[0015] The speech understanding and decision-making module includes: 1) Knowledge graph, used for semantic analysis and context understanding; 2) Multi-level classification models or hierarchical attention mechanisms for intent recognition; 3) Decision engine, which generates response strategies based on rule systems or reinforcement learning algorithms.

[0016] The multimodal feedback module includes: 1) Speech synthesis unit, supporting multiple timbres and emotional expressions; 2) Visual feedback unit, including display screen, projector or holographic display technology; 3) Tactile feedback unit, including a vibration motor or an ultrasonic tactile array.

[0017] The system connects the modules via a high-speed data bus or optical fiber, with an end-to-end interaction delay of less than 500 milliseconds.

[0018] An intelligent voice interaction method based on multimodal perception includes the following steps: 1) The voice acquisition and preprocessing module collects user voice signals and performs noise reduction and enhancement processing; 2) Synchronously collect visual, environmental, and user status data through a multimodal perception module and fuse them using a spatiotemporal alignment algorithm; 3) Converting speech signals into text information through the speech recognition module; 4) The speech understanding and decision-making module combines multimodal data to perform semantic analysis and intent recognition on text information and generate response strategies; 5) Feedback the interaction results to the user through the multimodal feedback module.

[0019] The spatiotemporal alignment algorithm includes timestamp-based data synchronization or deep learning-based multimodal feature matching.

[0020] The speech understanding and decision-making module introduces knowledge graph and context understanding technology to resolve semantic ambiguity and maintain coherence in multi-round dialogues.

[0021] The beneficial effects of the present invention are as follows: by integrating a high-sensitivity microphone array and an advanced preprocessing algorithm, the quality of the voice signal is significantly improved, and environmental noise and echo interference are effectively suppressed, enabling the system to maintain a high voice recognition accuracy in noisy environments or far-field conditions; by fusing voice, vision, and environmental information through a multimodal perception module, the limitation of traditional voice interaction systems that rely solely on single voice input is broken through, and the perception and understanding of multi-dimensional information such as user expressions and gestures is realized; the multimodal feature fusion method based on spatiotemporal alignment technology and attention mechanism effectively solves the spatiotemporal synchronization problem of voice, vision, and sensor data, and realizes the effective fusion of multimodal data; by introducing knowledge graphs and context understanding technology, the system can better understand the context information in multiple rounds of dialogue, solve the problem of semantic ambiguity, and make the interaction process more coherent and natural; the multimodal feedback mechanism provides multi-dimensional feedback such as voice, vision, and action, realizing a more comprehensive and natural interaction experience. Compared with the existing technology, the present invention significantly improves the naturalness and accuracy of interaction in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a system architecture diagram of the present invention; Figure 2 It is a flow chart of the multimodal spatiotemporal alignment algorithm of the present invention. DETAILED DESCRIPTION

[0023] The technical solutions of the present invention will be described clearly and completely below with reference to the accompanying drawings and embodiments. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are also within the scope of protection of the present invention.

[0024] Example 1 An intelligent voice interaction system based on multimodal perception includes a voice acquisition and preprocessing module, a multimodal perception module, a voice recognition module, a voice understanding and decision module, and a multimodal feedback module.

[0025] The voice acquisition and preprocessing module consists of a microphone array and a digital signal processor. The microphone array includes eight omnidirectional microphones evenly distributed in a circular pattern with a diameter of 10 cm, which are used to collect sound signals within a 360-degree range. The digital signal processor uses a 32-bit floating-point processor with a main frequency of 1.2 GHz and built-in 256MB of RAM. It is used to perform noise reduction, echo cancellation, and signal enhancement on the collected raw voice signals. Noise reduction uses an adaptive filtering algorithm to effectively suppress ambient noise and improve the signal-to-noise ratio by at least 15 dB. Echo cancellation uses a nonlinear adaptive algorithm, achieving an echo suppression ratio of up to 40 dB. Signal enhancement uses beamforming technology to achieve directional enhancement of the target sound source, with a gain of up to 6 dB.

[0026] The multimodal perception module includes a visual perception unit, an environmental perception unit, and a user state perception unit. The visual perception unit is equipped with a binocular camera with a resolution of 1920×1080 pixels, a frame rate of 60fps, and a field of view of 120 degrees, capable of capturing the user's facial expressions, body movements, and surrounding environmental information. The environmental perception unit integrates temperature and humidity sensors, light sensors, and gas sensors for real-time monitoring of environmental parameters. The user state perception unit uses bioelectric sensors and pressure sensors to detect the user's emotional state and body posture. The multimodal perception module integrates voice, visual, and sensor data through a spatiotemporal alignment algorithm to achieve collaborative processing of multi-dimensional information. The spatiotemporal alignment algorithm utilizes a timestamp-based data synchronization mechanism, achieving millisecond-level temporal accuracy and centimeter-level spatial alignment accuracy, ensuring the consistency and integrity of data from different modalities.

[0027] The speech recognition module utilizes a deep neural network architecture, including both acoustic and language models. The acoustic model is based on a Transformer architecture and comprises a 12-layer encoder, each with 8 attention heads. The hidden layer has a dimension of 768 and consumes approximately 100 million model parameters. The language model utilizes a hybrid N-gram and neural network architecture, with a vocabulary of 100,000 words covering both general and specialized domains. The speech recognition module supports real-time transcription with a latency of less than 200 milliseconds. Recognition accuracy reaches 98% in quiet environments and remains above 92% in noisy environments.

[0028] The speech understanding and decision-making module is responsible for semantic analysis and intent identification based on recognition results. This semantic analysis, based on knowledge graphs and natural language processing technologies, constructs a domain knowledge base containing 500,000 entities and 3 million relationships. Intent identification utilizes a multi-level classification model, supporting 200 common intent categories and 1,000 sub-intents. The decision-making engine, based on a rule-based system and reinforcement learning algorithms, generates the optimal response strategy based on the current conversation state, historical user interactions, and environmental information. The decision-making module's response time is less than 100 milliseconds, and its decision accuracy reaches 95%.

[0029] The multimodal feedback module includes a speech synthesis unit, a visual feedback unit, and a tactile feedback unit. The speech synthesis unit uses an end-to-end neural network model, supports a variety of timbres and emotional expressions, and the naturalness score of the synthesized speech reaches 4.5 points (out of a maximum of 5 points). The visual feedback unit includes a high-definition display and an LED indicator array. The display has a resolution of 2560×1440 pixels and a color accuracy of 100% sRGB color gamut, which can display text, images, and video content. The LED indicator array consists of 64 full-color LEDs and can express system status and emotions through different colors and brightness modes. The tactile feedback unit consists of a vibration motor and a pressure sensor, which can provide multiple tactile feedback modes. The vibration frequency range is 40-500Hz and the intensity is adjustable.

[0030] Each system module is connected via a high-speed data bus, achieving a data transmission rate of 10Gbps, ensuring real-time information flow. The entire system adopts a modular design, allowing each functional unit to be flexibly configured according to the application scenario. The system housing is made of engineering plastic, measures 200×150×80 mm, weighs 1.2 kg, and has an IP65 protection rating, making it suitable for various indoor and outdoor environments.

[0031] The system operates as follows: When a user issues a voice command, the Voice Acquisition and Preprocessing Module first captures the sound signal and performs preliminary processing. Simultaneously, the Multimodal Perception Module collects the user's facial expressions, body movements, and environmental parameters. The Voice Recognition Module converts the processed voice into text. The Voice Understanding and Decision Module combines multimodal information to analyze user intent and generate a response strategy. Finally, the Multimodal Feedback Module provides feedback to the user through voice, vision, and touch. The entire process has an end-to-end latency of less than 500 milliseconds, ensuring smooth and natural interaction.

[0032] Example 2 An intelligent voice interaction system based on multimodal perception includes a voice acquisition and preprocessing module, a multimodal perception module, a voice recognition module, a voice understanding and decision module, and a multimodal feedback module.

[0033] The voice acquisition and preprocessing module consists of a microphone array and a digital signal processor. The microphone array includes six directional microphones arranged linearly and measuring 15 cm in length, which are used to collect sound signals within a 180-degree range in front of the camera. The digital signal processor uses a 64-bit floating-point processor with a main frequency of 1.5 GHz and built-in 512 MB of RAM. It is used to perform noise reduction, echo cancellation, and signal enhancement on the collected raw voice signals. Noise reduction uses a deep learning noise suppression algorithm to effectively suppress ambient noise and improve the signal-to-noise ratio by at least 18 dB. Echo cancellation uses a frequency-domain adaptive algorithm, achieving an echo suppression ratio of up to 45 dB. Signal enhancement uses adaptive beamforming technology to achieve dynamic tracking and enhancement of the target sound source, with a gain of up to 8 dB.

[0034] The multimodal perception module consists of a visual perception unit, an environmental perception unit, and a user status perception unit. The visual perception unit is equipped with an RGB-D camera with an RGB resolution of 2560×1440 pixels, a depth resolution of 640×480 pixels, a frame rate of 30fps, and a field of view of 110 degrees. It can capture three-dimensional information about the user's facial expressions, body movements, and surrounding environment. The environmental perception unit integrates temperature and humidity sensors, light sensors, gas sensors, and an acoustic environment analyzer to monitor environmental parameters in real time and analyze acoustic characteristics. The user status perception unit uses infrared thermal imaging sensors and millimeter-wave radar for contactless detection of the user's emotional state and body posture. The multimodal perception module integrates voice, visual, and sensor data through a spatiotemporal alignment algorithm to achieve collaborative processing of multi-dimensional information. This spatiotemporal alignment algorithm utilizes a multimodal feature matching mechanism based on deep learning, achieving microsecond-level temporal accuracy and millimeter-level spatial alignment accuracy, ensuring the consistency and integrity of data from different modalities.

[0035] The speech recognition module utilizes a hybrid neural network architecture, combining the strengths of convolutional neural networks and recurrent neural networks. The acoustic model, based on the Conformer architecture, comprises a 16-layer encoder, each with 12 attention heads. The hidden layer dimension is 1024, resulting in approximately 200 million model parameters. The language model utilizes BERT pre-training and domain adaptation technology, with a vocabulary of 150,000 words covering general domains and multiple specialized fields. The speech recognition module supports real-time transcription with a latency of less than 150 milliseconds. Recognition accuracy reaches 99% in quiet environments and remains above 95% in noisy environments.

[0036] The speech understanding and decision-making module is responsible for semantic analysis and intent identification based on recognition results. This semantic analysis, based on a large-scale pre-trained language model and knowledge graph, constructs a multi-domain knowledge base containing one million entities and five million relationships. Intent identification utilizes a hierarchical attention mechanism, supporting 300 common intent categories and 2,000 sub-intents. The decision-making engine, based on deep reinforcement learning and a multi-agent collaborative system, generates the optimal response strategy based on the current conversation state, historical user interactions, and environmental information. The decision-making module's response time is less than 80 milliseconds, with a decision accuracy rate of 97%.

[0037] The multimodal feedback module includes a speech synthesis unit, a visual feedback unit, and a tactile feedback unit. The speech synthesis unit utilizes a diffusion model-based neural network architecture, supporting 30 different timbres and eight emotional expressions, achieving a naturalness rating of 4.8 out of 5 for the synthesized speech. The visual feedback unit incorporates a micro-projector and holographic display technology, with a projection resolution of 3840×2160 pixels and a brightness of 600 lumens, capable of projecting high-definition content onto any surface. The holographic display unit generates floating three-dimensional images with a viewing angle of 120 degrees. The tactile feedback unit comprises an ultrasonic tactile array, providing tangible tactile feedback in mid-air, with a resolution of 5×5 mm and adjustable force feedback intensity.

[0038] Each system module is connected via a fiber-optic data bus, achieving a data transmission rate of 40Gbps, ensuring real-time information flow. The entire system utilizes a distributed architecture, enabling remote collaboration between functional units via a wireless network. The system's main control unit is constructed from aircraft-grade aluminum alloy, measures 180×120×60 mm, weighs 0.9 kg, and has an IP67 protection rating, making it suitable for a variety of extreme environmental conditions.

[0039] The system works as follows: When the user enters the interactive area, the multimodal perception module first detects the user's presence and establishes a user portrait; when the user issues a voice command, the voice acquisition and preprocessing module captures the sound signal and performs enhanced processing; at the same time, the multimodal perception module collects the user's facial expressions, body movements and environmental parameters in real time; the voice recognition module converts the processed voice into text; the voice understanding and decision-making module combines multimodal information and user portraits to analyze user intentions and generate personalized response strategies; finally, the multimodal feedback module selects the most suitable feedback method to convey information to the user based on environmental conditions and user preferences. The end-to-end delay of the entire process is less than 350 milliseconds, ensuring the smoothness and naturalness of the interaction. Figure 2 As shown in the figure, the multimodal spatiotemporal alignment algorithm process is shown. The process steps are described as follows: Data collection: Collect voice, vision, and environmental data separately to ensure the integrity of the original information of each modality.

[0040] Timestamp: Add precise timestamps to speech, visual, and environmental data to record the time when each data point was generated.

[0041] Time synchronization engine: Through protocol calibration, the time axes of different sensors are unified into a global time axis to solve the time deviation problem.

[0042] Spatial matching module: Converts the local coordinate system of each sensor into a unified world coordinate system to ensure consistent spatial reference.

[0043] Motion Compensation Engine: Dynamically compensates for spatiotemporal deviations and addresses spatiotemporal inconsistencies caused by sensor motion or delays.

[0044] Feature fusion: Generate a spatiotemporal aligned feature matrix, fuse multimodal data through adaptive weight allocation, and output the final fused features.

[0045] This process ensures that multimodal data are precisely aligned in time and space, providing a consistent basis for subsequent analyses.

[0046] Time synchronization engine: achieves microsecond-level alignment across devices through PTP (Precision Time Protocol); Spatial registration module: Uses a checkerboard calibration plate to establish the camera-manipulator coordinate transformation matrix and performs online calibration error compensation; extracts ORB feature point clouds to match visual and LiDAR data, and dynamically updates spatial transformation parameters; Motion compensation engine: reconstructs motion trajectories based on IMU angular velocity data and uses Wiener filters to restore high-frequency details; predicts occluded areas through TOF depth data and generates reasonable completions based on the GAN network.

[0047] It should be noted that both Example 1 and Example 2 are a type of intelligent voice interaction system and method based on multimodal perception.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An intelligent voice interaction system based on multimodal perception, characterized in that: include: Voice acquisition and preprocessing module, used to collect user voice signals and perform noise reduction, echo cancellation and signal enhancement processing; A multimodal perception module, which is used to collect and integrate voice, visual, and environmental perception data. The multimodal perception module includes a visual perception unit, an environmental perception unit, and a user state perception unit. A speech recognition module is used to convert preprocessed speech signals into text information. A speech understanding and decision module is used to perform semantic analysis and intent recognition on text information in combination with multimodal perception information to generate a response strategy. The multimodal feedback module is used to provide feedback of interaction results to users through voice, vision or touch.

2. The intelligent voice interaction system based on multimodal perception according to claim 1, characterized in that: The voice collection and preprocessing module includes: 1) Microphone array, used for collecting sound signals within 360 degrees or directional range; 2) Digital signal processor, used to pre-process the speech signal using adaptive filtering algorithms, beamforming technology and sound source localization algorithms.

3. The intelligent voice interaction system based on multimodal perception according to claim 1, characterized in that: The multimodal perception module fuses voice, vision and sensor data through a spatiotemporal alignment algorithm based on timestamp synchronization or deep learning feature matching mechanism, with time accuracy reaching millisecond or microsecond level.

4. The intelligent voice interaction system based on multimodal perception according to claim 1, characterized in that: The speech recognition module adopts a deep learning model based on the Transformer or Conformer structure, supports real-time speech transcription, and has a recognition accuracy rate of no less than 92% in a noisy environment.

5. The intelligent voice interaction system based on multimodal perception according to claim 1, characterized in that: The speech understanding and decision-making module includes: 1) Knowledge graph, used for semantic analysis and context understanding; 2) Multi-level classification models or hierarchical attention mechanisms for intent recognition; 3) Decision engine, which generates response strategies based on rule systems or reinforcement learning algorithms.

6. The intelligent voice interaction system based on multimodal perception according to claim 1, characterized in that: The multimodal feedback module includes: 1) Speech synthesis unit, supporting multiple timbres and emotional expressions; 2) Visual feedback unit, including display screen, projector or holographic display technology; 3) Tactile feedback unit, including a vibration motor or an ultrasonic tactile array.

7. An intelligent voice interaction system based on multimodal perception according to any one of claims 1 to 6, characterized in that: The system connects the modules via a high-speed data bus or optical fiber, with an end-to-end interaction delay of less than 500 milliseconds.

8. An intelligent voice interaction method based on multimodal perception, characterized in that: The following steps are involved: 1) The voice acquisition and preprocessing module collects user voice signals and performs noise reduction and enhancement processing; 2) Synchronously collect visual, environmental, and user status data through a multimodal perception module and fuse them using a spatiotemporal alignment algorithm; 3) Converting speech signals into text information through the speech recognition module; 4) The speech understanding and decision-making module combines multimodal data to perform semantic analysis and intent recognition on text information and generate response strategies; 5) Feedback the interaction results to the user through the multimodal feedback module.

9. The intelligent voice interaction method based on multimodal perception according to claim 8, characterized in that: The spatiotemporal alignment algorithm includes timestamp-based data synchronization or deep learning-based multimodal feature matching.

10. The intelligent voice interaction method based on multimodal perception according to claim 8, characterized in that: The speech understanding and decision-making module introduces knowledge graph and context understanding technology to resolve semantic ambiguity and maintain coherence in multi-round dialogues.

Images