Intelligent multi-mode dialogue system and method

Abstract

The invention relates to an intelligent multi-modal dialogue system and method, and the system comprises a multi-modal input processing module which is responsible for collecting and processing the voice and video input of a user in real time, and guaranteeing the accurate transmission of information through a special data marking mechanism. And the voice recognition processing module is used for ensuring the accuracy and continuity of voice recognition by adopting an innovative segmented accumulation mechanism and a double filtering strategy. And the LLM controller module is used for realizing dynamic selection of models through intelligent analysis and balancing system performance and resource consumption. And the response generation module is responsible for integrating model output and generating a final response to ensure that the dialogue is natural and smooth. According to the method, a proper model type can be intelligently judged and used according to the dialogue content, and efficient integration of multi-modal information is realized through a special data marking and processing mechanism.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and multimodal interaction, and in particular to an intelligent multimodal dialogue system and method. Background Art

[0002] At present, intelligent dialogue systems are mainly divided into three categories: the first category is pure text dialogue systems, which interact through text; the second category is voice dialogue systems, which support voice input and output; the third category is multimodal dialogue systems, which can handle multiple input forms such as voice and video. With the development of large language model technology, the understanding ability and response quality of dialogue systems have been significantly improved.

[0003] However, existing intelligent dialogue systems lack intelligent judgment mechanisms in model selection. They either use heavyweight visual models throughout the process, resulting in a waste of resources, or fail to provide visual understanding capabilities when needed. Summary of the invention

[0004] In view of the problems existing in the prior art, the purpose of the present invention is to provide an intelligent multimodal dialogue system and method, which realizes on-demand switching between text LLM and visual LLM through an intelligent model selection mechanism, avoids the cumbersome model training process in traditional solutions, and realizes efficient reuse of existing models.

[0005] To achieve the above object, the technical solution adopted by the present invention is: An intelligent multimodal dialogue system, comprising: The multimodal input processing module includes an audio input unit, a video input unit, an audio processor and a video processor, wherein the audio input unit is communicatively connected to the audio processor, and the video input unit is communicatively connected to the video processor; the audio input unit is used to collect the user's voice input, and the audio processor packages the collected voice input into an audio data packet; the video unit is used to collect the user's video input, and the video processor performs frame processing on the collected video input and packages it into a video data packet containing visual features; The speech recognition module includes an ASR processor, which performs validity detection on audio data packets, determines whether valid speech is contained by calculating the RMS value and short-time energy characteristics of the audio stream, and then processes the valid speech segments using a segmented accumulation strategy, and finally recognizes the processed speech segments as text information; The LLM controller module includes a controller LLM, a text LLM and a visual LLM. The LLM controller performs deep semantic analysis on the text information output by the ASR processor and identifies whether visual understanding capability is required. When visual understanding capability is not required, the text LLM is called to process the text information and output a response and a response tag. When visual understanding capability is required, the visual LLM is called to obtain visual features of the video data packet at the same time. The visual features and text information are used as inputs of the visual LLM. After processing, the visual LLM outputs a response and a response tag. The response tags include "[S.LISTEN]", "[C.LISTEN]", and "[S.SPEAK]". [S.SPEAK] indicates that the system is outputting, [C.LISTEN] indicates that the system is receiving user input, and [S.LISTEN] indicates that the system is waiting for a user response. The response generation module includes a system response unit and a speech synthesis unit. The system response unit detects the response tags output by the text LLM and the visual LLM. If the output is "[S.LISTEN]", no speech synthesis processing is performed. If it is "[S.SPEAK]", the speech synthesis unit is triggered: "[S.SPEAK]" is replaced with "empty", and then input into the speech synthesis service to give a voice response to the user.

[0006] The ASR processor performs validity check specifically as follows: an adaptive dual threshold mechanism is set, and when the RMS value of the audio exceeds a times the background noise level and the short-time energy exceeds the preset threshold, it is determined to be a valid speech segment; the segment accumulation strategy is: audio processing is performed with b seconds as the basic unit, and an overlapping interval of c seconds is maintained between adjacent segments, where c<b.

[0007] The ASR processor uses a double filtering mechanism to filter the recognized text information: the time interval between adjacent recognition results is required to be greater than d seconds; the similarity between the new and old recognized texts is calculated at the same time, and the new recognized text information is retained only when the similarity is less than m%.

[0008] The speech recognition module also includes an interruption control unit, which matches the text information recognized by the ARS processor through a preset high-priority keyword library, and performs an interruption operation when matching the preset keyword; when processing, the text LLM or visual LLM combines user input and conversation context information for deep semantic analysis to identify the user's implicit interruption intention, and when the interruption intention is detected, a response mark [S.STOP] is output, indicating that the user wants to stop or pause the conversation; the system response unit detects the response mark "[S.STOP]", immediately clears the TTS output queue, and the system remains silent.

[0009] The multimodal dialogue system adopts the WebSocket communication mechanism to perform data transmission among the multimodal input processing module, the speech recognition processing module, the LLM controller module and the response generation module. When the audio processor packages the audio data, it adds a 0x00 mark to the header of the data packet; when the video processor packages the video data, it adds a 0xFF mark to the header of the data packet.

[0010] An intelligent multimodal dialogue method comprises the following steps: Step S1, collecting user input information and packaging it; Collect the audio information input by the user, sample and package the audio information to form an audio data packet; Step S2: information preprocessing; First, the validity of the audio data packet is checked to determine whether it contains valid speech by calculating the RMS value and short-time energy characteristics of the audio stream; Then, the segmented accumulation strategy is used to process the effective speech segments; Finally, the processed audio is recognized and converted into text information; Step S3, using the controller LLM to perform deep semantic analysis on the text information to determine whether visual comprehension capability is required. If visual comprehension capability is required, the text LLM is called to process the text information, and then a response and a response mark are output; if visual comprehension capability is not required, the visual LLM is called, the text information is used as the input of the visual LLM, and video data is collected to obtain visual features as the input of the visual LLM. After the visual LLM model processes the text information and visual features, a response and a response mark are output; The response tags include "[S.LISTEN]", "[C.LISTEN]", "[S.SPEAK]", and "[S.STOP]". [S.SPEAK] means that the system is outputting, [C.LISTEN] means that the system is receiving user input, and [S.LISTEN] means that the system is waiting for the user's response. Step S4, detect the response character output by the visual LLM or text LLM. If the response character is "[S.LISTEN]", no speech synthesis processing is performed. If it is "[S.SPEAK]", speech synthesis processing is triggered: "[S.SPEAK]" is replaced with "blank", and then the speech synthesis service is input to give a voice response to the user.

[0011] In step S2, the effectiveness identification is specifically as follows: an adaptive dual threshold mechanism is set, and when the RMS value of the audio exceeds a times the background noise level and the short-time energy exceeds the preset threshold, it is determined to be a valid speech segment; the segment accumulation strategy is as follows: audio processing is performed with b seconds as the basic unit, and an overlapping interval of c seconds is maintained between adjacent segments, where c<b.

[0012] In step S3, after obtaining the text information, a double filtering mechanism is used to filter the recognized text information: the time interval between adjacent recognition results is required to be greater than d seconds; the similarity between the new and old recognized texts is calculated at the same time, and the new recognized text information is retained only when the similarity is less than m%.

[0013] The method also includes interrupting the control process, as follows: During the execution of step S2, the first level of interruption control is started, and the text information recognized by the ARS processor is matched through a preset high-priority keyword library. When the preset keyword is matched, the interruption operation is performed; when the preset keyword is not matched, the second level of interruption control is started; The second level of interruption control is as follows: when processing the text LLM or visual LLM in step S3, deep semantic analysis is performed in combination with the user input and the conversation context information to identify the user's implicit interruption intention. When the interruption intention is detected, a response mark [S.STOP] is output, indicating that it is detected that the user wants to stop or pause the conversation; In step 4, if the response mark "[S.STOP]" is detected, the TTS output queue is immediately cleared and the system remains silent.

[0014] The method adopts the WebSocket communication mechanism for data transmission. In step S1, when packaging audio data, a 0x00 mark is added to the header of the data packet; in step S3, when collecting video data, video data is packaged and a 0xFF mark is added to the header of the data packet.

[0015] After adopting the above scheme, the present invention has the following beneficial effects: First, in terms of deployment efficiency, due to the adoption of a training-free design, the system can directly reuse existing open source models, significantly reducing deployment costs. Compared with traditional methods that require collecting data and training models for specific scenarios, this system can achieve plug-and-play, and the deployment time is shortened from weeks to hours.

[0016] Second, in terms of resource utilization efficiency, through the intelligent model selection mechanism, the system achieves the goal of calling the visual LLM on demand. Test data shows that in a typical conversation scenario, about 70% of the interactions can be completed using only the text LLM. Compared with the solution of always enabling the visual LLM, the system's computing resource consumption is reduced by about 50%, and the average response delay is reduced by 40%.

[0017] Third, in terms of system adaptability, the training-free architecture adopted by the present invention enables it to flexibly adapt to different types of application scenarios. Experimental verification shows that the same system architecture can be seamlessly applied to multiple fields such as customer service consultation, educational counseling, and medical inquiries without the need for targeted training. In actual application tests, the system's conversation completion rate in different scenarios exceeded 95%, and user satisfaction was 15% higher than that of traditional solutions.

[0018] Fourth, in terms of maintenance costs, due to the modular design and standardized interfaces, the system can easily update or replace various functional modules. Especially in terms of model iteration, when a new open source model is released, it can be directly integrated without retraining, which greatly reduces the maintenance cost and upgrade difficulty of the system. Through the analysis of the above technical effects, it can be seen that the untrained multimodal dialogue framework proposed in the present invention is not only innovative in technical implementation, but also shows significant cost advantages and use value in practical applications. Through innovative architectural design and control mechanism, the system effectively solves the training dependency problem in traditional solutions and provides a new solution for the large-scale application of intelligent dialogue systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a principle block diagram of the present invention. DETAILED DESCRIPTION

[0020] The present invention discloses an intelligent multimodal dialogue system, which realizes efficient and accurate human-computer dialogue interaction through innovative real-time data processing architecture and dynamic model control mechanism. The system adopts modular design and includes a complete data collection, processing, analysis and response generation process. The core feature of the system is that it can intelligently judge the use of the appropriate model type (text LLM or visual LLM) according to the content of the dialogue, and realize efficient integration of multimodal information through special data marking and processing mechanism.

[0021] like Figure 1As shown, the intelligent multimodal dialogue system of the present invention includes: (1) a multimodal input processing module responsible for real-time acquisition and processing of user voice and video input, and ensuring accurate information transmission through a special data tagging mechanism. (2) a speech recognition processing module: adopting an innovative segmented accumulation mechanism and a double filtering strategy to ensure the accuracy and coherence of speech recognition. (3) an LLM (large language model) controller module: realizing dynamic model selection through intelligent analysis, balancing system performance and resource consumption. (4) a response generation module: responsible for integrating model outputs and generating final responses to ensure the natural and smooth dialogue.

[0022] The multimodal input processing module adopts an innovative parallel processing architecture to realize real-time acquisition and processing of front-end data. The multimodal input processing module includes an audio input unit and a video input unit at the input layer, and an audio processor and a video processor at the processing layer, wherein the audio input unit is connected to the audio processor, and the video input unit is connected to the video processor.

[0023] In this embodiment, the audio input unit uses WebAPI to call the device microphone, uses a 16kHz sampling rate and a mono format to obtain data, and ensures audio quality through pre-processing steps such as real-time volume monitoring, automatic gain control, and noise suppression. In order to achieve streaming processing, the audio processor packages each 4096 (the number is the set value N, which can be set accordingly according to actual needs) sampling points collected to form an audio data packet, which is transmitted to the speech recognition processing module for processing.

[0024] The video input unit uses WebAPI to call the device camera and sets the acquisition resolution to 320x240 and the sampling frequency to 1fps. The video processor optimizes the image quality through steps such as automatic exposure adjustment and contrast optimization, and then compresses it into a data packet using the JPEG format to form a video data packet containing visual features.

[0025] At the data transmission level, this embodiment establishes a WebSocket channel to achieve two-way real-time communication, designs a unified data packet format containing information such as type tag, timestamp, and data length, and implements a complete transmission control mechanism, including automatic reconnection, heartbeat maintenance, flow control, and packet loss detection to ensure the reliability of data transmission. Therefore, the header of the audio data packet needs to be marked with 0x00, and the header of the video data packet needs to be marked with 0xFF. Through this design, the system can efficiently and stably collect and transmit multimodal data, providing high-quality input for subsequent processing modules.

[0026] The speech recognition processing module includes an ASR processor and an interruption control unit, wherein the ASR processor realizes high-quality speech recognition through an innovative multi-level processing mechanism. First, the ASR processor performs validity detection on the received audio data packet, and determines whether it contains valid speech by calculating the RMS value and short-time energy characteristics of the audio stream. Specifically, an adaptive dual threshold mechanism is set, and when the RMS value of the audio exceeds a times (1.5 times in this embodiment) the background noise level and the short-time energy exceeds the preset threshold, it is determined to be a valid speech segment. Secondly, the ASR processor adopts an innovative segmented accumulation strategy, performs audio processing with b seconds as the basic unit, and maintains an overlapping interval of c seconds (where c < b, in this embodiment, c is 0.5 and b is 2) between adjacent segments. This design not only ensures the real-time performance of speech recognition, but also effectively avoids the problem of broken speech segments. In terms of speech recognition quality control, the ASR processor implements a dual filtering mechanism based on time interval and text similarity: by requiring the time interval between adjacent recognition results to be greater than d seconds (1 second in this embodiment), overly frequent recognition is avoided; at the same time, the similarity between the new and old recognition texts is calculated, and the new recognition results are retained only when the similarity is less than m% (80% in this embodiment), effectively solving the problem of repeated recognition. Through this multiple processing mechanism, the user's voice input can be accurately extracted and converted into high-quality text information, providing reliable input data for the subsequent LLM controller module.

[0027] The interruption control unit adopts an innovative two-level interruption detection architecture, and realizes an efficient and intelligent interruption mechanism through a hierarchical judgment strategy. At the first level, the interruption control unit quickly matches through a preset high-priority keyword library. The keywords include "stop", "stop", "don't talk", "wait", "wait a minute", etc. When these keywords are detected, the system immediately performs interruption operations, including stopping the current speech synthesis, clearing the audio buffer, resetting the conversation state, etc., to ensure millisecond-level response speed. When the first-level keyword match is not hit, the system starts the second-level semantic analysis mechanism, sends the user input together with the conversation context to the LLM for deep semantic analysis, and guides the LLM to identify the user's implicit interruption intention through specific system prompt words. When the LLM detects the interruption intention, it will include a special control mark in the response. The system continuously analyzes the LLM output through the state monitoring mechanism and triggers the interruption process. To ensure the reliability of the interruption mechanism, the system implements a complete state synchronization control mechanism, establishes a two-way state communication channel between the ASR processor and the speech synthesis processor, and synchronizes the processing state in real time. At the same time, the system introduces an interruption cooling-off period mechanism to maintain a high detection threshold for a short period of time after speech synthesis stops, effectively avoiding false triggering. This hierarchical judgment dual-level interruption control mechanism not only ensures immediate response to explicit interruption instructions through keyword matching, but also can intelligently understand various implicit interruption intentions through LLM when the keyword is not hit, significantly improving the system's natural interaction and intelligence level.

[0028] The LLM controller module implements an intelligent model selection and dialogue management mechanism. The LLM controller module includes a controller LLM, a text LLM, and a visual LLM. The controller LLM connects the text LLM and the visual LLM. The text information output by the ASR processor is input into the controller module, which performs deep semantic analysis on the text information (i.e., user input) and identifies whether visual understanding is required through keyword matching and context understanding. For example, it detects whether visual indicators such as "see", "in the image", and "in the video" appear in the input, or analyzes the dialogue history to determine whether the current dialogue is in a visual context. When it is determined that visual understanding is not required, the lightweight text LLM is called to process the text information, significantly reducing the consumption of computing resources; when visual understanding is indeed required, the system seamlessly switches to the visual LLM and processes the text information and the visual features of the video data packet as input to ensure that the visual content can be accurately understood. In order to maintain the continuity of the dialogue, the system maintains the most recent 10 dialogue records as context information, and records whether each dialogue involves visual content through special tags. This design enables the system to accurately track the dialogue status and avoid understanding interruptions caused by model switching. Through this intelligent control mechanism, resource utilization is optimized while ensuring functional integrity.

[0029] After the text LLM and visual LLM processing is completed, the response tags will be output. The response tags include "[S.LISTEN]", "[C.LISTEN]", "[S.SPEAK]", "[S.STOP]", etc. [S.SPEAK] means that the system is outputting, [C.LISTEN] means that the system is receiving user input, [S.LISTEN] means that the system is waiting for the user's response, and [S.STOP] means that it is detected that the user wants to stop or pause the conversation. Accurate conversation flow control is achieved through these status tags.

[0030] The response generation module ensures the coherence and naturalness of the system output through innovative state management and multi-level processing mechanisms. The response generation module includes the system response unit and the speech synthesis (TTS) unit of the output layer. The system response unit normalizes the output of the text LLM and the visual LLM, including removing special characters, unifying punctuation marks, adjusting modal particles, etc., to ensure the standardization of the output text. Specifically, the response mark of the text LLM and the visual LLM output is detected. If the output is "[S.LISTEN]", no speech synthesis processing is performed. If it is "[S.SPEAK]", the speech synthesis processing is triggered: "[S.SPEAK]" is replaced with "empty", and then the speech synthesis service is input to respond to the user; if it is "[S.STOP]", the TTS output queue is immediately cleared and the system remains silent. The speech synthesis unit will select the appropriate output form according to the current dialogue state. For example, in video-related answers, the system will adjust the use of indicator words in combination with the picture information to make the answer more natural. In order to achieve real-time speech output, the system adopts a streaming synthesis strategy to perform speech synthesis while generating text, and transmit it to the front end in real time through the WebSocket channel. This design not only improves the timeliness of system response, but also ensures the synchronization of multimodal output through precise state control. Through this multi-level response generation mechanism, the system can provide a smooth and natural interactive experience and effectively improve user satisfaction.

[0031] Based on the above multimodal dialogue system, the present invention also discloses an intelligent multimodal dialogue method, which includes the following steps: Step S1: Collect user input information and package it.

[0032] Specifically, the audio information input by the user is collected, the audio information is sampled and packaged, and a 0x00 mark is added to the header of the data packet to form an audio data packet.

[0033] Step S2: Information preprocessing.

[0034] The validity check is performed on the audio data packets, and the RMS value and short-time energy characteristics of the audio stream are calculated to determine whether it contains valid speech. Specifically, an adaptive dual threshold mechanism is set, and when the RMS value of the audio exceeds 1.5 times the background noise level and the short-time energy exceeds the preset threshold, it is determined to be a valid speech segment.

[0035] Then, an innovative segmented accumulation strategy is used to process the valid speech segments: audio processing is performed in m (e.g., 2) seconds as the basic unit, and an overlap interval of n (e.g., 0.5) seconds is maintained between adjacent segments. This design ensures the real-time performance of speech recognition and effectively avoids the problem of speech segment fragmentation.

[0036] Finally, the processed audio is recognized and converted into text information.

[0037] In order to improve the recognition effect, the present invention recognizes audio based on double filtering of time interval and text similarity. Specifically, the time interval between adjacent recognition results is required to be greater than 1 second to avoid too frequent recognition. At the same time, the similarity between the new and old recognized texts is calculated, and the new recognition result is retained only when the similarity is lower than 80%, which effectively reduces the problem of repeated recognition.

[0038] Step S3, use the controller LLM to perform deep semantic analysis on the text information to determine whether visual comprehension ability is required. If visual comprehension ability is required, call the text LLM to process the text information, and then output a response and a response mark; if visual comprehension ability is not required, call the visual LLM, use the text information as the input of the visual LLM, and collect video data to obtain visual features as the input of the visual LLM. After the visual LLM model processes the text information and visual features, it outputs a response and a response mark.

[0039] Whether visual understanding is needed can be identified through keyword matching and context understanding. For example, detect whether visual indicators such as "see", "in the image", "in the video" appear in the input, or analyze the conversation history to determine whether the current conversation is in a visually relevant context. When it is determined that visual understanding is not needed, the lightweight text LLM is called to process the conversation, significantly reducing the consumption of computing resources; when visual understanding is indeed needed, the system seamlessly switches to the visual LLM to ensure that the visual content can be accurately understood. In order to maintain the continuity of the conversation, the system maintains the 10 most recent conversation records as context information, and records whether each conversation involves visual content through special tags. This design enables the system to accurately track the state of the conversation and avoid understanding breaks caused by model switching. Through this intelligent control mechanism, resource usage is optimized while ensuring functional integrity.

[0040] In this embodiment, WebAPI is used to call the device camera for video acquisition, and the acquisition resolution of 320x240 and the sampling frequency of 1fps are set. The acquired video is processed by the video processor and then input into the visual LLM.

[0041] Response tags include "[S.LISTEN]", "[C.LISTEN]", "[S.SPEAK]", "[S.STOP]", etc. [S.SPEAK] indicates that the system is outputting, [C.LISTEN] indicates that the system is receiving user input, [S.LISTEN] indicates that the system is waiting for the user's response, and [S.STOP] indicates that it is detected that the user wants to stop or pause the conversation. These status tags can be used to achieve precise control of the conversation process.

[0042] Step S4, detect the response character output by the visual LLM or text LLM. If the response character is "[S.LISTEN]", no speech synthesis processing is performed. If it is "[S.SPEAK]", speech synthesis processing is triggered: "[S.SPEAK]" is replaced with "empty", and then the speech synthesis service is input to give a voice response to the user. If it is "[S.STOP]", the TTS output queue is immediately cleared and the system remains silent.

[0043] The present invention adopts a two-level interruption detection architecture, and realizes an efficient and intelligent interruption mechanism through a hierarchical judgment strategy. At the first level, during the execution of step S2, a quick match is performed through a preset high-priority keyword library, and the keywords include "stop", "stop", "don't say anymore", "wait", "wait a minute", etc. When these keywords are detected, the system immediately performs an interruption operation, including stopping the current speech synthesis, clearing the audio buffer, resetting the dialogue state, etc., to ensure a millisecond-level response speed. When the first-level keyword match is not hit, the system starts the second-level semantic analysis mechanism, sends the user input together with the dialogue context to the text LLM or visual LLM for deep semantic analysis, and guides the text LLM or visual LLM to identify the user's implicit interruption intention through specific system prompt words. When the interruption intention is detected, a special control mark [S.STOP] will be included in the response, and the system will continuously analyze the LLM output through the state monitoring mechanism and trigger the interruption process. To ensure the reliability of the interruption mechanism, the system implements a complete state synchronization control mechanism, establishes a two-way state communication channel between the ASR processor and the speech synthesis processor, and synchronizes the processing state in real time. At the same time, the system introduces an interruption cooling-off period mechanism to maintain a high detection threshold for a short period of time after speech synthesis stops, effectively avoiding false triggering. This hierarchical judgment dual-level interruption control mechanism not only ensures immediate response to explicit interruption instructions through keyword matching, but also can intelligently understand various implicit interruption intentions through LLM when the keyword is not hit, significantly improving the system's natural interaction and intelligence level.

[0044] The present invention solves the model dependency problem in the prior art through an intelligent model selection mechanism and a sophisticated control strategy. Unlike traditional technologies that require model training or fine-tuning for specific scenarios, this solution achieves efficient reuse of existing models through an innovative dynamic control framework. At the model selection level, the system automatically determines the dialogue requirements based on deep semantic analysis, realizes on-demand switching between text LLM and visual LLM, and avoids the heavy model training process in traditional solutions. In terms of data processing, the system uses a special marking mechanism (0x00 / 0xFF) to achieve accurate classification of audio and video data, and ensures data quality through differentiated preprocessing strategies. The speech processing module innovatively designs a dual threshold detection mechanism based on RMS value and short-time energy, combined with a segmented accumulation strategy of 2 seconds and a 0.5 second overlapping interval, which effectively solves the quality problem of speech recognition. The core innovation of the system lies in the design of a complete set of training-free adaptation mechanisms, which intelligently schedules the existing model capabilities through the LLM controller, ensuring the adaptability of the system to different scenarios and avoiding the heavy training overhead in traditional solutions. In the response generation phase, the system implements state management through special tags and uses streaming processing to ensure real-time performance. This design enables the system to be quickly deployed and adapted to different application scenarios. Through the organic combination of these key technologies, the present invention not only solves the training dependency problem in existing multimodal dialogue systems, but also demonstrates excellent versatility and scalability in practical applications.

[0045] The use of the above-mentioned untrained multimodal dialogue system can significantly improve the overall performance of the system. In terms of technical effects, it is mainly reflected in the following aspects: First, in terms of deployment efficiency, due to the adoption of untrained design ideas, the system can directly reuse existing open source models, significantly reducing deployment costs. Compared with the traditional method that requires data collection and model training for specific scenarios, this system can achieve plug-and-play, and the deployment time is shortened from several weeks to several hours. Secondly, in terms of resource utilization efficiency, through the intelligent model selection mechanism, the system achieves the goal of calling visual LLM on demand. Test data show that in a typical dialogue scenario, about 70% of the interactions can be completed using only text LLM. Compared with the solution that always enables visual LLM, the system's computing resource consumption is reduced by about 50%, and the average response delay is reduced by 40%. Third, in terms of system adaptability, the untrained architecture adopted by the present invention enables it to flexibly adapt to different types of application scenarios. Experimental verification shows that the same system architecture can be seamlessly applied to multiple fields such as customer service consultation, educational counseling, and medical inquiries without targeted training. In actual application tests, the system's dialogue completion rate in different scenarios exceeded 95%, and user satisfaction was 15% higher than that of traditional solutions. Finally, in terms of maintenance costs, due to the modular design and standardized interfaces, the system can easily update or replace various functional modules. Especially in terms of model iteration, when a new open source model is released, it can be directly integrated without retraining, which greatly reduces the maintenance cost and upgrade difficulty of the system. Through the analysis of the above technical effects, it can be seen that the untrained multimodal dialogue framework proposed in the present invention is not only innovative in technical implementation, but also shows significant cost advantages and use value in practical applications. Through innovative architectural design and control mechanism, the system effectively solves the training dependency problem in traditional solutions and provides a new solution for the large-scale application of intelligent dialogue systems.

[0046] The above description is merely an embodiment of the present invention and does not limit the technical scope of the present invention. Therefore, any minor modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. An intelligent multimodal dialogue system, characterized in that: include: The multimodal input processing module includes an audio input unit, a video input unit, an audio processor and a video processor, wherein the audio input unit is communicatively connected to the audio processor, and the video input unit is communicatively connected to the video processor; the audio input unit is used to collect the user's voice input, and the audio processor packages the collected voice input into an audio data packet; the video unit is used to collect the user's video input, and the video processor performs frame processing on the collected video input and packages it into a video data packet containing visual features; The speech recognition module includes an ASR processor, which performs validity detection on audio data packets, determines whether valid speech is contained by calculating the RMS value and short-time energy characteristics of the audio stream, and then processes the valid speech segments using a segmented accumulation strategy, and finally recognizes the processed speech segments as text information; The LLM controller module includes a controller LLM, a text LLM and a visual LLM. The LLM controller performs deep semantic analysis on the text information output by the ASR processor and identifies whether visual comprehension capability is required. When visual comprehension capability is not required, the text LLM is called to process the text information and output a response and a response tag. When visual comprehension capability is required, the visual LLM is called to obtain visual features of the video data packet at the same time. The visual features and text information are used as inputs of the visual LLM. After processing, the visual LLM outputs a response and a response tag. The response tags include "[S.LISTEN]", "[C.LISTEN]", and "[S.SPEAK]". [S.SPEAK] indicates that the system is outputting, [C.LISTEN] indicates that the system is receiving user input, and [S.LISTEN] indicates that the system is waiting for a user response. The response generation module includes a system response unit and a speech synthesis unit. The system response unit is used to detect the response tags output by the text LLM and the visual LLM. If the output is "[S.LISTEN]", no speech synthesis processing is performed. If it is "[S.SPEAK]", the speech synthesis unit is triggered: "[S.SPEAK]" is replaced with "empty", and then input into the speech synthesis service to give a speech response to the user.

2. The intelligent multimodal dialogue system according to claim 1, characterized in that: The ASR processor performs validity check specifically as follows: an adaptive dual threshold mechanism is set, and when the RMS value of the audio exceeds a times the background noise level and the short-time energy exceeds the preset threshold, it is determined to be a valid speech segment; the segment accumulation strategy is: audio processing is performed with b seconds as the basic unit, and an overlapping interval of c seconds is maintained between adjacent segments, where c<b.

3. The intelligent multimodal dialogue system according to claim 1, characterized in that: The ASR processor uses a double filtering mechanism to filter the recognized text information: the time interval between adjacent recognition results is required to be greater than d seconds; the similarity between the new and old recognized texts is calculated at the same time, and the new recognized text information is retained only when the similarity is less than m%.

4. The intelligent multimodal dialogue system according to claim 1, characterized in that: The speech recognition module also includes an interruption control unit, which matches the text information recognized by the ARS processor through a preset high-priority keyword library, and performs an interruption operation when matching the preset keyword; when processing, the text LLM or visual LLM combines user input and conversation context information for deep semantic analysis to identify the user's implicit interruption intention, and when the interruption intention is detected, a response mark [S.STOP] is output, indicating that the user wants to stop or pause the conversation; the system response unit detects the response mark "[S.STOP]", immediately clears the TTS output queue, and the system remains silent.

5. The intelligent multimodal dialogue system according to claim 1, characterized in that: The multimodal dialogue system adopts the WebSocket communication mechanism to perform data transmission among the multimodal input processing module, the speech recognition processing module, the LLM controller module and the response generation module. When the audio processor packages the audio data, it adds a 0x00 mark to the header of the data packet; when the video processor packages the video data, it adds a 0xFF mark to the header of the data packet.

6. An intelligent multimodal dialogue method, characterized in that: The following steps are involved: Step S1, collecting user input information and packaging it; Collect the audio information input by the user, sample and package the audio information to form an audio data packet; Step S2: information preprocessing; First, the validity of the audio data packet is checked to determine whether it contains valid speech by calculating the RMS value and short-time energy characteristics of the audio stream; Then, the segmented accumulation strategy is used to process the effective speech segments; Finally, the processed audio is recognized and converted into text information; Step S3, using the controller LLM to perform deep semantic analysis on the text information to determine whether visual comprehension capability is required. If visual comprehension capability is required, calling the text LLM to process the text information, and then outputting a response and a response tag; If visual understanding is not required, the visual LLM is called, and the text information is used as the input of the visual LLM. At the same time, video data is collected to obtain visual features as the input of the visual LLM. After the visual LLM model processes the text information and visual features, it outputs a response and a response tag. Response tags include "[S.LISTEN]", "[C.LISTEN]", "[S.SPEAK]", and "[S.STOP]". [S.SPEAK] indicates that the system is outputting, [C.LISTEN] indicates that the system is receiving user input, and [S.LISTEN] indicates that the system is waiting for user response. Step S4, detect the response character output by the visual LLM or text LLM. If the response character is "[S.LISTEN]", no speech synthesis processing is performed. If it is "[S.SPEAK]", speech synthesis processing is triggered: "[S.SPEAK]" is replaced with "blank", and then the speech synthesis service is input to give a voice response to the user.

7. The intelligent multimodal dialogue method according to claim 6, characterized in that: In step S2, the effectiveness identification is specifically as follows: an adaptive dual threshold mechanism is set, and when the RMS value of the audio exceeds a times the background noise level and the short-time energy exceeds the preset threshold, it is determined to be a valid speech segment; the segment accumulation strategy is as follows: audio processing is performed with b seconds as the basic unit, and an overlapping interval of c seconds is maintained between adjacent segments, where c<b.

8. The intelligent multi-modal dialogue method according to claim 6, characterized in that: In step S3, after obtaining the text information, a double filtering mechanism is used to filter the recognized text information: the time interval between adjacent recognition results is required to be greater than d seconds; the similarity between the new and old recognized texts is calculated at the same time, and the new recognized text information is retained only when the similarity is less than m%.

9. The intelligent multi-modal dialogue method according to claim 6, characterized in that: The method also includes interrupting the control process, as follows: During the execution of step S2, the first level of interruption control is started, and the text information recognized by the ARS processor is matched through a preset high-priority keyword library. When the preset keyword is matched, the interruption operation is performed; when the preset keyword is not matched, the second level of interruption control is started; The second level of interruption control is as follows: when processing the text LLM or visual LLM in step S3, deep semantic analysis is performed in combination with the user input and the conversation context information to identify the user's implicit interruption intention. When the interruption intention is detected, a response mark [S.STOP] is output, indicating that it is detected that the user wants to stop or pause the conversation; In step 4, if the response mark "[S.STOP]" is detected, the TTS output queue is immediately cleared and the system remains silent.

10. The intelligent multi-modal dialogue method according to claim 6, characterized in that: The method adopts the WebSocket communication mechanism for data transmission. In step S1, when packaging audio data, a 0x00 mark is added to the header of the data packet; in step S3, when collecting video data, video data is packaged and a 0xFF mark is added to the header of the data packet.

Images